Product and service reviews are conducted independently by our editorial team, but we sometimes make money when you click on links. Learn more.

EMC DSSD D5, Up Close And Personal: Flashtec NVMe1032 SSD Controller Inside

By - Source: Toms IT Pro
Tags :

EMC unveiled a new striping capability and converged offerings for its DSSD D5 All-Flash Arrays here at EMC World 2016, and we were also able to take a close look at its internal components. The addition of a converged system was not entirely unexpected, as EMC alluded to the pending product during its DSSD D5 launch in February. However, the addition of a striping capability, which doubles IOPS up to 20 million and bandwidth to 200 GB/s, was somewhat unexpected. Unfortunately, for now the applications that can utilize the dual-controller architecture are limited (more on that in a bit.)

DSSD D5 Storage Module Controller

The DSSD D5 employs 36 all-flash storage modules that slot into the front of the appliance. Each module features either 2 TB or 4 TB of flash and is hot-pluggable to ensure easy serviceability. The ability to swap out modules will also facilitate simple capacity upgrades to denser modules in the future.

EMC is one of the few vendors that actually employs custom-designed flash modules in its appliances, whereas many flash vendors simply use standard form factor SSDs. We seized the unique opportunity to examine the interior of a DSSD D5 flash module at EMC World 2016.

Surprisingly, a first-generation PMC-Sierra (now Microsemi) Flashtec NVMe1032 SSD controller (89HF32P08CG3) powers the DSSD D5 SSD modules.

The NVMe1032 controllers originally made their datacenter debut in the Samsung XS1715 in October 2014. The commodity controllers are used in a multitude of SSDs, such as the OCZ ZD6000 and aforementioned Samsung XS1715, along with the Memblaze PBlaze 4 and HGST SN150 SSDs we recently evaluated. We are also wrapping up a product evaluation of the QCT XC100, which also employs the venerable controller.

The field-hardened NVMe1032 controllers come in both 16- and 32-channel variants. The aforementioned SSDs utilize the 16-channel model, but EMC is utilizing the 32-channel model for the DSSD D5.

Third party SSD controllers are generally broken into two categories: those with locked firmware that vendors cannot modify, and those that allow vendors free reign to modify the internal controller functions. Flashtec controllers fall into the latter category, and merely provide a base level of functionality as a canvas, such as protocol support (NVMe in this case), ECC, flash channel RAID and internal data path protection, among others.

The fact that there is already a NAND-level parity scheme embedded into the controller is interesting. EMC is employing its own three dimensional Cubic RAID implementation. Cubic RAID spreads the parity across the entire global NAND pool to boost efficiency and reduce parity overhead to only 17 percent. EMC indicated that Cubic RAID is a cell-level approach, which means it has either heavily altered the PMC parity scheme or removed it entirely and replaced it with its own implementation.

The SSD controllers are also compatible with numerous types of NAND, such as MLC, SLC and TLC NAND with either ONFI or Toggle interfaces. This affords EMC a broad range of options for flash vendors, which is important to ensure the fabless EMC can secure access to competitively priced flash. The board we analyzed employed planar Toshiba eMLC. The PMC-Sierra controllers also provide a clear path evolutionary path forward--its next-generation of Flashtec NVMe flash controllers are already sampling.

The PMC controllers also support up to 8 TB of capacity, which will allow EMC to double the capacity of the current design while still employing the same controller and firmware architecture. We also note four DRAM emplacements to the right of the controller, which suggests the DSSD D5 module employs a standard 1 MB-per-GB DRAM allocation for LBA table management.

PMC-Sierra SSD controllers also support dual-port NVMe connectivity, which is a key feature of the DSSD D5 module design (as evidenced by the dual PCIe 3.0 x2 connections on the right of the module). The OCZ ZD6000 was the first PMC-powered NVMe SSD to utilize the dual-port NVMe functionality, and Intel also has its dual-port DC D3700/3600 coming to market, though it employs an Intel-proprietary controller.

The emergence of wide industry support for dual-port NVMe is important because it brings enterprise-class redundancy and HA features to NVMe SSDs. The lack of dual-port functionality has been hampering adoption for NVMe SSDs in all-flash arrays. EMC is ahead of the curve in this aspect, but others are catching up quickly, such as the Axellio AFA from X-IO.

We also observed the PCIe HBA that connects the DSSD to its hosts. Flexible PCIe cabling brings the speed of PCIe and the goodness of NVMe Over Fabrics directly to the host through thin flexible cabling.The dual-ported HBA connects to the rear of the appliance, with one connection being routed to the high-performance PCIe connections on the top, and the other routed to the bottom row. Each 48-port row of PCIe connections are self-contained into separate hot-swappable PCIe switching trays, which serve as the storage fabric. The redundancy provides either HA features or increased performance, depending upon configuration.

DSSD D5 Data Striping

EMC prioritizes performance above all other considerations with the DSSD D5, so the company did not include latency-inducing deduplication and compression functionality. The lack of data reduction technology limits capacity to 144 TB.

Striping data between two DSSD D5 arrays allows users to double the capacity of the 144 TB appliance, and it also doubles the attainable IOPS performance and throughput. Representatives on the show floor indicated that the striping capability is not borne of a hardware-based approach, which means the DSSD D5 relies upon the host servers to manage data placement. Unfortunately, common Linux RAID stacks incur a heavy CPU tax, as was painfully evident in our recent evaluation of the One Stop Storage FSA 200 flash-based appliance with 64 PCIe SSDs. RAID overhead inevitably robs applications of precious CPU cycles.

EMC is merely allowing Oracle to handle the data placement to circumvent RAID limitations and reduce host overhead. However, this currently limits the dual-DSSD D5 implementation to specific Oracle applications, though the company is working to expand striping to broader use-cases.

Paul Alcorn is a Contributing Editor for Tom's IT Pro, covering Storage. Follow him on Twitter and Google+.