Product and service reviews are conducted independently by our editorial team, but we sometimes make money when you click on links. Learn more.

Avago Ships PCIe Express Fabric Switch Technology

By - Source: Avago Technologies

Avago Technologies has been purchasing companies at a rapid clip lately. LSI joined Avago in December of 2013, followed by PLX Technologies in 2014 and Emulex in 2015. Each of these companies brings a wealth of experience and IP, and it's natural to assume the plan is to merge the best technology from each company, thereby spawning a new cohesive set of products. 

The big picture is starting to come into focus with the release of Avago's PEX9700 Series of third-generation switches with ExpressFabric technology. The ExpressFabric technology enables the use of the ubiquitous PCIe connection as a rack-scale interconnect for servers, storage and networking. Avago designed the products to deliver a 50 percent reduction in cost and power consumption for hyperconverged, NVMe and rack-scale systems when compared with standard connectivity platforms. These products are derived from the core technology brought over with the PLX acquisition.

PCIe has broad compatibility, and it scales easily; 8G (x1), 32G (x4), 64G (x8) or 128G (x16) connections allow low-latency communication (~150ns per hop) between components. PCIe is already directly embedded into the CPU. Utilizing PCIe as a fabric reduces redundant I/O devices, which alleviates acquisition, power and cooling costs. 

One of the initial hurdles to deploying PCIe as a fabric stemmed from its lack of High-Availability features. The PCI-SIG standards committee created Downstream Port Containment to isolate endpoint errors, thus preventing them from bringing down the entire system. The committee also added other features to provide SAS-like robustness (such as multi-pathing, failover and zoning), which Avago utilizes with the ExpressFabric platform. The redundancy and failover features enable six 9's of availability. Avago positions its controllers, brought in from the LSI acquisition, as a storage component in the new architectures. 

The ExpressFabric architecture also includes a transparent bridging function that can connect two topologies together. This system can use DMA to transfer data between the clusters, and it also provides failover functionality.

The PEX9700 platform interconnects up to 24 (+1 management host) nodes with a single chip. With a Tunneled Windows Connection, users can cascade multiple chips to connect up to 72 nodes. Each port features a NIC-DMA engine to streamline data transfers between multiple hosts' memory, and flexible port configuration allows up to 23 endpoints to be combined with ports up to 16 lanes wide (nearly 16 GB/s of throughput per link). Avago provides dedicated port management and software modules that control switch features for customized implementations.

Peer-to-peer data transfers between GPGPU clusters are an attractive proposition for the HPC market, and they're indicative of the new type of architectures enabled by PCIe fabric. The PEX9700 Series of ExpressFabric products are now shipping to leading OEMs, with seven new models ranging from 12-Lanes/5-Ports to 97-Lanes/25-Ports. These products are compatible with PCIe 3.0, and Avago is already developing PCIe 4.0 products.