Year in Review: Data-Center Networks Outpace Others
28nm Technology Finally Reaches Networking Chips
By Bob Wheeler
Call them laggards, but networking-silicon vendors have been slow to adopt new process technologies. In particular, vendors of standard products (ASSPs) have been happily shipping 40nm and even 65nm chips. FPGA vendors, however, are addressing networking designs with cutting-edge technology.
On both fronts, 2014 saw waves of new products that moved to the next node or to a new foundry. For ASSPs, it was the year of 28nm, led by Ethernet switches and PHYs. Xilinx shipped its first 20nm-generation (UltraScale) FPGAs. Intel made an impact as a foundry and ASIC vendor, with two customers shipping devices using its 22nm FinFET technology. Because transistor costs are rising in new process nodes, these ASSP and FPGA designs are likely to have long production lives.
The high growth rate and rapid technology innovation of data centers were behind some of the most exciting new 28nm silicon. Many vendors upgraded their Ethernet client (NIC) offerings in conjunction with Intel’s 3Q14 Grantley server launch. Mellanox, which was early with 40G Ethernet NICs, saw a huge volume ramp at the end of 2014. But industry efforts to standardize 25G and 50G Ethernet overshadowed the ramp of 40GbE. Broadcom’s Tomahawk grabbed the spotlight as the first high-density 100GbE switch to reach sampling, and the 28nm chip also enables 25G/50G Ethernet ports.
Although the enterprise networking market has been stagnant, faster Wi-Fi speeds and 10G Ethernet caused some much needed activity. Delivering speeds greater than 1Gbps, 802.11ac Wave 2 access points drive the need for faster Ethernet connections but also compete with Ethernet client access. The industry responded with new products that enable 2.5Gbps and 5Gbps Ethernet rates over twisted pair (UTP), as well as low-cost 10GbE switch chips suitable for SMB, all built in 28nm technology.
The pace of change in service-provider networks seems glacial compared with that of cloud data centers. Although network-function virtualization (NFV) got most of the attention, Carrier Ethernet continued to advance as the access and aggregation technology for new carrier infrastructure. In residential broadband, operators are developing competing approaches to gigabit services using twisted-pair copper, coax, or fiber. For those wishing to extend the life of their copper local loops, new G.fast chipsets offer a path to gigabit-class speeds. For service-provider routers, the biggest news was Marvell’s entry into search coprocessors, a market Broadcom dominates with its KBPs.
Intel will be the vendor to watch in 2015 as it introduces several new products targeting data centers. These offerings should include new Ethernet switch chips, a new low-latency 100Gbps fabric, and related fiber-optic products using silicon photonics.
Data Centers Deploy 40GbE as 100GbE Looms
Since Broadcom holds nearly a 90% share of the 10GbE/40GbE switch-ASSP market, customers would welcome greater competition in data-center switching. Yet the leading vendor is first to market for 100GbE, threatening to extend its reign. Tomahawk also enables 25G and 50G Ethernet ports using one or two 25Gbps serdes lanes, respectively. As Figure 1 shows, the new chip represents Broadcom’s fifth-generation design for data-center switching. Compared with Trident II, it enables a larger number of ACL or SDN rules and adds new visibility features, but it remains fundamentally a Layer 3 switch.
Figure 1. The evolution of 10GbE, 40GbE, and 100GbE switch chips. Broadcom’s relentless execution has left few openings for competitors. (Source: vendors, except *schedule estimate by The Linley Group)
Through its acquisition of Xpliant, Cavium is poised to disrupt the status quo in data-center switching. It is promising a 100GbE switch chip that matches Tomahawk’s density and feature set but uses a protocol-agnostic architecture. Although Cavium will initially deliver the XPA as a Layer 3 device similar to Tomahawk, the underlying design appears capable of implementing both existing and future software-defined-networking (SDN) protocols. Owing to the new silicon and new software, however, the company faces a long road to customer production.
After developing its multiprotocol SwitchX chips, Mellanox surprisingly chose to fork its roadmap for 100Gbps (EDR) InfiniBand and 100GbE switch chips. It says a dedicated design yielded the lowest possible InfiniBand latency: less than 90ns for the 28nm Switch-IB silicon. The company has not disclosed plans for a 100GbE switch chip, but we expect it will sample such a device in 2015. Mellanox’s decision to prioritize IB over 100GbE may be due to forthcoming HPC-interconnect competition from Intel’s Omni Scale fabric in 2015.
In server connectivity, Mellanox should be first to production with a mainstream 100GbE NIC, its ConnectX-4 EN adapter. QLogic is sampling an unannounced 100GbE NIC it gained by acquiring Broadcom’s NetXtreme II line in March 2014, but we expect the adapter will reach production about two quarters later than ConnectX-4. Meanwhile, 40GbE NIC competition heated up in 2014 as Emulex and Intel joined Chelsio and Mellanox in this fast-growing segment. The wildcard is Broadcom, which we expect to reenter the Ethernet controller market in 2015 with a new design for 25GbE/50GbE and possibly 100GbE as well.
Whereas “dumb” NICs remain the standard for high-volume server connectivity, SDN and cloud computing have spawned a new crop of intelligent NICs. They range from Cavium’s LiquidIO 10GbE adapters to Netronome’s new FlowNIC 100GbE adapters. A new entrant is EZchip, which is shipping multiport 10GbE NICs from its Tilera acquisition. These processor-based adapters can offload Open vSwitch, IPSec, DPI, and other processing-intensive functions from server CPUs.
Enabling the Mostly Wireless Enterprise
Given that shipment of the first high-density 10GbE switch chips was in 2006, it’s about time that optimized chips became available for SMB and embedded applications. To that end, Broadcom and Marvell both sampled new 10GbE switch chips with only 16–24 ports. Both use 28nm technology, integrate an ARM CPU, and provide limited Layer 3 features. Embedded applications for these devices include high-density servers, control-plane switching, and macro base stations. SMB pizza boxes based on these chips can connect servers as well as Wi-Fi access points.
For the 802.11ac generation, however, 10GbE is overkill. Furthermore, 10GBase-T requires Category 6a or 7 cabling, which is available in only a minority of enterprise networks. Despite the lack of standards, several vendors therefore introduced Ethernet products for 2.5Gbps (2.5GbE) or 5Gbps (5GbE) rates. For switches, Vitesse sampled its SparX-IV-80, which can handle 24x2.5GbE ports plus 4x10GbE uplinks and should be less costly than the Broadcom and Marvell 10GbE switches.
To enable enterprises to use installed cables, Aquantia and Broadcom developed competing technologies for 2.5GbE and 5GbE over Category 5e and 6 UTP. They also formed dueling alliances and helped initiate an 802.3 study group to create standards. Both vendors’ initial PHY implementations, however, are 10GBase-T designs with 2.5GbE and 5GbE modes added. As a result, switch systems for the new speeds are unlikely to provide significant cost savings relative to 10GBase-T designs. Still, end customers could install 10GbE switches that support 2.5GbE/5GbE and delay upgrading their cable infrastructure.
The availability of 802.11ac Wave 2 access points is expected to drive enterprises to upgrade their Wi-Fi infrastructure. The main new features for enterprise networks are 4x4 MIMO and multiuser MIMO (MU-MIMO), which increase maximum and effective bandwidth, respectively. Broadcom, Qualcomm Atheros, and Quantenna offer chipsets with these features, but the Wi-Fi Alliance has yet to upgrade its “Wi-Fi Certified ac” program. Once certified access points become available in 2015, the chip-vendor landscape will become clearer.
Carriers Gain Alternatives
In wired carrier infrastructure, G.fast emerged as the successor to VDSL2. Offering a reach of only 250 meters, it requires a fiber-to-the-distribution-point (FTTDP) topology, as Figure 2 shows. When coexisting with VDSL in the same bundle, it delivers up to 700Mbps. Surprisingly, a startup is among the first vendors to sample a G.fast chipset: Sckipio has formed a partnership with larger Lantiq for joint CPE-gateway designs. VDSL incumbent Broadcom also demonstrated G.fast silicon. The new standard requires vectoring, whose implementation could be a major differentiator for distribution-point-unit (DPU) chipsets. Another factor will be power dissipation—Sckipio claims a leading 1.5W per line.
Figure 2. Spectrum use with fiber to a distribution point (DP). Carriers can move customers from VDSL2 to G.fast line by line. G.fast lines can use the VDSL spectrum once all customers in a bundle have upgraded.
In aggregation and routing, Marvell’s increased investments are bearing fruit. In 2014, the company brought its Bobcat 2 Carrier Ethernet switch to production and introduced the low-power PONCat 3 for access designs. These 28nm chips give Marvell competitive advantages over Broadcom’s Katana switch chips.
In another challenge to Broadcom, Marvell sampled its Questflo search coprocessor late in 2014. Providing twice the ACL-rule capacity of Broadcom’s biggest KBP, the new chip looks impressive. When it comes to the delivered performance of algorithmic solutions, however, what looks great in a vendor’s lab can become a different story in customer hands. Thus, we will reserve judgment on Questflo until we hear the results of customer evaluations.
Xilinx Kintex First to Production in TSMC 20nm
In the networking arena, FPGA vendors remained at the forefront of process-technology adoption. In 4Q14, the first Xilinx 20nm FPGA reached production; the highest-volume application for this Kintex midrange device is in remote radio heads for wireless base stations. UltraScale Virtex devices should reach production in 1H15. Altera is sampling its 20nm Arria 10 midrange FPGAs, but it is using Intel’s 14nm FinFET technology for Stratix 10, which is due to sample in mid-2015—about a year behind Xilinx’s 20nm Virtex. The other company vying for high-end networking designs is Tabula, which has reached production with its Abax2 chips built in Intel’s older 22nm FinFET process.
Once viewed as the silicon jack-of-all-trades, FPGAs and their associated tools have become increasingly specialized. Tabula is sharply focused on high-speed networking designs, as Abax2’s hard- and soft-IP blocks evince. Xilinx demonstrated its networking investment during 2014 by launching its SDNet environment, which includes a high-level language for packet processing. Rather than develop a dedicated environment for network processing, Altera has been enhancing its OpenCL implementation to link multiple kernels in an FPGA.
Intel Aims to Disrupt Market
Although wireless technologies now capture much of the industry’s attention, 2015 will see some exciting developments in wired networking. For data centers, Intel could upset established technologies on two related fronts. The company should reach production with 100GbE active optical cables using its silicon-photonics technology, which employs CWDM with four wavelengths operating at 25Gbps each to deliver 100Gbps over a fiber pair. Although it has yet to announce pricing for these products, we expect they will dramatically reduce the cost of 100GbE optical links in data centers, making 100GbE more attractive relative to 40GbE.
Intel’s more speculative development is a new switch chip designed to fit into the company’s Rack Scale Architecture (RSA). The switch is designed to sit in a server shelf and connect with Xeon processors using PCIe instead of Ethernet, appearing to the server as a NIC. We expect the chip will have both 40GbE and 100GbE uplink ports to connect with the data-center fabric. Intel should sample it to customers in 2015, but a formal announcement may not occur until 2016.
In the enterprise and SMB markets, 2015 will mostly bring more of the same: more 802.11ac chipsets with 4x4 MIMO and MU-MIMO, and the first access points with Wave 2 certification. The first switch systems and access points that offer 2.5GbE/5GbE over copper should also reach the market in 2015.
As work progresses toward a 400G Ethernet draft standard, we expect chip vendors will begin introducing 56Gbps serial PHYs for CAUI-8. Broadcom recently announced sampling of the industry’s first 50Gbps PAM4 PHY, but this chip targets 40GbE/50GbE applications in data centers rather than 400GbE carrier designs. Avago has been another proponent of PAM4 and is likely to develop 56Gbps PHYs in 2015.
Most of the networking chips that sample or reach production in 2015 will remain in 28nm technology. Moving to 20nm increases wafer cost with little power savings. For high-performance devices, many vendors will skip 20nm and instead wait for 16nm FinFET technology, which significantly reduces leakage current. Xilinx expects to tape out its first 16nm FPGA in 1Q15, and Broadcom plans several 16nm tapeouts in 2015; the latter should include at least one Ethernet ASSP. As is the case in other markets, network-chip vendors will continue to use 28nm technology for cost-sensitive products while employing 16nm FinFET where power constrains integration or performance. After the jump to 16nm, these vendors will be laggards no more.
Networking-Chip Events of 2014
OFC saw the introduction of new 100G PHYs from AppliedMicro, Avago, and Broadcom (see NWR 3/24/14). Aquantia announced 10GBase-T PHYs that add 2.5GbE and 5GbE speeds over UTP (see NWR 10/20/14), and it formed the NBase-T Alliance.
Broadcom sampled its Tomahawk 100GbE switch chip (see NWR 10/13/14) as well as its Greyhound 10GbE switch for SMB (see NWR 7/21/14). It also formed the MGBase-T Alliance and sampled a 28nm 10GBase-T PHY that supports the 2.5GbE/5GbE specification (see NWR 12/22/14).
Cavium acquired startup Xpliant and announced its first switch chip (see NWR 10/6/14). Emulex reentered the Ethernet market with new 10GbE and 40GbE controllers (see NWR 2/3/14), whereas QLogic increased its Ethernet investment by acquiring Broadcom’s NetXtreme II line (see NWR 3/17/14). Inphi purchased the majority of Cortina Systems (see NWR 8/18/14).
Intel formally announced its Fortville 40GbE controller in conjunction with the Grantley server launch (see NWR 9/22/14). It also introduced related 40GbE NICs and active optical cables.
Marvell filled out its 28nm Prestera DX line with SMB and carrier-access switch chips (see NWR 5/12/14). Using technology acquired from Solarflare, it sampled a 28nm 10GBase-T PHY (see NWR 4/7/14). The company also sampled its first network-search engine (see NWR 12/1/14) in a bid to compete with Broadcom’s KBPs.
Mellanox advanced InfiniBand with a 100Gbps switch chip (see NWR 7/14/14), and MoSys disclosed its third-generation Bandwidth Engine network memory (see NWR 2/10/14). Netronome sampled its NFP-6xxx processor and released intelligent NICs for 40GbE and 100GbE using the new chip (see NWR 4/28/14).
Qualcomm Atheros announced Wi-Fi chipsets for 802.11ac Wave 2 access points (see NWR 4/28/14). Startup Sckipio sampled its first product, a chipset for G.fast broadband access (see NWR 11/10/14). Texas Instruments addressed wireless backhaul using a new KeyStone II SoC (see NWR 3/24/14), and Vitesse sampled its SparX-IV GbE switch for SMB (see NWR 7/28/14).
Xilinx announced its SDNet specification environment, which offers a high-level language for packet processing (see NWR 5/5/14).