Fulcrum's Alta switch chips add programmable pipeline to keep pace with standards
Part 2: Ethernet switch chips
Fulcrum Microsystems has announced its latest FocalPoint chip family of Ethernet switches. The Alta FM6000 series family supports up to 72 10-Gigabit ports and can process over one billion packets a second.
“Instead of every top-of-rack switch having a CPU subsystem, you could put all the horsepower into a set of server blades”
Gary Lee, Fulcrum Microsystems
The company’s Alta FM6000 series is its third generation of FocalPoint Ethernet switches. Based on a 65nm CMOS process, the Alta switch architecture includes a programmable packet-processing pipeline that can support emerging standards for data centre networking. These include Data Center Bridging (DCB), Transparent Interconnection of Lots of Links (TRILL), and two server virtualisation protocols: the IEEE 802.1Qbg Edge Virtual Bridging and the IEEE 802.1Qbh Bridge Port Extension.
Why is this important?
Data centre networking is undergoing a period of upheaval due to server virtualisation. Data centre operators must cope with the changing nature of traffic flows, as the predominant traffic becomes server-to-server (east-west traffic) rather than between the servers and end users (north-south).
In turn, IT staff want to consolidate the multiple networks they must manage - for LAN, storage and high-performance computing - onto a single network based on DCB.
They also want to reduce the number of switch platforms they must manage. This is leading switch vendors to develop larger, flatter architectures; instead of the traditional three tiers of switches, vendors are developing sleeker two-tier and even a single-layer, logical switch architecture that spans the data centre.
“There are people out there that have enterprise gear where their data centre connection has to go through the access, aggregation and core [switches],” says Gary Lee, director of product marketing at Fulcrum Microsystems. “They may not want to swap out that gear so they are going to continue to have three tiers even if it is not that efficient.”
But other customers such as large cloud computing players do not require such a switch hierarchy and its associated software complexity, says Lee: “They are the ones that are driving to a ‘lean core’, made up of top-of-rack and the end-of-row switch that acts as a core switch.”
Switch vendor Voltaire, a customer of Fulcrum’s ICs, uses such an arrangement to create 288 10-Gigabit Ethernet ports based on two tiers of 24-port switch chips. With the latest 72-port FM6000 series, a two-tier architecture with over 2,500 10-Gigabit ports becomes possible. “The software can treat the entire structure of chips as a single large virtual switch,” says Lee.
Alta architecture
Fulcrum's Alta FM6000 series architecture. Source: Fulcrum MicrosystemsFulcrum’s FocalPoint 6000 series comprises nine devices with capacities from 160 to 720 Gigabit-per-second (Gbps). The Alta chip architecture has three main components:
- Input-output ports
- RapidArray shared memory and
- the FlexPipe array pipeline.
Like Fulcrum’s second generation Bali architecture, the 6000 series has 96 serial/ deserialiser (serdes) ports but these have been upgraded from 3.125Gbps to 10Gbps.“We have very flexible port logic,” says Lee. “We can group four serdes to create a XAUI [10 Gigabit Ethernet] port or create an IEEE 40 Gigabit Ethernet port.”
RapidArray is a single shared memory which can be written to and read from at full line rate from all the ports simultaneous, says Fulcrum. Each memory output port has a set of eight class-of-service queues, while the shared memory can be partitioned to separate storage traffic from data traffic.
“The shared memory design is where we get the low latency, and good multicast performance which people in the broadband access market like for video distribution,” says Lee.
The architecture’s third main functional block is the FlexPipe array pipeline. The pipeline, new to Alta, is what enables up to a billion 64 byte packets to be processed each second. The packet-processing pipeline combines look-up tables and microcode-programmable functional blocks that process a packet’s fields. Being able to program the array pipeline means the device can accommodate standards’ changes as they evolve, as well as switch vendors’ proprietary protocols.
OpenFlow
The FocalPoint software development kit that comes with the chips supports OpenFlow. OpenFlow is an academic initiative that allows networking protocols to be explored using existing hardware but it is of growing interest to data centre operators.
“It creates an industry-standard application programming interface (API) to the switches,” explains Lee. It would allow the likes of a Google or a Yahoo! to switch vendors’ switch platforms as long as both vendors supported OpenFlow.
OpenFlow also establishes the idea of a central controller that would run on a server to configure the network. “Instead of every top-of-rack switch having a CPU subsystem, you could put all the horsepower into a set of server blades,” says Lee. This promises to lower the cost of switches and more importantly enable operators to unshackle themselves from switch vendors’ software.
Lee points out that OpenFlow is still in its infancy. But Fulcrum has added an ‘OpenFlow vSwitch stub’ to its software that translates between OpenFlow APIs and FocalPoint APIs.
What next?
Fulcrum says it continues to monitor the various evolving standards such as DCB, TRILL and the virtualisation work. Fulcrum is also getting requests to support latency measurement on its chips using techniques such as synchronous Ethernet to ensure service level agreements are met.
As for future FocalPoint designs, these will have greater throughput, with larger table sizes, packet buffers and higher-speed 100 Gigabit Ethernet interfaces.
Meanwhile all nine FM6000 series’ chip members will be available from the second quarter, 2011.
Click here for Part 1: Single-layer switch architectures
Click here for Part 3: Networking developments
Is Broadcom’s chip powering Juniper’s Stratus?
Part 1: Single-layer switch architectures
Juniper Networks’ Stratus switch architecture, designed for next-generation data centres, is several months away from trials. First detailed in 2009, Stratus is being engineered as a single-layer switch with an architecture that will scale to support tens of thousands of 10 Gigabit-per-second (Gbps) ports.
Stratus will be in customer trials in early 2011.
Andy Ingram, Juniper Networks
Data centres use a switch hierarchy, made up of three layers commonly. Multiple servers are connected to access switches, such as top-of-rack designs, which are connected to aggregation switches whose role is to funnel traffic to large, core data centre switches.
Moving to a single-layer design promises several advantages. Not only does a single-layer architecture reduce the overall number of managed platforms, bringing capital and operational expense savings, it also reduces switch latency.
Broadcom’s IC for Stratus?
The Stratus architecture has yet to be detailed by Juniper. But the company has said that the design will be based on a 64x10Gbps ASIC building block dubbed a path-forwarding engine (PFE).
“The building block – the PFE – that can have that kind of density (64x10Gbps) gives us the ability to build out the network fabric in a very economical way,” says Andy Ingram, vice president of product marketing and business development of the fabric and switching technologies business group at Juniper Networks.
Stratus is being designed to provide any-to-any connectivity and operate at wire speed. “You have a very dense, very high-cross-sectional bandwidth fabric,” says Ingram. “The only way to make it economical is to use dense ASICs.”
Broadcom’s latest StrataXGS Ethernet switch family - the BCM56840 series - comprises three devices to date, the largest of which - the BCM56845 – also has 64x10Gbps ports.
Juniper will not disclose whether it is using its own ASIC or a third-party device for Stratus.
Broadcom, however, has said that its BCM56840 series is being used by vendors developing flat, single-layer switch architectures. “Anyone using merchant Ethernet switching silicon to build a single-stage environment is probably using our technology,” says Nick Ilyadis, chief technical officer for Broadcom’s infrastructure networking group.
Stratus will be in customer trials in early 2011. “In a lot less than 6 months”, says Ingram. “We have some customers that have some very difficult networking challenges that are signed up to be our early field trials and we will work with them extensively.”
The timeline aligns with Broadcom’s claim that samples of the BCM56840 ICs have been available for months and will be in production by year-end.
According to Broadcom, only a handful of switch vendors have the resources to design such a complex switch ASIC and also expect to recoup their investment. Moreover, a switch vendor using Broadcom's IC has plenty of scope to differentiate their design using software, and even FPGA hardware if needed. It is software that brings out the many features of the BCM56845, says Broadcom.
The BCM56845
Broadcom’s BCM56840 ICs share a common feature set but differ in their switching capacity. The largest, the BCM56845, has a switching capacity of 640Gbps. The device’s 64x10 Gigabit Ethernet (GbE) ports can also be configured as 16x40 GbE ports.
The BCM56845 supports data center bridging (DCB), the Ethernet protocol enhancement that enables lossless transmission of storage and high-performance computing traffic. It also supports the Fibre Channel over Ethernet (FCoE) protocol that frames Fibre Channel storage traffic over DCB-enhanced Ethernet.
Besides DCB Ethernet, the series switch includes layer 3 packet processing and routeing. There is also a multi-stage content-aware engine that allows higher layer, more complex packet inspection (layer 4 to 7 of the Open Systems Interconnection model) and policy management.
The content-aware functional block can also be used for packet cut-through; a technique to reduce switch latency by inspecting header information and forwarding all the while the packet’s payload is arriving. Broadcom says the switch’s latency is less than one microsecond.
Most importantly, the BCM56845 addresses the move to a flatter switching architecture in the data centre.
It supports the Transparent Interconnection of Lots of Links (TRILL) standard ratified by the Internet Engineering Task Force (IETF) in July. Ethernet uses a spanning tree technique to avoid the creation of loops within a network. However the spanning tree becomes unwieldy as the Ethernet network size grows and works only at the expense of halving the available networking bandwidth. TRILL is designed to allow much larger Ethernet networks while using all available bandwidth.
Broadcom has its own protocol know as HiGig that adds tags to packets. Using HiGig, a very large logical switch can be created and managed, made up of multiple interconnected switches. Any port of the IC can be configured as a HiGig port.
So has Broadcom’s BCM56845 been chosen by Juniper Networks for Stratus? “I really can’t comment on which customers are using this,” says Ilyadis.
Click here for Part 2: Ethernet switch chips
Click here for Part 3: Networking developments