Broadcom samples the first 51.2-terabit switch chip
- Broadcom’s Tomahawk 5 marks the era of the 51.2-terabit switch chip
- The 5nm CMOS device consumes less than 500W
- The Tomahawk 5 uses 512, 100-gigabit PAM-4 (4-level pulse amplitude modulation) serdes (serialisers-deserialisers)
- Broadcom will offer a co-packaged version combining the chip with eight 6.4 terabit-per-second (Tbps) optical engines
Part 1: Broadcom’s Tomahawk 5
Broadcom is sampling the world’s first 51.2-terabit switch chip.
With the Tomahawk 5, Broadcom continues to double switch silicon capacity every 24 months; Broadcom launched the first 3.2-terabit Tomahawk was launched in September 2014.
“Broadcom is once again first to market at 51.2Tbps,” says Bob Wheeler, principal analyst at Wheeler’s Network. “It continues to execute, while competitors have struggled to deliver multiple generations in a timely manner.”
Tomahawk family
Hyperscalers use the Tomahawk switch chip family in their data centres.
Broadcom launched the 25.6-terabit Tomahawk 4 in December 2019. The chip uses 512 serdes, but these are 50-gigabit PAM-4. At the time, 50-gigabit PAM-4 matched the optical modules’ 8-channel input-output (I/O).
Certain hyperscalers wanted to wait for 400-gigabit optical modules using four 100-gigabit PAM-4 electrical channels, so, in late 2020, Broadcom launched the Tomahawk4-100G switch chip, which employs 256, 100-gigabit PAM-4 serdes.
Tomahawk 5 doubles the 100-gigabit PAM-4 serdes to 512. However, given that 200-gigabit electrical interfaces are several years off, Broadcom is unlikely to launch a second-generation Tomahawk 5 with 256, 200-gigabit PAM-4 serdes.
Switch ICs
Broadcom has three switch chip families: Trident, Jericho and the Tomahawk.
The three switch chip families are needed since no one switch chip architecture can meet all the markets’ requirements.
With its programable pipeline, Trident targets enterprises, while Jericho targets service providers.
According to Peter Del Vecchio, Broadcom’s product manager for the Tomahawk and Trident lines, there is some crossover. For example, certain hyperscalers favour the Trident’s programmable pipeline for their top-of-rack switches, which interface to the higher-capacity Tomahawk switches chips at the aggregation layer.
Monolithic design
The Tomahawk 5 continues Broadcom’s approach of using a monolithic die design.
“It [the Tomahawk5] is not reticule-limited, and going to [the smaller] 5nm [CMOS process] helps,” says Del Vecchio.
The alternative approach – a die and chiplets – adds overall latency and consumes more power, given the die and chiplets must be interfaced. Power consumption and signal delay also rise whether a high-speed serial or a slower, wider parallel bus is used to interface the two.
Equally, such a disaggregated design requires an interposer on which the two die types sit, adding cost.
Chip features
Broadcom says the capacity of its switch chips has increased 80x in the last 12 years; in 2010, Broadcom launched the 640-gigabit Trident.
Broadcom has also improved energy efficiency by 20x during the same period.
“Delivering less than 1W per 100Gbps is pretty astounding given the diminishing benefits of moving from a 7nm to a 5nm process technology,” says Wheeler.
“In general, we have achieved a 30 per cent plus power savings between Tomahawk generations in terms of Watts-per-gigabit,” says Del Vecchio.
These power savings are not just from advances in CMOS process technology but also architectural improvements, custom physical IP designed for switch silicon and physical design expertise.
“We create six to eight switch chips every year, so we’ve gotten very good at optimising for power,” says Del Vecchio
The latest switch IC also adds features to support artificial intelligence (AI)/ machine learning, an increasingly important hyperscaler workload.
AI/ machine learning traffic flows have a small number of massive ‘elephant’ flows alongside ‘mice’ flows. The switch chip adds elephant flow load balancing to tackle congestion that can arise when the two flow classes mix.
“The problem with AI workloads is that the flows are relatively static so that traditional hash-based load balancing will send them over the same links,” says Wheeler. “Broadcom has added dynamic balancing that accounts for link utilisation to distribute better these elephant flows.”
The Tomahawk 5 also provides more telemetry information so data centre operators can better see and tackle overall traffic congestion.
The chip has added virtualisation support, including improved security of workloads in a massively shared infrastructure.
Del Vecchio says that with emerging 800-gigabit optical modules and 1.6 terabit ones on the horizon, the Tomahawk 5 is designed to handle multiples of 400 Gigabit Ethernet (GbE) and will support 800-gigabit optical modules.
The chip’s 100-gigabit physical layer interfaces are combined to form 800 gigabit (8 by 100 gigabit), which is fed to the MAC, packet processing pipeline and the Memory Management Unit to create a logical 800-gigabit port. “After the MAC, it’s one flow, not at 400 gigabits but now at 800 gigabits,” says Del Vecchio.
Market research firm, Dell’Oro, says that 400GbE accounts for 15 per cent of port revenues and that by 2026 it will rise to 57 per cent.
Broadcom also cites independent lab test data showing that its support for RDMA over Converged Ethernet (RoCE) matches the performance of Infiniband.
“We’re attempting to correct the misconception promoted by competition that Infiniband is needed to provide good performance for AI/ machine learning workloads,” says Del Vecchio. The tests used previous generation silicon, not the Tomahawk 5.
“We’re saying this now since machine learning workloads are becoming increasingly common in hyperscale data centres,” says Del Vecchio.
As for the chip’s serdes, they can drive 4m of direct attached copper cabling, with sufficient reach to connect equipment within a rack or between two adjacent racks.
Software support
Broadcom offers a software development kit (SDK) to create applications. The same SDK is common to all three of its switch chip families.
Broadcom also supports the Switch Abstraction Interface (SAI). This standards-based programming interface sits on top of the SDK, allowing the programming of switches independent of the silicon provider.
Broadcom says some customers prefer to use its custom SDK. It can take time for changes to filter up, and a customer may want something undertaken that Broadcom can develop quickly using its SDK.
System benefits
Doubling the switch chip’s capacity every 24 months delivers system benefits.That is because implementing a 51.2-terabit switch using the current generation Tomahawk 4 requires six such devices.
Now a single 2-rack-unit (2RU) Tomahawk 5 switch chip can support 64 by 800-gigabit, 128 by 400-gigabit and 256 by 200-gigabit modules.
These switch boxes are air-cooled, says Broadcom.
Co-packaged optics
In early 2021 at a J.P Morgan analyst event, Broadcom revealed its co-packaged optics roadmap that highlighted Humboldt, a 25.6-terabit switch chip co-packaged with optics, and Bailly, a 51.2-terabit fully co-packaged optics design.
At OFC 2022, Broadcom demonstrated a 25.6Tbps switch that sent half of the traffic using optical engines.
Also shown was a mock-up of Bailly, a 51.2 terabit switch chip co-packaged with eight optical engines, each at 6.4Tbps.
Broadcom will offer customers a fully co-packaged optics Tomahawk 5 design but has not yet given a date.
Broadcom can also support a customer if they want tailored connectivity with, say, 3/4 of the Tomahawk 5 interfaces using optical engines and the remainder using electrical interfaces to front panel optics.
North American operators in an optical spending rethink
Optical transport spending by the North American operators dropped 13 percent year-on-year in the third quarter of 2014, according to market research firm Dell'Oro Group.
Operators are rethinking the optical vendors they buy equipment from as they consider their future networks. "Software-defined networking (SDN) and Network Functions Virtualisation (NFV) - all the futuristic next network developments, operators are considering what that entails," says Jimmy Yu, vice president of optical transport research at Dell’Oro. "Those decisions have pushed out spending."
NFV will not impact optical transport directly, says Yu, and could even benefit it with the greater signalling to central locations that it will generate. But software-defined networks will require Transport SDN. "You [as an operator] have to decide which vendors are going to commit to it [Transport SDN]," says Yu.
SDN and NFV - all the futuristic next network developments, operators are considering what that entails. Those decisions have pushed out spending
The result is that the North American tier-one operators reduced their spending in the third quarter 2014. Yu highlights AT&T which during 2013 through to mid 2014 undertook robust spending. "What we saw growing [in that period] was WDM metro equipment, and it is that spending that has dropped off in the third quarter," says Yu. For equipment vendors Ciena and Fujitsu that are part of AT&T's Domain 2.0 supplier programme, the Q3 reduced spending is unwelcome news. But Yu expects North American optical transport spending in 2015 to exceed 2014's. This, despite AT&T announcing that its capital expenditure in 2015 will dip to US $18 billion from $21 billion in 2014 now that its Project VIP network investment has peaked.
But Yu says AT&T has other developments that will require spending. "Even though AT&T may reduce spending on Project VIP, it is purchasing DirecTV and the Mexican mobile carrier, lusacell," he says. "That type of stuff needs network integration." AT&T has also committed to passing two million homes with fibre once it acquires DirecTV.
Verizon is another potential reason for 2015 optical transport growth in North America. It has a request-for-proposal for metro DWDM equipment and the only issue is when the operator will start awarding contracts. Meanwhile, each year the large internet content providers grow their optical transport spending.
Dell'Oro expects 2014 global optical transport spending to be flat, with 2015 forecast to experience three percent growth
Asia Pacific remains one of the brighter regions for optical transport in 2014. "Partly this is because China is buying a lot of DWDM long-haul equipment, with China Mobile being one of the biggest buyers of 100 Gig," says Yu. EMEA continues to under-perform and Yu expects optical transport spending to decline in 2014. "But there seems to be a lot of activity and it's just a question of when that activity turns into revenue," he says.
Dell'Oro expects 2014 global optical transport spending to be flat compared to 2013, with 2015 forecast to experience three percent growth. "That growth is dependent on Europe starting to improve," says Yu.
One area driving optical transport growth that Yu highlights is interconnected data centres. "Whether enterprises or large companies interconnecting their data centres, internet content providers distributing their networks as they add more data centres, or telecom operators wanting to jump on the bandwagon and build their own data centres to offer services; that is one of the more interesting developments," he says.
Optical transport to grow at a 10% CAGR through 2017
- Global optical transport market to reach US $13bn in 2017
- 100 Gigabit to grow at a 75% CAGR
"I won't be surprised if it [100 Gig] grows even faster"
Jimmy Yu, Dell'Oro Group
The Dell'Oro Group forecasts that the global optical transport market will grow to US $13 billion in 2017, equating to a 10-percent compound annual growth rate (CAGR).
In 2012 SONET/SDH sales declined by over 20 percent, greater than Dell'Oro expected, while wavelength-division multiplexing (WDM) equipment sales held their own.
Regions
Dell'Oro expects optical transport growth across all the main regions, with no one region dominating. The market research company does foresee greater growth in Europe given the prolonged underspend of recent years.
European operators are planning broadband access investment such as fibre-to-the-cabinet/ VDSL vectoring as well as fibre-to-the-home. "That will drive demand for backhaul bandwidth and that is where WDM fits in well," says Jimmy Yu, vice president, microwave transmission, mobile backhaul and optical transport at Dell'Oro.
Technologies
Forty and 100 Gigabit optical transport will be the main WDM growth areas through 2017. Yu expects 40 Gigabit demand to grow over the forecast period even if the growth rate will taper off due to demand for 100 Gigabit.
The 100 Gigabit market continues to exceed Dell'Oro's forecasted growth. The market research company predicts 100-Gbps wavelength shipments to grow at a 75 percent CAGR over the next five years, accounting for 60 percent of the WDM capacity shipments by 2017. "I won't be surprised if it [100 Gig] grows even faster," says Yu.
"A lot of people wonder why have 40 Gig when there is 100 Gig? But that granularity does help service providers; having 40 Gig and 100 Gig rather than going straight from 10 Gig to 100 Gig," says Yu. The 100 Gig sales span metro and long-haul networks with the latter generating greater revenue due to the systems being pricier. "Forty Gigabit sales were predominantly long haul originally but we are seeing a good chunk of growth in metro as well," says Yu.
The current forecast does not include 400Gbps optical transport sales though Yu does expect sales to start in 2016.
Dell'Oro is seeing sales of 100 Gigabit direct detection but says it will remain a niche market. "We are talking tens of [shipped] units a quarter," says Yu.
There are applications where customers will need links of 80km or several hundred kilometers and will want the lowest cost solution, says Yu: "There is a market for direct detection; it will not be a significant driver for 100 Gig but it will be there."
60-second interview with .... Dell'Oro's Jimmy Yu
"For the year, it is going to be a fivefold growth rate [for 100 Gig transport]."
Jimmy Yu, Dell'Oro
Q: That fact that the market is down 5 percent on a year ago. Why is this?
A: There are a few factors. First, the macro-economy in Europe continues to get worse; that causes a slowdown.
A second factor is that in North America there was a decline in the second quarter, which is pretty unusual. Part of it, we think, might be that operators have caught up with a lot of the spending to increase broadband, after adding [to the network] for a couple of good years.
The third issue is that the China market has had a really slow start. And while there has been talk about the Chinese market softening, it seems that the CapEx [capital expenditure] is there for a strong second half.
What categories does Dell'Oro include when it talks about optical transport?
There are two main pieces: WDM [wavelength division multiplexing], both metro and long haul, and the multi-service multiplexer used for aggregation. The third piece, which is really small, is optical switching - optical cross-connect used in the core and lately more so in the metro.
According to Dell'Oro, wavelength division multiplexing was up 5 percent in the first half of 2012 compared to the same period a year ago, due to demand for 40 Gig and 100 Gig. What is happening in these two markets?
At 100 Gig we are at an inflection point where demand growth rates are really high. We've got a doubling in demand and shipments quarter-on-quarter [in the second quarter]. For the year, it is going to be a fivefold growth rate.
Also the 40 Gig is still growing. It has been around for a few years so its growth rate is not as strong [as 100 Gig transport] but it is still a significant part of the market.
Has the market settled on particular modulation scheme, especially at 40 Gig?
For 100 Gig the majority [deployed] is coherent. There is one company at least, ADVA Optical Networking, which is coming out with its direct-detection scheme for 100 Gig. This has now been shipping for one quarter. There is a market for the price point and the lower-span link of direct-detection.
For 40 Gig there is still a mix of modulations. Vendors coming out with 100 Gig coherent are also coming out with 40 Gig coherent options. So coherent at 40 Gig is now approaching half of the total market and is happening pretty quickly.
As for [40 Gig] DQPSK [differential quadrature phase-shift keying] modulation, it is probably a little bit more than DPSK [differential phase-shift keying].
You also report a rise in the adoption of optical packet products and that it contributed close to one-third of the optical market revenues in the first half 2012. Why is that?
The optical packet platform is a wider definition than just packet optical transport systems (P-OTS).
One reason why optical packet is growing is that with traditional P-OTS, you have cross-connect and switching capabilities in a WDM system so as you go to higher 40 and 100 Gig wavelengths you want some bandwidth management in that system.
Another thing is that people are trying to make the aggregation layer - the traditional SONET/SDH - more Ethernet friendly and MPLS-TP [multiprotocol label switching, transport profile] is gaining traction.
Combined, we are seeing this optical packet market has grown 12 percent year-on-year in the second quarter whereas the overall market has declined.
Dell'Oro said Huawei has 20 percent market share, which other vendors have double-digit market share?
Besides Huawei, the other vendors with double-digit percentage for the quarter - in order - are ZTE, Alcatel-Lucent and Ciena.
Did you see anything in this latest study that was surprising?
There was nothing in this quarter but I saw it last quarter. The legacy equipment – traditional SONET/SDH – is declining. Most of the market decline for optical is in legacy.
SONET/SDH sales in the second quarter of 2012 declined by 20 percent year-on-year. It is finally happening: the market is shifting away from SONET/SDH.