Broadcom’s 14.4-terabit Jericho2c+ router chip

The inexorable growth of IP traffic is being driven by ever more powerful devices being connected to the network and greater numbers of machines talking to each other.

In turn, Covid-19 has contributed its own traffic spike: AT&T reported that in September its core network traffic was 20 per cent up compared to March’s figures.

Jericho2c+ architecture. Source: Broadcom

The growth means that each new generation of router platform must at least double the traffic throughput while keeping the power consumption fixed.

This is a considerable challenge but one that the router chip designers continue to meet.

Jericho2c+

Broadcom announced recently its 14.4-terabit Jericho2c+ chip that increases router capacity threefold compared to the Jericho2c.

The Jericho2c+ is Broadcom’s third Jericho2 chip in just two years, samples of which are now with customers. Broadcom says the device should be qualified by year-end.  

“The surprise is the generational cadence Broadcom is maintaining for the Jericho line,” says Bob Wheeler, vice president of The Linley Group and principal analyst for networking. “They remain committed to leading in chassis-based switch and router designs.”

Jericho2, a 9.6-terabit chip aimed at data centre router applications, was announced in 2018.  The 4.8-terabit Jericho2c followed a year later, a derivative targeting the service-provider market.

Now, the Jericho2c+ triples throughput using a 7nm CMOS process instead of a 16nm used for the Jericho2c and has enabled Broadcom to boost the on-chip cores’ clock frequency.

“From an architecture standpoint, Jericho2c+ is an evolution of Jericho2, not Jericho2c,” explains Wheeler. “But yes, for Jericho2c customers, J2c+ does triple throughput.”

Router trends

Oozie Parizer, senior marketing director, Broadcom, says the company has played an important role in the telecom operators moving from router platforms using custom ASICs to platforms using merchant silicon.

“Recently we see more of this transitioning, not just to merchant silicon but also to white boxes,” says Parizer. This transition applies to core and edge routers.

“It has been possible because of the availability of merchant silicon to all, enabling a different price structure,” says Parizer. “All the OEMs are designing platforms; at the same time, some are maintaining their own [router] ASIC designs.” Router vendors developing ASICs include Cisco Systems, Juniper Networks and Nokia.

Telecom operators are keen for the established router vendors to use merchant silicon, says Parizer: “The carriers are buying into this story because they see what happened in data centers 10 years ago [with switching silicon].”

Wheeler sees a role for both custom ASICs and merchant silicon.

“The leading router OEMs remain committed to ASICs for their core routers; these are “halo” products meant to lead in features rather than being cost efficient,” he says. “Meanwhile, many of these same OEMs use Jericho for higher volume routers and transport platforms.”

Jericho architecture

The 14.4 terabit-per-second (Tbps) Jericho2c+ splits its interface capacity with 7.2Tbps facing the front panel and 7.2Tbps to Broadcom’s accompanying fabric chip, dubbed the Ramon. The Ramon is a cell-based switch chip that features 192, 50-gigabit serialiser/ deserialisers (serdes).

The fabric interface of the Jericho2c+ is indeed 9.6Tbps, the extra 2.4Tbps used for redundancy. “This is a carrier-grade system; they care about robustness,” says Parizer.

Should a line card fail, traffic is routed to a spare card to ensure continued traffic flow. The spare card may cover three working cards, but systems also use higher ratios: a spare card for every five, seven or even nine cards.

The Jericho2c+ has added on-chip security. “We have integrated MACSec and IPSec at line rate for all the network interfaces of the Jericho2c+,” says Gadi Singer, principal product manager at Broadcom.

Hyperscalers and service providers want to secure communications beyond their premises and integrating security hardware on each port reduces system cost and overall power consumption, says Singer.

Implementing a 14.4Tbps router with security and redundancy using two Jericho2C+ devices halves the power of a similar capacity platform using the Jericho2c. Such a design requires 21 chips: three Jericho2 ICs and the rest security ICs.

The Jericho2c+ doubles the number of meters while supporting more queues than previous Jericho devices. Meters and queues are used to enforce quality-of-service schemes to prioritise traffic flows and services.

“We wanted to ensure that even for the most demanding cases, we have sufficient meters and queues,” says Parizer.

The chip also features two stacked high-bandwidth memory (HBM) for packet buffering; 8 gigabytes of high-speed store in total.

Programmable pipeline

The Jericho devices feature a programmable pipeline.

A set of known features can always be implemented more efficiently with dedicated hardware; extra logic is needed to make the pipeline programmable. But a programmable pipeline can support different use cases. “You future-proof it for something that might come up after you design or even deploy the chip,” says Parizer.

The Jericho2c+ includes parsers and special function units for both the pipeline’s input and output paths.

There are also shared resources used by both pipeline paths: a modular database (MDB), a programmable element matrix (PEM), and the knowledge-based processor (KBP) external to the device.

The MDB uses one of several packet-processing database options that define the device’s profile, and which resides in a centralised memory pool.

The PEM is a programmable hardware block, effectively a small embedded FPGA. “It can solve futuristic scenarios not mapped into the device,” says Singer.

Meanwhile, the pipeline uses the KBP chip to extend its forwarding, counter and access-list capabilities. Broadcom gained the KBP after acquiring NetLogic in 2011.

Line cards and systems

Broadcom cites two line card configurations.

One, for service providers, is a 12.8-terabit N+1 redundancy fabric card that uses two Jericho devices and supports 32x400 Gigabit Ethernet (GbE) interfaces.

“It uses this knowledge-based processor to extend the forwarding scale, counters and the access list for edge and core router scenarios,” says Singer.

The second card, for the service provider edge, data centre interconnect and metro, uses two Jericho 2c+ devices connected back-to-back, acting as a single packet processing and traffic manager element.

The 14.4-terabit line card fits in a 1-rack-unit box and supports 36x400GbE ports.

Distributed disaggregated chassis

The Jericho2 is used for the distributed disaggregated chassis, a white box design that started when AT&T submitted its specifications to the Open Compute Project (OCP) over a year ago.

Last December, Drivenets announced a partnership with Broadcom and the original design manufacturers (ODMs) Ufispace, EdgeCore Networks and Delta to support the design. Since then, other third-party network operating system (NOS) players have backed the design.

Broadcom says that without AT&T, the white-box router would not have happened. “This is our entry into the core router market,” says Parizer.

And significantly, AT&T announced that part of its network backbone traffic is using such disaggregated core routing hardware running Drivenets’ software.

Jericho3 and Ramon2

Broadcom has started work on its next-generation Jericho and Ramon chips. The 5nm CMOS devices will use 100-gigabit serdes to double the throughput. The next Jericho device will also support 800-gigabit interfaces and 800-gigabit optics.

“We are investing a lot in design-for-power because that is one of the key things that enable our design wins,” says Parizer. “And we will update our pipeline and capabilities to address new requirements and integrate more into the device.”

Moore’s law

Parizer says Moore’s law has yet to slow down. “It hasn’t happened yet, look at the cadence of [foundry] TSMC moving from [CMOS] process to process: 7nm, 5nm, 3nm,” he says. “I can see two processes ahead; I’m relaxed for now.”

There are also chip industry developments such as die stacking where innovation is taking place.

“It is moving more and more of the line card and packaging its functionality into a device,” says Parizer. “This helps address some of the challenges.”