Enabling 800-gigabit optics with physical layer ICs
Broadcom recently announced a family of 800-gigabit physical layer (PHY) chips. The device family is the company’s first 800-gigabit ICs with 100-gigabit input-output (I/O) interfaces.
Source: Broadcom
Moving from 50-gigabit to 100-gigabit-based I/O enables a new generation of 800-gigabit modules aligned with the latest switch chips.
“With the switch chip having 100-gigabit I/Os, PHYs are needed with the same interfaces,” says Machhi Khushrow, senior director of marketing, physical layer products division at Broadcom.
Broadcom’s latest 25.6 terabit-per-second (Tbps) Tomahawk 4 switch chip using 100-gigabit I/O was revealed at the same time.
800-gigabit PHY devices
The portfolio comprises three 800-gigabit PHY ICs. All operate at a symbol rate of 53 gigabaud, use 4-level pulse amplitude modulation (PAM-4) and are implemented in a 7nm CMOS process.
Two devices are optical PHYs: the BCM87800 and the BCM87802. These ICs are used within 800-gigabit optical modules such as the QSFP-DD800 and the OSFP form factors. The difference between the two chips is that the BCM87802 includes an integrated driver.
The third PHY - the BCM87360 - is a retimer IC used on line cards. Whether the chip is needed depends on the line card design and signal-integrity requirements; for example, whether the line card is used within a pizza box or part of a chassis-based platform.
Source: Broadcom
“If it is a higher-density card that is relatively small, it may only need 15 per cent of the ports with retimers,” says Khushrow. “If the line card is larger, where things fan out to longer traces, retimers may be needed for all the ports.”
All three 800-gigabit PHYs have eight 100-gigabit transmit and eight receive channels (8:8, as shown in the top diagram).
Applications
The optical devices support several 800-gigabit module designs that use either silicon photonics, directly modulated lasers (DMLs) or externally-modulated lasers (EMLs).
The 800-gigabit PHYs support the DR8 module (8 single-mode fibres, 500m reach), two 400-gigabit DR4 (4 single-mode fibres, 500m) or two FR4 in a module (each 4 wavelengths on a single-mode fibre, 2km) as well as the SR8, a parallel VCSEL-based design with a reach of 100m over parallel multi-mode fibre.
Timescales
Given the availability of these PHYs and that 800-gigabit modules will soon appear, will the development diminish the 400-gigabit market opportinity?
“This is independent of 400-gigabit [module] deployments,” says Khushrow.
The hyperscalers are deploying different architectures. There are hyperscalers that are only now transitioning to 200-gigabit modules while others are transitioning to 400- gigabit. They will all transition to 800 gigabit, he says: “How and when they transition are all at different points.”
Some of the hyperscalers deploying 400-gigabit modules are looking at 800 gigabit, and their deployment plans are maybe two to three years out. “We don’t expect 800 gigabit to cannibalise 400 gigabit, at least not in the near term,” he says.
Broadcom says 800-gigabit modules to ship in the second half of this year. “It all depends on how the switch infrastructure, line cards and optics become available,” says Khushrow.
Next developments
The landscape for high-speed networking in the data centre is changing and optics is moving closer to the switch chip, whether it is on-board optics or co-packaged optics.
“People are looking at both options,” says Khushrow.”It depends on the architecture of the data centre whether they use on-board optics or co-packaged optics.”
Meanwhile, the OIF is working on a 200-gigabit electrical interface standard.
Co-packaged optics is challenging and the technology has its own issues whereas optical transceivers are easier to use and deploy, says Khushrow.
Current industry thinking is that some form of co-packaged optics will be used with the adevnt of next-generation 51.2-terabit switch chips. But even with such capacity switches, pluggables will continue to be used, he says.
There will still be a need for PHYs, whether for pluggables, co-packaged designs or on the linecard.
“We will continue to provide those on our roadmap,” says Khushrow. “It is just a matter of what the form factor will be, whether it will be a packaged part or a die part.”
Windstream to add ICE6 as it stirs its optical network
Windstream has sent an 800-gigabit optical signal between the US cities of Phoenix and San Diego. The operator used Infinera’s Groove modular chassis fitted with its latest ICE6 infinite capacity engine for the trial.
Infinera reported in March sending an 800-gigabit signal 950km with another operator but this is the first time a customer, Windstream, is openly discussing a trial and the technology.
The bulk of Windstream’s traffic is sent using 100-gigabit wavelengths. Moving to 800-gigabit will reduce its optical transport costs.
Windstream will also be able to cram more digital traffic down its fibre. It sends 12 terabits and that could grow to 40 terabits.
Motivation
Windstream provides residential broadband, business and wholesale services in the US.
“We operate a national footprint for wholesale and enterprise services,” says Art Nichols, vice president of architecture and technology at Windstream. “The optical focus is for wholesale and enterprise.”
Art Nichols
The communications service provider has 160,000 miles of fibre, 3,700 points-of-presence (PoPs) and operates in 840 cities. “We are continually looking to expand that,” says Nichols. “Picking up new PoPs, on-ramps and landing spots to jump onto the long-haul network.”
If Windstream’s traffic is predominantly at 100-gigabit, it also has 200-gigabit wavelengths and introduced recently 400-gigabit signals. In April Windstream and Infinera trialled Gigabit Ethernet (GbE) client-side services using LR8 modules.
Windstream is interested in adopting 800-gigabit wavelengths to reduce transport costs. “To try to draw as much efficiency as you can, using as few lasers as you can, to push down the cost-per-bit,” says Nichols.
The operator is experiencing traffic growth at a 20-30 per cent compound annual growth rate that is eroding its revenue-per-bit.
Weekly traffic has also jumped a further 20 per cent during the COVID-19 pandemic. Video traffic is the main driver, with peak traffic hours starting earlier in the day and continuing into the evenings.
Sending more data on a wavelength reduces cost-per-bit and improves revenue-per-bit figures.
In addition to sending a 800-gigabit signal over 730km, the operator sent a 700-gigabit signal 1,460km. The two spans are representative of Windstream’s network.
“Eight hundred gigabits is an easier multiple - better to fit two 400GbE clients - but 700 gigabits has tons of applications,” says Nichols. “We are predominantly filling 100-gigabit orders today so being able to multiplex them is advantageous.”
Another reason to embrace the new technology is to fulfill wholesale orders in days not months. “The ability to turn around multi-terabit orders from webscale customers,” says Nichols. “That is increasingly expected of us.”
One reason order fulfilment is faster is that the programming interfaces of the equipment are exposed, allowing Windstream to connect its management software. “We instantiate services in a short turnaround,” says Nichols.
ICE6 technology
Infinera’s ICE6 uses a 1.6-terabit photonics integrated circuit (PIC) and its 7nm CMOS FlexCoherent 6 digital signal processor (DSP). The 1.6 terabits is achieved using two wavelengths, each able to carry up to 800 gigabits of traffic.
The ICE6 uses several techniques to achieve its optical performance. One is Nyquist sub-carriers where data is encoded onto several sub-carriers rather than modulating all the data onto a single carrier.
The benefit of sub-carriers is that high data rates are achieved despite the symbol rate of each sub-carrier being much lower. The lower symbol rate means the optical transmission is more tolerant to non-linear channel impairments. Sub-carriers also have sharper edges so can be squeezed together enabling more data in a given slice of spectrum.
Infinera also applies probabilistic constellation shaping to each sub-carrier, enabling just the right amount of data to be placed on each one.
The FlexCoherent 6 DSP also uses soft-decision forward-error correction (SD-FEC) gain sharing. The chip can redistribute processing to the optical channel that needs it the most.
Some of the strength of the stronger signal can be cashed in to strengthen the weaker one, extending its reach or potentially allowing more bits to be sent by enabling a higher modulation scheme to be used.
Windstream cannot quantify the cost-per-bit advantage using the ICE6. “We don’t have finalised pricing,” says Nichols. But he says the latest coherent technology has significantly better spectral efficiency.
Spectral efficiency can be increased in two ways, says Rob Shore, Infinera’s senior vice president of marketing.
One is to increase the modulation scheme and the other is to close the link and maintain the high modulation over longer spans. If the link can’t be closed, lowering the modulation scheme is required which reduces the bits carried and the spectral efficiency.
Windstream will be able to increase capacity per fibre by as much as 70 per cent compared to the earlier generation 400-gigabit coherent technology and by as much as 35 per cent compared to 600-gigabit coherent.
A total of 42.4 terabits can be sent over a fibre using 800-gigabit wavelengths, says Shore, but the symbol rate needs to be reduced to 84 gigabaud shortening the overall reach.
Trial learnings
The rate-reach performance of the ICE6 was central to the trial but what Windstream sought to answer was how the ICE6 would perform across its network.
“We paid really close attention to margins and noise isolation as indicators as to how it would work across the network,” says Nichols. “The exciting thing is that it is extremely applicable.”
Windstream is also upbeat about the technology’s optical performance.
“We have a fair amount of information as to what the latest optical engines are capable of,” says Nichols. “This trial gave us a good view of how the ICE6 performs and it turns out it has advantages in terms of rate-reach performance.”
Ciena, Huawei and Infinera all have 800-gigabit coherent technology. Nokia recently unveiled its PSE-V family of coherent devices that does not implement 800-gigabit wavelengths.
Michael Genovese, a financial analyst at MKM Partners, puts the ICE6 on a par with Ciena’s WaveLogic 5 that is already shipping to over 12 customers.
“We expect 800 gigabit to be a large and long cycle," says Genovese in a recent research note. “We think most of the important internet content providers, telcos and subsea consortia will adopt a duel-vendor strategy, benefitting Ciena and Infinera over time.”
Windstream will adopt Infinera’s ICE6 technology in the first half of 2021. First customers to adopt the ICE6 will be the internet content providers later this year.
COBO: specification work nearing completion
The Consortium for On-board Optics (COBO) is on target to complete its specifications work by the year end. The work will then enter a final approval stage that will take up to a further three months.
On-board optics, also known as mid-board or embedded optics, have been available for years but vendors have so far had to use custom products. The goal of COBO, first announced in March 2015 and backed by such companies as Microsoft, Cisco Systems, Finisar and Intel, is to develop a technology roadmap and common specifications for on-board optics to ensure interoperability.
Brad Booth (pictured), the chair of COBO and principal architect for Microsoft’s Azure Global Networking Services, says that bringing optics inside systems raises a different set of issues compared to pluggable optical modules used on the front panel of equipment. “If you have a requirement for 32 ports on a faceplate, you know mechanically what you can build,” says Booth.
With on-board optics, the focus is less about size considerations and more about the optical design itself and what is needed to make it work. There is also more scope to future-proof the design, something that can not be done so much with pluggable optics, says Booth.
COBO is working on a 400-gigabit optical module based on the 8-by–50 gigabit interface. The focus in recent months has been on defining the electrical connector that will be needed. The group has narrowed down the choice of candidates to two and the final selection will be based on the connector's signal integrity performance and manufacturability. Also being addressed is how two such modules could be placed side-by-side to create an 800-gigabit (16-by–50 gigabit) design.
COBO’s 400-gigabit on-board optics will support multi-mode and single-mode fibre variants. “When we do a comparison with what the pluggable people are pushing, there are a lot of pluggables that won’t be able to handle the power envelope,” says Booth.
There is no revolutionary change that goes on with technology, it all has to be evolutionary
On-board optics differs from a pluggable module in that the optics and electronics are not confined within a mechanical enclosure and therefore power dissipation is less of an design issue. But by supporting different fibre requirements and reaches new design issues arise. For example, when building a 16-by–50 gigabit design, the footprint is doubled and COBO is looking to eliminate the gap between the two such that a module can be plugged in that is either 8- or 16-lanes wide.
COBO is also being approached about supporting other requirements such as coherent optics for long-distance transmission. A Coherent Working Group has been formed and will meet for the first time in December in Santa Barbara, California. Using on-board optics for coherent avoids the power constraint issues associated with using a caged pluggable module.
On-board optics versus co-packaging
On-board optics is seen as the next step in the evolution of optics as it moves from the faceplate onto the board, closer to the ASIC. There is only so many modules that can fit on a faceplate. The power consumption also raises as the data rate of a pluggable modules increases, as does the power associated with driving faster electrical traces across the board.
Using on-board optics shortens the trace lengths by placing the optics closer to the chip. The board input-output capacity that can be supported also increases as it is fibres not pluggable optics that reside on the front panel. Ultimately, however, designers are already exploring the combining of optics and the chip using a system-in-package design, also known as 2.5D or 3D chip packaging.
Booth says discussions have already taken place between COBO members about co-packaged optics. But he does not expect system vendors to stay with pluggable optics and migrate directly to co-packaging thereby ignoring the on-board optics stage.
“There is no revolutionary change that goes on with technology, it all has to be evolutionary,” says Booth, who sees on-board optics as the next needed transition after pluggables. “You have to have some pathway to learn and discover, and figure out the pain points,” he says. “We are going to learn a lot when we start the deployment of COBO-based modules.”
Booth also sees on-board optics as the next step in terms of flexibility.
When pluggable modules were first introduced they were promoted as allowing switch vendors to support different fibre and copper interfaces on their platforms. The requirements of the cloud providers has changed that broad thinking, he says: “We don’t need that same level of flexibility but there is still a need for suporting different styles of optical interfaces on a switch.”
There are not a lot of other modules that can do 600 gigabit but guess what? COBO can
For example, one data centre operator may favour a parallel fibre solution based on the 100-gigabit PSM4 module while another may want a 100-gigabit wavelength-division multiplexing (WDM) solution and use the CWDM4 module. “This [parallel lane versus WDM] is something embedded optics can cater for,” says Booth.
Moving to a co-packaged design offers no such flexibility. What can a data centre manager do when deciding to change from parallel single-mode optics to wavelength-division multiplexing when the optics is already co-packaged with the chip? “Also how do I deal with an optics failure? Do I have to replace the whole switch silicon?” says Booth. We may be getting to the point where we can embed optics with silicon but what is needed is a lot more work, a lot more consideration and a lot more time, says Booth.
Status
COBO members are busy working on the 400-gigabit embedded module, and by extension the 800-gigabit design. There is also ongoing work as to how to support technologies such as the OIF’s FlexEthernet. Coherent designs will soon support rates such as 600-gigabit using a symbol rate of 64 gigabaud and advanced modulation. “There are not a lot of other modules that can do 600 gigabits but guess what? COBO can,” says Booth.
The good thing is that whether it is coherent, Ethernet or other technologies, all the members are sitting in the same room, says Booth: “It doesn’t matter which market gets there first, we are going to have to figure it out.”
Story updated on October 27th regarding the connector selection and the Coherent Working Group.