If electrical interfaces are becoming an impediment, is co-packaged optics the answer? Broadcom certainly thinks so.
One reason for the growing interest in co-packaged optics is the input-output (I/O) demands of switch chips. If the packet processing capacity of such chips is doubling every two years, their I/O must double too.
Alexis BjörlinBut repeatedly doubling the data throughput of a switch chip is challenging.
Each new generation of switch chip must either double the number of serialiser-deserialiser (serdes) circuits or double their speed.
A higher serdes count - the latest 25.6-terabit switch ICs have 256, 100 gigabit-per-second serdes - requires more silicon area while both approaches - a higher count and higher speed - increase the chip's power consumption.
Faster electrical interfaces also complicate the system design since moving the data between the chip and the optical modules on the switch's front panel becomes more challenging.
Brad Booth, director, leading-edge architecture pathfinding team in Azure hardware systems and infrastructure at Microsoft, and president of the Consortium of On-Board Optics (COBO), describes the issue of electrical interfaces as a massive impediment.
"How do we move forward with what we need to keep the bandwidth-density growing on these devices?" says Booth. "That is not going to be solved with electronics anymore."
This explains why co-packaged optics is getting so much attention and why standardisation efforts have been started by COBO and by the OIF.
Co-packaged optics
Leading switch-chip specialist Broadcom says the relentless growth in switch-chip capacity is due to the hyperscalers' insatiable need for cross-sectional bandwidth to flatten the data centre’s networking architecture.
The hyperscalers have particular architectural needs and commonly skip a switch platform generation.
“A few hyperscalers will deploy in very large volumes a 25.6-terabit switch and either later go to 51.2 terabits or wait for 102-terabit switches to appear,” says Manish Mehta, head of product line management and strategy, optical systems division, Broadcom. “Others will be very broad with their 51.2-terabit switch deployments.”
Manish MehtaCo-packaged optics, as the name implies, brings together optics and the chip in one package. Moving optics inside effectively creates a chip with optical I/O.
Co-packaged optics also shortens the length of the high-speed electrical links between the chip and optics, simplifying signal-integrity issues.
But co-packaged optics brings its own technical challenges. The technology also changes how data centre operators use optics and will have industry supply-chain ramifications.
Performance merits
Broadcom highlights several performance advantages using co-packaged optics.
The technology promises to reduce the power consumption by 30 per cent and the cost-per-bit by 40 per cent compared to using pluggable optics. Co-packaged optics also doubles rack density bandwidth.
Broadcom is embracing the same semiconductor techniques used for embedded memory for its on-packaged optics whereby multiple chips can be placed on a common substrate.
“We have taken a design approach that we believe can translate across multiple markets,” says Alexis Björlin, senior vice president and general manager, optical systems division, Broadcom. “Not just Ethernet switching but compute interconnects as well.”
The 30 per cent power savings is achieved by placing the optics as close as possible to the switch chip to shorten the interconnect length of high-speed electrical signals. Currently, getting the high-speed electrical signals to the pluggables requires retimer chips.
“By eliminating all that drive circuitry and putting the optics close to the chip, we are driving the optics from the serdes on the switch chip,” says Björlin. This accounts for the power savings.
The 40 per cent cost-per-bit saving is due to miniaturisation, the removal of retimers, and the improved yield using semiconductor manufacturing techniques for co-packaging.
The doubling in switch rack density using co-packaged optics is because a 51.2-terabit switch will fit in a single rack unit (1RU) box. A 25.6-terabit 1RU box is possible with pluggables.
Pluggable and co-packaged optics
Broadcom expects co-packaged optics to complement pluggable modules; it does not yet see a point where switch designs will only be possible using co-packaged optics.
"But we do know that consumption happens based on cost and power," says Björlin. "We believe there is a huge value proposition for co-packaged optics."
The merits of co-packaged optics also strengthen with each doubling of electrical interface speed. The latest 25.6-terabit switch chips use 100-gigabit serdes and the OIF has started work on a 224-gigabit electrical interface.
The power savings and optical performance co-packaged optics brings will enable new data centre architectures, says Mehta. For example, collapsing the top-of-rack layer or not having to reduce the number of platforms in a rack with each new switch because of their rising power consumption and the fact that a data centre can supply only so much power to a rack.
"Power is what is driving a lot of the investment and interest [in co-packaged optics]," says Mehta. "Cost has to hit the points necessary for adoption and deployment but power is the driver."
Ecosystem issues
One key benefit of optical modules for the data centre operators is its broad ecosystem; the operators value multiple sources and competitive pricing.
With co-packaged optics, only a few key switch-chip companies - Intel, Cisco and Broadcom among them - can deliver a complete co-packaged design. This runs counter to what the hyperscalers like.
"Publicly it may appear there are only a few companies that can provide co-packaged optics solutions," says Mehta. "But if you strip it back, there is a very broad ecosystem that is required to deliver that to market."
These include wafer fabs, packaging, and optical assembly and test companies, he says, adding that in this respect, it is similar to the optical module ecosystem.
"In the end, the hyperscalers providing infrastructure-as-a-service are monetising computing and storage; the network is an overhead," says Björlin. "We need to provide the value to reduce that overhead, and co-packaged optics technologies provide that."
Roadmap
Broadcom says it is not just addressing the switch-chip market; its optical interconnect design is being developed for disaggregated designs, compute clusters and storage. "We are all looking at this as the next generation of optical I/O to interconnect disaggregated resources," says Björlin.
The company has yet to announce a co-packaged product but in a presentation it gave at a JP Morgan event earlier this year, it outlined three upcoming designs.
First, however, it will combine its 8x100-gigabit 4-level pulse-amplitude modulation (PAM4) digital signal processor (DSP) and its silicon photonics for use in 800-gigabit pluggables. "This is a new technology and we want to bring it to market," says Björlin.
This will be followed by its co-packaged optics products codenamed Humboldt, Bailly and Janssen.
Humboldt will be a 25.6-terabit design while Bailly will be a 51.2 terabit one. Humboldt will be offered as a combination design, with different mixes of electrical and optical interfaces. "It's a single switch that you can use in any configuration," says Björlin.
Bailly will be a design with optical interfaces only. The two designs will appear either side of 2023.
Janssen is Broadcom's optical chiplet design for different types of ASICs that need optical I/O. "We have this in 800-gigabit increments," says Björlin. This will appear later on as the market isn't ready for such devices.
The next six-to-nine months will involve a lot of testing in the lab, says Mehta.
Broadcom announces products when they are sampling, so why is the company detailing its optical roadmap now?
Björlin explains that Broadcom is a platform company and this development involves working across multiple company divisions. It is important that Broadcom shares its work given how co-packaged optics involves changes in the ecosystem.
The company has increased significantly its R&D investment in the last decade. "Part of that R&D investment is to share areas of technologies which we believe will be very important in networking," says Björlin.