Brad Booth admits the hyperscalers have a problem.
“Our operational inefficiencies are massive and it is only going to get worse,” says Booth, principal network architect for Microsoft’s Azure Infrastructure and chair of the Consortium for On-Board Optics (COBO).
The COBO-enabled 12.8-terabit demonstrator switch. Source: COBO
The issue, he says, is that when a switch arrives at the data centre, it comes without the optics installed. The operations staff must unpack the optical modules, plug them into the switch and verify that each is working; an exercise that is repeated thousands of times when they commission a new data centre.
“The time it takes for us to get the network up and running impacts how quickly we can monetise the data centre,” says Booth.
Moreover, Microsoft wants to use optics to link its servers to the switches. “That is an order of magnitude more optics, which means an order of magnitude more hours,” says Booth. The sum of these hours approaches almost a year, just for the data centre’s optics.
Operational efficiencies
Booth admits that the focus of the COBO venture has been to overcome the technical challenges and publish the first release of the on-board optics specification. Only recently have the operational benefits of on-board optics become evident.
One of these benefits is that, by its very nature, the on-board optics is already installed and tested before the switch is delivered to the data centre. Booth compares on-board optics to the inveterate RJ45 Ethernet connector: “You didn't plug the RJ45 PHY layer in and then plug in the cable; it was already built into the switch.”
COBO also uses new connector designs that simplify the cleaning required when connecting the fibre to the switch.
OFC demonstrator
Microsoft, working with electronic manufacturer Delta, module maker Applied Optoelectronics, and connector suppliers Sumitomo Electric, 3M and Senko, showcased at the OFC show a demonstrator switch using COBO modules.
The proof-of-concept four-rack-unit switch uses Broadcom’s 12.8-terabit Tomahawk 3 Ethernet switch chip (see picture). The top part of the demonstrator switch has been modified to incorporate four 8-lane COBO modules, four 16-lane COBO modules and the new connector types. The switch’s front panel also uses QSFP-DD and OSFP modules that support 400-gigabit pluggable interfaces.
Our operational inefficiencies are massive and it is only going to get worse
As for the COBO modules, Applied Optoelectronics has developed a 400-gigabit COBO module that supports four 100-gigabit PSM4 interfaces. Other vendors are also developing COBO modules but they have yet to be disclosed. The proof-of-concept platform also offers a choice of COBO connectors from Sumitomo, 3M and Senko.
Sumitomo’s contactless connector that has an air gap when it mates. The result is a more open connector that is far simpler to clean than traditional pluggable connectors. The 3M COBO connector has two polymer faces that slide over each other to create the mate. “You just squirt a can of air [onto the surface], insert it, and you are connected,” says Booth. Meanwhile, Senko's SN connector has also been designed with easy cleaning in mind.
Such operational simplification using the connectors promises significant time savings. “We have got to start doing things smarter,” says Booth. “And COBO is enabling that to happen.”
In a separate development, Petra has developed a 16x25-gigabit 400-gigabit COBO multi-mode module that has a 300m reach. Petra is using two such modules on an FPGA card for artificial intelligence tasks. “They have 800 gigabits of bandwidth coming off what looks like a standard network interface card,” says Booth. Microsoft is also exploring using Petra's module for the switch.
At OFC, the proof-of-concept platform was not operational. Delta only received the platform two weeks before the show, fitted the COBO modules and sent it straight back untested. Delta does have tested platforms up and running, however, says Booth.
Market Status
COBO was founded in 2015 and only now is the first hardware appearing. At the time, developing 400-gigabit pluggables was seen as a significant technical challenging and there were even questions raised as to whether they would deliver the required density needed for data centre switches. Now, a range of 400-gigabit interfaces in the QSFP-DD and OSFP form factors have been announced.
By 2021, I expect Microsoft will be deploying COBO switches
Meanwhile, the topic of co-packaged optics - which brings the optics from the faceplate to reside alongside the chip in the one package - is now receiving industry attention.
Do both developments limit the COBO opportunity?
“When we did this [COBO], we knew people would be able to do a faceplate with 400-gigabit [pluggable] optics,” says Booth. “And what I only got in the last few months is the operational impact [of using COBO].”
At a panel discussion at OFC, companies said they would have 800-gigabit QSFP-DD and OSFP modules by OFC 2020. “At ECOC [in September 2019], we are going to have 800 gigabits in one of these,” says Booth, pointing to a COBO module. “We are already ahead.”
COBO modules can also support 1.6 terabit interfaces once 100-gigabit PAM4 electrical interfaces become available.
InnoLight and Huawei recently joined COBO and during OFC, four other companies - systems vendors and module makers - told COBO they would be joining.
“By 2021, I expect Microsoft will be deploying COBO switches,” says Booth.
We got ourselves trained to be lazy
Paving the way
COBO is also paving the way to co-packaged optics, or as Booth puts it, COBO has opened up Pandora’s box.
When the idea of an interoperable on-board optics standard was first discussed, there was skepticism regarding how such modules would be serviced, how they couldn't be replaced in the field, and how the switch would need to be disabled when accessing the modules.
“All these things people were saying were correct if you had certain assumptions,” says Booth. “And that was part of the problem with our industry: we got ourselves trained to be lazy.”
But by pursuing COBO, there has been a realisation that things can be done differently such as the new style of connectors.
“By cracking that open, COBO has started a more serious discussion about co-packaged optics because now we have moved the optics away from the faceplate.”