Latest news

A group of companies, led by Macom and Semtech, have launched a multi-source agreement (MSA) to offer an alternative to using a digital signal processor (DSP) in high-speed client-side optical modules.The Open Eye MSA is developing a set of specifications for optical modules that use 50-gigabit 4-level pulse-amplitude modulation (PAM-4) signals whereby only analogue clock and data recovery (CDR) circuitry is required at the receiver.

By using the CDR instead of a PAM-4 DSP, the optical module will consume less power, have lower latency and be less costly to make, says the MSA.

To ensure interoperability, however, module makers using a PAM-4 DSP will need to meet the new MSA specification.

MSA motivation

The IEEE’s 50-gigabit PAM-4 specification was created several years ago before hardware existed, says Marek Tlalka, director of product marketing, high-performance analogue at Macom.

The IEEE specifications assumed that the received signal would be severely distorted such that the ‘eyes’ would be closed when sampled and displayed on an oscilloscope. Recovering the transmitted 4-level encoded signal would thus be a challenge.

“They had to plan for the worst-case transmitters and receivers which, in the end, led them to define the specifications such that a DSP would be required for signal recovery,” says Tlalka.

Marek Tlalka, MacomBut what has transpired is that the transmitters and receivers now used in high-speed modules have much better optical performance than originally anticipated. “You can get a nice eye on the receive side such that an analogue CDR is quite capable of the recovery,” says Tlalka.

Both Semtech and Macom, which already have such CDR solutions, went independently to their customers to promote such PAM-4 DSP-free module designs. The customers responded favourably to the claimed power, cost and latency savings, but the two companies were told that a standard was needed to generate industry backing and ensure wide availability of components and modules.

The led Macom and Semtech to found the MSA which now has 19 members. And since announcing the MSA in May, some 20 companies have enquired about joining. The companies include systems vendors, module makers, chip, and test companies.

The MSA’s current focus is on 50, 100, 200 and 400-gigabit single-mode module designs that have a reach of 500m to 2km. A draft specification, which involves the optical transmitter and receiver, is almost done, and the final version will be published in the autumn.

The MSA work will then address multi-mode module designs with the final phase tackling 100-gigabit PAM-4 (see table above).

Coexistence

The Open Eye MSA expects PAM-4 DSP modules to co-exist with its own modules.

“We want to be clear that we are not excluding DSPs,” says Timothy Vang, vice president marketing and applications at Semtech. “In this MSA spec, the analogue solution may have a power and a latency advantage, that is all.”

Moreover, there are interface applications where a PAM-4 DSP chip will be needed. One example is for longer reaches, for data centre interconnect up to distances of 80km that are not coherent-based.

Another is for applications where a ‘gearbox’ function is used to translate the electrical signal speeds, for example, from 25-gigabit non-return-to-zero (NRZ) to 50-gigabit PAM-4, or 50-gigabit PAM-4 to 100-gigabit PAM-4.

Issue of interoperability

Tlalka stresses that what the Open Eye MSA is doing is tightening the specification of the module’s optical transmitter: “That open eye can be recovered with an analogue CDR, it can also be recovered with a DSP.”

Inphi, a maker of PAM-4 DSP chips, says it has no current plans to join the Open Eye MSA.

Eric Hayes, senior vice president, networking interconnect at Inphi, agrees that with a wide-open eye, a DSP-based optical module will be compliant with the MSA. He also points out that with an open eye, less signal processing will be required and hence the power consumption will be less.

DSP-based modules using older laser technology, however, may be IEEE compliant but not pass the Open Eye specification. “That is where you potentially have interop issues,” says Tlalka.

“What we believe is going to happen is that the DSP module makers will just have to modify the test procedure for their modules and they will become compliant,” says Tlalka.

Inphi says the current testing procedure for DSP-based optical modules that are IEEE compliant is simple and cost-effective.

“TDECQ-compliance measurement is a highly effective yet simple quantitative procedure already supported by all test equipment,” says Hayes. TDECQ - the transmitter dispersion eye closure, quaternary - is the test used for PAM-4 rather than a traditional eye mask used for NRZ signals.

By removing the TDECQ requirement and using an eye mask to measure the PAM-4 received eye, the Open Eye MSA is forcing optical transceiver module makers to have to implement an extra layer of testing, adding cost, says Hayes.

Tlalka says the MSA does not expect the PAM-4 DSP module makers to adopt two test procedures, nor will two be necessary. “They will just have to test once [using the Open Eye test procedure],” he says.

There are already test companies as part of the Open Eye consortium and two more are joining as part of the 20 companies that have expressed an interest in joining the MSA.

Hyperscalers' preference

Inphi claims that the hyperscalers have a preference for PAM-4 DSP-based modules. This is also a view shared by Dale Murray, principal analyst at LightCounting, the market research firm.

“We do see that hyperscalers prefer to stay with DSP-based modules,” says Hayes. “With DSP-based modules, hyperscalers and their supply chains can take advantage of lower bandwidth optics, which is already in high volume production for several years.”

DSP-based solutions also provide value to customers, such as diagnostics, health and performance monitoring, and firmware upgrades in the field. “These features could lower the total cost of ownership for the hyperscalers and reduce the operational expenditure,” says Hayes.

“I think they are going to go with what is the cheapest and lowest power,” counters Tlalka. “A generic statement is easy to make that they will stay with what is out there, but if they start getting modules that are tens of per cent lower cost and tens of per cent lower power, I think they will pretty quickly change their minds”

“We heard from the end users what they wanted the technology to enable in terms of latency, power and cost,” adds Semtech’s Vang. “We are hopefully being responsive as an ecosystem to the big data centre market.”

“At the end of the day, it will be up to the MSA players to make Open Eye a success,” says LightCounting's Murray.

Operators are already deploying the first reference designs published by the Open Networking Foundation (ONF). Three of the ONF’s five reference designs have now been made public.

Just over a year ago, eight operators - AT&T, Comcast, China Unicom, Deutsche Telekom, Google, NTT Group, Telefonica and Turk Telekom - took the step to design key components of their edge and access networks after becoming frustrated with what they perceived as foot-dragging by the systems vendors.

AT&T is deploying one of the reference designs - the SDN-enabled broadband access scheme (SEBA). Deutsche Telekom and Telefónica have also said they will deploy SEBA during 2019 and 2020.

Andrew Fuetsch, president AT&T Labs and CTO of AT&T, said in a keynote talk at the recent Open Networking Summit North America event that SEBA is resulting in more commoditised hardware solutions and that such open source solutions are bringing higher speeds and better services across AT&T’s wireline infrastructure.

Meanwhile, an undisclosed operator has deployed a second reference design - Trellis, a leaf-spine switch fabric that supports network functions virtualisation (NFV). Trellis can be deployed as part of SEBA although the operator is deploying it for a different application, says the ONF.

The final of the three published reference designs is the Open Disaggregated Transport Network (ODTN). However, unlike SEBA and Trellis, ODTN is not yet available as a deployable platform, what ONF calls an exemplar platform.

Reference designs

The ONF started developing four reference designs as part of the open source organisation’s ongoing cloud optimised remote data centre work, known as CORD. In addition to SEBA, Trellis and ODTN, it is developing a next generation software-defined networking (NG-SDN) solution, previously known as Unified, Programmable & Automated Network (UPAN).

In February, the ONF announced a fifth reference design dubbed Converged Multi-Access and Core (COMAC) that combines 5G mobile and fixed access.

A reference design defines the assembly of components used for key platforms in the access and edge of the network. By creating reference designs, the operators are defining what they need while also committing to deploy the resulting solutions in their networks. This gives the vendors working alongside the operators a confidence that the work will lead to sales.

The components making up a reference design may be ONF-developed but need not be. They may also be open-source or a vendor-specific solution.

In parallel, the ONF operators have been working with vendors to develop an exemplar platform that is a working implementation of a reference design. The ONF only publishes a reference design once the specification is complete and the associated exemplar platform is ready for trialling. However, the ODTN reference design work is an exception in being published while its exemplar platform is unfinished.

“The ONF board felt the work was substantial and mature enough that it was worth getting out there,” says Timon Sloane, the ONF’s vice president of marketing and ecosystem.

CORD evolution

The ONF is known for its software-defined networking (SDN) work - the OpenFlow protocol and ONOS open-source SDN controller - and for its CORD open networking access and edge projects.

Originally known as the Central Office Re-architected as a Data Centre, CORD comprises two key initiatives: a broadband residential CORD (R-CORD) and a mobile CORD (M-CORD). “Once we had the foundation [of SDN], we created R-CORD, the first incarnation, and then we did M-CORD for the mobile space,” says Sloane.

The ONF has since disaggregated both M-CORD and R-CORD into their access and core components. “We realised they both covered a pretty big space,” says Sloane.

R-CORD has been disaggregated into the SEBA reference design and VOLTHA, an open-source abstraction of broadband hardware. Meanwhile, M-CORD has been split into Open Radio Access Network (ORAN) and a newer project, the Open Mobile Evolved Core (OMEC).

The ONF has also upgraded Trellis to not only provide a leaf-spine architecture switch fabric for inside a data centre but also as a multi-tier architecture that spans sites.

Meanwhile, the ODTN reference design is based on first a point-to-point and then a multi-point dense WDM (DWDM) scheme that connects the CORD edge to the network core.

The ONF has been working with the Telecom Infra Project’s (TIP) Open Optical Packet Transport group with a view to using its Cassini white box platform.

“CORD has grown in scope and we recognised that people want to consume the inner pieces in separate ways and that they have separate value,” said Sloane. “And everything of CORD ride on the SDN stack that is being gracefully migrated to the next generation SDN.”

The SDN software stack is based on the ONF’s Stratum project and the P4 language, which is now under the ONF’s stewardship. Stratum provides a software layer that supports various application programming interfaces (APIs) and the underlying hardware. The Stratum project started in 2018 and builds on software provided by Google. In addition, the ONF is also developing a micro-service ONOS.

The newest reference design, COMAC, creates a unified converged access for the radio access network and broadband, says Sloane.

First, the various elements of the fixed and mobile network access and core elements are disaggregated before convergence will bring a common framework for both. For example, a single subscriber management system for users will be used whether they are connected over broadband or mobile. This is a sophisticated project and will be developed in line with the work of the 3GPP and Broadband Forum organisations, says the ONF.

“We expect this migration to [COMAC to] be smooth but to enable a whole new set of capabilities for all the applications that run above it,” says Sloane.

Each time a new reference design is created within the ONF, it means operators committing yet more staff from different parts of their organisation to fulfil the project, says Sloane: “Additional people to pursue yet another angle of attack to transform and push their networks forward.”

Next steps

The ONF has started working on version 2.0 releases for SEBA, Trellis and the ODTN reference designs.

“This will be the case for some time as the work matures,” says Sloane. “There will be versions under development for the foreseeable future.”

Operators have a ‘bunch of work’ they want to progress and drive via this work, he says: “We don’t see an end in sight right now.”

Further information

SEBA, click here

Trellis, click here

ODTN, click here

Brad Booth admits the hyperscalers have a problem.

“Our operational inefficiencies are massive and it is only going to get worse,” says Booth, principal network architect for Microsoft’s Azure Infrastructure and chair of the Consortium for On-Board Optics (COBO).

The issue, he says, is that when a switch arrives at the data centre, it comes without the optics installed. The operations staff must unpack the optical modules, plug them into the switch and verify that each is working; an exercise that is repeated thousands of times when they commission a new data centre.

“The time it takes for us to get the network up and running impacts how quickly we can monetise the data centre,” says Booth.

Moreover, Microsoft wants to use optics to link its servers to the switches. “That is an order of magnitude more optics, which means an order of magnitude more hours,” says Booth. The sum of these hours approaches almost a year, just for the data centre’s optics.

Operational efficiencies

Booth admits that the focus of the COBO venture has been to overcome the technical challenges and publish the first release of the on-board optics specification. Only recently have the operational benefits of on-board optics become evident.

One of these benefits is that, by its very nature, the on-board optics is already installed and tested before the switch is delivered to the data centre. Booth compares on-board optics to the inveterate RJ45 Ethernet connector: “You didn't plug the RJ45 PHY layer in and then plug in the cable; it was already built into the switch.”

COBO also uses new connector designs that simplify the cleaning required when connecting the fibre to the switch.

OFC demonstrator

Microsoft, working with electronic manufacturer Delta, module maker Applied Optoelectronics, and connector suppliers Sumitomo Electric, 3M and Senko, showcased at the OFC show a demonstrator switch using COBO modules.

The proof-of-concept four-rack-unit switch uses Broadcom’s 12.8-terabit Tomahawk 3 Ethernet switch chip (see picture). The top part of the demonstrator switch has been modified to incorporate four 8-lane COBO modules, four 16-lane COBO modules and the new connector types. The switch’s front panel also uses QSFP-DD and OSFP modules that support 400-gigabit pluggable interfaces.

As for the COBO modules, Applied Optoelectronics has developed a 400-gigabit COBO module that supports four 100-gigabit PSM4 interfaces. Other vendors are also developing COBO modules but they have yet to be disclosed. The proof-of-concept platform also offers a choice of COBO connectors from Sumitomo, 3M and Senko.

Sumitomo’s contactless connector that has an air gap when it mates. The result is a more open connector that is far simpler to clean than traditional pluggable connectors. The 3M COBO connector has two polymer faces that slide over each other to create the mate. “You just squirt a can of air [onto the surface], insert it, and you are connected,” says Booth. Meanwhile, Senko's SN connector has also been designed with easy cleaning in mind.

Such operational simplification using the connectors promises significant time savings. “We have got to start doing things smarter,” says Booth. “And COBO is enabling that to happen.”

In a separate development, Petra has developed a 16x25-gigabit 400-gigabit COBO multi-mode module that has a 300m reach. Petra is using two such modules on an FPGA card for artificial intelligence tasks. “They have 800 gigabits of bandwidth coming off what looks like a standard network interface card,” says Booth. Microsoft is also exploring using Petra's module for the switch.

At OFC, the proof-of-concept platform was not operational. Delta only received the platform two weeks before the show, fitted the COBO modules and sent it straight back untested. Delta does have tested platforms up and running, however, says Booth.

Market Status

COBO was founded in 2015 and only now is the first hardware appearing. At the time, developing 400-gigabit pluggables was seen as a significant technical challenging and there were even questions raised as to whether they would deliver the required density needed for data centre switches. Now, a range of 400-gigabit interfaces in the QSFP-DD and OSFP form factors have been announced.

Meanwhile, the topic of co-packaged optics - which brings the optics from the faceplate to reside alongside the chip in the one package - is now receiving industry attention.

Do both developments limit the COBO opportunity?

“When we did this [COBO], we knew people would be able to do a faceplate with 400-gigabit [pluggable] optics,” says Booth. “And what I only got in the last few months is the operational impact [of using COBO].”

At a panel discussion at OFC, companies said they would have 800-gigabit QSFP-DD and OSFP modules by OFC 2020. “At ECOC [in September 2019], we are going to have 800 gigabits in one of these,” says Booth, pointing to a COBO module. “We are already ahead.”

COBO modules can also support 1.6 terabit interfaces once 100-gigabit PAM4 electrical interfaces become available.

InnoLight and Huawei recently joined COBO and during OFC, four other companies - systems vendors and module makers - told COBO they would be joining.

“By 2021, I expect Microsoft will be deploying COBO switches,” says Booth.

Paving the way

COBO is also paving the way to co-packaged optics, or as Booth puts it, COBO has opened up Pandora’s box.

When the idea of an interoperable on-board optics standard was first discussed, there was skepticism regarding how such modules would be serviced, how they couldn't be replaced in the field, and how the switch would need to be disabled when accessing the modules.

“All these things people were saying were correct if you had certain assumptions,” says Booth. “And that was part of the problem with our industry: we got ourselves trained to be lazy.”

But by pursuing COBO, there has been a realisation that things can be done differently such as the new style of connectors.

“By cracking that open, COBO has started a more serious discussion about co-packaged optics because now we have moved the optics away from the faceplate.”