It may have taken the industry five years to get 400 Gigabit Ethernet (GbE) modules shipping, but for Mark Nowell, Advisory Board Chair at the Ethernet Alliance, the long gestation period is understandable given the innovation that has been required.
Mark Nowell
The industry has had to cram complex technology into a small form factor for 400GbE while meeting the requirements of two very different end-customers: webscale players and communications service providers.
400 Gigabit Ethernet
“The pressure is still on to get 400-gigabit out given that systems are shipping,” says Nowell. “Like with 100-gigabit QSFP28, it is always a rocky road at the beginning, getting all the reaches and getting all the [module] suppliers qualified.”
The 400GbE work is also continuing, with ICs for second-generation optics being tested in the labs that will reduce overall module power consumption.
Nowell notes the scale at which 400 Gigabit Ethernet will be adopted. The number of 400GbE modules expected to ship over the next five years will be 20x the 100GbE module units shipped during its first five years, based on forecast data from LightCounting Market Research.
“100 Gigabit Ethernet started very slowly because it was predominantly service providers [as adopters],” says Nowell. “What is happening with 400 Gigabit Ethernet is that now there is dual-demand from webscale players and service providers.”
This demand has pressured the industry in recent years to achieve a compact form factor, resulted in another noteworthy aspect of 400GbE. “400GbE is unique in that it is the first time we have come out with a new higher-speed Ethernet data rate where all of the reaches - from copper cables to coherent optics - fit in the same form factor,” says Nowell. “Unfortunately, we have two form factors: the QSFP-DD and the OSFP.”
The industry has also had to develop technologies such as 50-gigabit serialiser-deserialiser (serdes), with 100-gigabit serdes now following, and 100-gigabit-per-wavelength optics.
Creating working QSFP-DD and OSFP designs has also been a challenge; how to cool them and achieve the required signal integrity.
“This is the background to where we are today,” says Nowell.
800-gigabit pluggables
Two 800-gigabit multi-source agreements (MSAs) have been announced in the last year.
The 800G Pluggable MSA developing optical specifications for 800-gigabit pluggable modules, and the QSFP-DD800 MSA that extends the QSFP double-density form factor to support 800 gigabits.
The QSFP-DD800 MSA will ensure the connector supports 100-gigabit electrical signals and extend the module’s power envelope from 20W to 24W.
The QSFP-DD800 form factor will also be backwards-compatible with three other QSFPs: the 100-gigabit QSFP28, the 400-gigabit QSFP-DD and the 100-gigabit electrical input 400-gigabit QSFP400.
The first QSFP-DD800 MSA hardware specification was issued in March, to coincide with OFC. The release (version 1.0) has now been passed to the QSFP-DD MSA.
Nowell points out that the technology being used for 800-gigabit MSAs is the same as that being used for 400GbE.
“The amount of investment that we, as an industry, have put into getting these 400-gigabit building blocks, we need to recoup that,” he says. “We need to start building dense 400GbE, making more units and cost-reducing them; 800-gigabit pluggables will do just that.”
Thus, the same form factor - whether an OSFP or QSFP-DD800 - will deliver 2x400GbE or 8x100GbE. Accordingly, no 800GbE work needs to happen.
Next Ethernet rate
“What the 800-gigabit MSAs show is that there is a real interest in what is coming next,” says Nowell, who also co-chairs the QSFP-DD800 MSA.
Building 'denser 400-gigabit' in the form of 800-gigabit modules has an economic benefit for component vendors in that they can recover the investment they have made in the technology.
As for whether the network needs greater speed links, the reason is often one of two things, says Nowell. It either reduces the cost of networking or it benefits networking in terms of hashing efficiencies by using a higher capacity link.
“So does 800 Gigabit Ethernet need to exist to solve the hashing problem?” says Nowell. “I don’t hear a lot of people complaining about this being an issue.” And it would be the webscale players, given the scale of their operations, encountering this first, he says.
Nowell cites the emergence of co-packaged optics, whereby optics are added alongside an ASIC to provide the chip with high-speed optical input-output (I/O).
The requirement for co-packaged optics is coming from the cloud players and he cites the Co-Packaged Optics Consortium created by Microsoft and Facebook that issued a guidance document last year.
“They said this is what we need as an industry, and if you look at that, it is all based on dense 400 Gigabit Ethernet,” says Nowell.
The webscale players’ interest in co-packaged optic stems from the emergence of higher-capacity switch chips at 25.6 terabit and 51.2 terabits.
“The bandwidth demands that customers are seeing and the power efficiencies that customers are looking for, they need a roadmap of silicon development,” says Nowell. “The purpose of optics is to support the switch silicon.”
Adopting dense 400GbE will ensure that equipment using co-packaged optics will be able to talk to equipment using pluggables only.
Given work on 400GbE started around 2014, and that the capacity of Ethernet switch chips is doubling every two years, is there no urgency to define the next Ethernet rate?
Not yet, says Nowell, but it is something the Ethernet Alliance recognises.
It organises an event dubbed the Technology Exploration Forum (TEF) that brings together industry experts and the latest was scheduled for April 15th, focussing on the next Ethernet rate after 400GbE. However, due to the COVID-19 pandemic, the event has been postponed till October 7th.
“I don’t think there is any inconsistency yet but it is the right time to start, and that is why the Ethernet Alliance wants to drive the TEF event and have the discussion,” he says.
There is also the IEEE’s New Ethernet Applications group that acts as an incubator for new ideas. Such ideas can lead to a Call For Interest which, if sufficiently supported, is how new IEEE projects begin.
One such project that has just started is entitled Beyond 400 Gigabit that will look at market needs and define, from an IEEE perspective, what should be done in terms of technologies.
The IEEE 400GbE work took five years because it introduced several new technologies: 50-gigabit serdes, 100-gigabit-per-lambda optics, 4-level pulse amplitude modulation (PAM-4), and the use of 8-lanes for certain 400GbE implementations.
Nowell points out that the 800G Pluggable MSA includes 200-gigabit-per-lambda optics and FEC which are complex elements that require time to work through.
“My opinion is that if you want to do something in the IEEE, you want more headroom,” says Nowell. “It would be a mistake for the IEEE to only pursue 800 Gigabit Ethernet and then come up for air in several years and find that the industry has moved on.”
So the likelihood is that the next Ethernet rate will be at least 2.5x the current one (1TbE) and even as high as 4x (1.6TbE), similar to how 10GbE moved to 40GbE.
But this will be challenging, says Nowell. Issues such as what comes after 100-gigabit serdes and 100-gigabit-per-wavelength, and when will coherent technology be cost-effective enough for use within the data centre remain unresolved.
Meanwhile, the Ethernet Alliance’s TEF aims to shape the debate by bringing together different industry voices, with an emphasis on end-users and thought leaders.
Will the COVID-19 pandemic impede development and standards work?
“In this industry, we are used to working with collaborative tools so it’s not such a shock to pick up these to progress,” says Nowell. “I’ve been pleasantly surprised we have been able to do standards work this way without too much disruption other than it takes longer.”
But what is missing is meeting people face-to-face and the discussions that ensue.
“It does make a difference getting out and talking to customers,” says Nowell. “There is an unquantifiable thing that is missing.”