The compound complexity of co-packaged optics
Large-scale data centres consume huge amounts of power; one building on a data centre campus can consume 100MW. But there is a limit as to the overall power that can be supplied.
Jeff Hutchins
The challenge facing data centre operators is that networking, used to link the equipment inside the data centre, is consuming more and more of the power.
That means less power remains for the servers; the computing that does the revenue-generating work.
This is forcing a rethink regarding networking and explains the growing interest in co-packaged optics, a technique that effectively adds optical input-output (I/O) to a chip.
Two industry organisations - the OIF and The Consortium for On-Board Optics (COBO) - have each started work to identify the requirements needed for co-packaged optics adoption.
“We are seeing this activity because co-packaged optics is hard and requires prework to figure out how and when it is going to happen, and how the ecosystem changes,” says Nathan Tracy, TE Connectivity and the OIF’s vice president of marketing.
All change
Semiconductors and optics have always been separate domains but with a co-packaged design, silicon is suddenly only a handful of millimetres away from the optics, says Tracy: “It’s a very different environment.”
Hot chips sit next to the optics, so thermal characteristics must be shared and the cooling needs worked out. The electrical interface linking the optics to the chip will need to be optimised while there are new challenges such as how faults are dealt with.
“All these things come together and it changes what is done in the industry,” says Jeff Hutchins, Ranovus and OIF Physical and Link Layer (PLL) Working Group – Co-Packaging Vice-Chair.
“To be fair, there are companies that are not totally on-board with co-packaging,” says Hutchins. “But if you think about what is driving it, as you go to higher and higher electrical rates to connect things, you start to run more power and it is just more difficult to get a signal from Point A to Point B.”
For next-generation designs, companies are also considering ‘fly-over’ cables as well as the intermediate step of on-board optics, moving optics from the front panel onto the line card to be closer to the ASIC.
“But a good part of the industry thinks that, if you look forward, the only way to get there is co-packaging,” says Hutchins.
Using co-packaged optics will also impact the supply chain. The switch and pluggable modules are typically bought separately whereas a co-packaged design integrates the two. “Economically, it changes the way the industry works,” says Hutchins.
OIF and COBO
Nathan Tracy
Hutchins, who is also a board member of COBO, says the co-packaging work of the two organisations will be complementary.
Co-packaged optics resides deep on the line card and fibre must connect the package to the system’s front panel. In turn, an external laser is commonly used as the light source for the optics. Such a laser is linked to the package using fibre.
“What COBO is doing is focussing on the optical connectivity part of this solution; the stuff outside the co-packaged assembly,” says Hutchins. “The OIF is concentrating on what the co-package assembly is, what goes inside, and what agreements can be made for interoperability for the whole assembly.”
The membership of the two organisations also differs: the OIF members include hyperscalers as well as optical and switch companies. “We have a good cross-section of the membership of this ecosystem,” says Tracy. COBO’s membership includes companies with connector and materials expertise.
Framework project
The OIF Framework Project will first study the applications where co-packaged optics will be used, identifying commonalities. It will then address the technology to determine what interoperability agreements are needed.
Applications for co-packaged optics besides Ethernet switches include machine learning and disaggregation. A disaggregated design refers to separating the chips found on a server motherboard - general processors (CPUs), graphics processor units (GPUs) and memory - into separate pools. A workload can then access the pools and configure the hardware elements it needs.
For each application, issues such as density, power, latency, and wavelength-count-per-fibre will be explored. “These must be understood as they differ as you go across the applications,” says Hutchins.
The OIF will identify what interoperability agreements to pursue and what should remain open for now before kicking-off specific Implementation Agreements.
Hutchins stresses that are many aspects that can be standardised such as the mechanical design, environmental issues, power, electrical interfaces and reliability. “That is enough work to keep the whole group busy for quite a while,” he says
As an example, such work could lead to a common socket design that would allow different optical specifications and reliability requirements, says Hutchins.
The OIF expects to complete the first two stages within the coming year.
“People are ready to go but they need to see the whole picture,” says Hutchins.
Roadmap
The OIF expects a gradual introduction of co-packaged Ethernet switches in the data centre with the technology spanning several generations.
Demonstrations could start with 25.6-terabit switches emerging now whereas many think the next-generation 51.2-terabit platforms will be the place to do initial demonstrations and small-scale deployments. After that, 100-terabit switches will likely be the sweet spot for co-packaged optics. And once 200-terabit switches appear, co-packaged optics will be a necessity.
This may be a wide range of entry points, says Hutchins, but technology is being put together in a new way.
“The industry has to learn how to make this cost-effectively and achieve good yields,” says Hutchins. “There has to be a starting point somewhere but where the intercept point is, I don’t know.”
“Pluggables have served the market really well; they are flexible and [optical module] innovation continues,” adds Tracy. “The methodology is working so the question is when does it no longer suit the market.”
Tracy does not rule out pluggables being used for 100-terabit switches but inevitably it will be much harder to satisfy that requirement. “That is when co-packaged optics starts to become compelling,” says Tracy.
Data centre interconnect drives coherent
-
NeoPhotonics announced at OFC a high-speed modulator and intradyne coherent receiver (ICR) that support an 800-gigabit wavelength
-
It also announced limited availability of its nano integrable tunable laser assembly (nano-ITLA) and demonstrated its pico-ITLA, an even more compact silicon photonics-based laser assembly
-
The company also showcased a CFP2-DCO pluggable
NeoPhotonics unveiled several coherent optical transmission technologies at the OFC conference and exhibition held in San Diego last month.
“There are two [industry] thrusts going on right now: 400ZR and data centre interconnect pizza boxes going to even higher gigabits per wavelength,” says Ferris Lipscomb, vice president of marketing at NeoPhotonics.
Ferris Lipscomb
The 400ZR is an interoperable 400-gigabit coherent interface developed by the Optical Internetworking Forum (OIF).
Optical module makers are developing 400ZR solutions that fit within the client-side QSFP-DD and OSFP pluggable form factors, first samples of which are expected by year-end.
800-gigabit lambdas
Ciena and Infinera announced in the run-up to OFC their latest coherent systems - the WaveLogic 5 and ICE6, respectively - that will support 800-gigabit wavelengths. NeoPhotonics announced a micro intradyne coherent receiver (micro-ICR) and modulator components that are capable of supporting such 800-gigabit line-rate transmissions.
NeoPhotonics says its micro-ICR and coherent driver modulator are class 50 devices that support symbol rates of 85 to 90 gigabaud required for such a state-of-the-art line rate.
The OIF classification defines categories for devices based on their analogue bandwidth performance. “With class 20, the 3dB bandwidth of the receiver and the modulator is 20GHz,” says Lipscomb. “With tricks of the trade, you can make the symbol rate much higher than the 3dB bandwidth such that class 20 supports 32 gigabaud.” Thirty-two gigabaud is used for 100-gigabit and 200-gigabit coherent transmissions.
Class 50 refers to the highest component performance category where devices have an analogue bandwidth of 50GHz. This equates to a baud rate close to 100 gigabaud, fast enough to achieve data transmission rates exceeding a terabit. “But you have to allow for the overhead the forward-error correction takes, such that the usable data rate is less than the total,” says Lipscomb (see table).
Source: Gazettabyte, NeoPhotonics
Silicon photonics-based COSA
NeoPhotonics also announced a 64-gigabaud silicon photonics-based coherent optical subassembly (COSA). The COSA combines the receiver and modulator in a single package that is small enough to fit within a QSFP-DD or OSFP pluggable for applications such as 400ZR.
Last year, the company announced a similar COSA implemented in indium phosphide. In general, it is easier to do higher speed devices in indium phosphide, says Lipscomb, but while the performance in silicon photonics is not quite as good, it can be made good enough.
“It [silicon photonics] is now stretching certainly into the Class 40 [that supports 600-gigabit wavelengths] and there are indications, in certain circumstances, that you might be able to do it in the Class 50.”
Lipscomb says NeoPhotonics views silicon photonics as one more material that complements its indium phosphide, planar lightwave circuit and gallium arsenide technologies. “Our whole approach is that we use the material platform that is best for a certain application,” says Lipscomb.
In general, coherent products for telecom applications take time to ramp in volumes. “With the advent of data centre interconnect, the volume growth is much greater than it ever has been in the past,” says Lipscomb.
NeoPhotonics’ interested in silicon photonics is due to the manufacturing benefits it brings that help to scale volumes to meet the hyperscalers’ requirements. “Whereas indium phosphide has very good performance, the infrastructure is still limited and you can’t duplicate it overnight,” says Lipscomb. “That is what silicon photonics does, it gives you scale.”
NeoPhotonics also announced the limited availability of its nano integrable tunable laser assembly (nano-ITLA). “This is a version of our external cavity ITLA that has the narrowest line width in the industry,” says Lipscomb.
The nano-ITLA can be used as the source for Class 50, 800-gigabit systems and current Class 40 600 gigabit-per-wavelength systems. It is also small enough to fit within the QDFP-DD and OSFP client-side modules for 400ZR designs. “It is a new compact laser that can be used with all those speeds,” says Lipscomb.
NeoPhotonics also showed a silicon-photonics based pico-ITLA that is even smaller than the nano-ITLA.“The [nano-ITLA’s] optical cavity is now made using silicon photonics so that makes it a silicon photonics laser,” says Lipscomb.
Instead of having to assemble piece parts using silicon photonics, it can be made as one piece. “It means you can integrate that into the same chip you put your modulator and receiver on,” says Lipscomb. “So you can now put all three in a single COSA, what is called the IC-TROSA.” The IC-TROSA refers to an integrated coherent transmit-receive optical subassembly, defined by the OIF, that fits within the QSFP-DD and OSFP.
Despite the data centre interconnect market with its larger volumes and much faster product uptakes, indium phosphide will still be used in many places that require higher optical performance. “But for bulk high-volume applications, there are lots of advantages to silicon photonics,” says Lipscomb.
400ZR and 400ZR+
A key theme at this year’s OFC was the 80km 400ZR. Also of industry interest is the 400ZR+, not an OIF specification but an interface that extends the coherent range to metro distances.
Lipscomb says that the initial market for the 400ZR+ will be smaller than the 400ZR, while the ZR+’s optical performance will depend on how much power is left after the optics is squeezed into a QSFP-DD or OSFP module.
“The next generation of DSP will be required to have a power consumption low enough to do more than ZR distances,” he says. “The further you go, the more work the DSP has to do to eliminate the fibre impairments and therefore the more power it will consume.”
Will not the ZR+ curtail the market opportunity for the 400-gigabit CFP2-DCO that is also aimed at the metro?
“It’s a matter of timing,” says Lipscomb. “The advantage of the 400-gigabit CFP2-DCO is that you can almost do it now, whereas the ZR+ won’t be in volume till the end of 2020 or early 2021.”
Meanwhile, NeoPhotonics demonstrated at the show a CFP2-DCO capable of 100-gigabit and 200-gigabit transmissions.
NeoPhotonics has not detailed the merchant DSP it is using for its CFP2-DCO except to say that it working with ‘multiple ones’. This suggests it is using the merchant coherent DSPs from NEL and Inphi.
Oclaro’s 400-gigabit plans
Adam Carter, Oclaro’s chief commercial officer, discusses the company’s 400-gigabit and higher-speed coherent optical transmission plans and the 400-gigabit client-side pluggable opportunity.
Oclaro showcased its first coherent module that uses Ciena’s WaveLogic Ai digital signal processor at the ECOC show held recently in Gothenburg.
Adam CarterOclaro is one of three optical module makers, the others being Lumentum and NeoPhotonics, that signed an agreement with Ciena earlier this year to use the system vendor’s DSP technology and know-how to bring coherent modules to market. The first product resulting from the collaboration is a 5x7-inch board-mounted module that supports 400-gigabits on a single-wavelength.
The first WaveLogic Ai-based modules are already being tested at several of Oclaro’s customers’ labs. “They [the module samples] are very preliminary,” says Adam Carter, the chief commercial officer at Oclaro. “The really important timeframe is when we get towards the new year because then we will have beta samples.”
DSP developments
The coherent module is a Ciena design and Carter admits there isn’t going to be much differentiation between the three module makers’ products.
“We have some of the key components that sit inside that module and the idea is, over time, we would design in the rest of the componentry that we make that isn’t already in there,” says Carter. “But it is still going to be the same spec between the three suppliers.”
The collaboration with the module makers helps Ciena promote its coherent DSP to a wider market and in particular China, a market where its systems are not deployed.
Over time, the scope for differentiation between the three module makers will grow. “It [the deal] gives us access to another DSP chip for potential future applications,” says Carter.
Here, Oclaro will be the design authority, procuring the DSP chip for Ciena before adding its own optics. “So, for example, for the [OIF’s] 400G ZR, we would ask Ciena to develop a chip to a certain spec and then put our optical sub-assemblies around it,” says Carter. “This is where we do believe we can differentiate.”
Oclaro also unveiled at ECOC an integrated coherent transmitter and an intradyne coherent receiver optical sub-assemblies using its indium phosphide technology that operate at up to 64 gigabaud (Gbaud).
We expect to see 64Gbaud optical systems being trialed in 2018 with production systems following at the end of next year
A 64Gbaud symbol rate enables a 400-gigabit wavelength using 16-ary quadrature amplitude modulation (16-QAM) and a 600-gigabit wavelength using 64-QAM.
Certain customers want such optical sub-assemblies for their line card designs and Oclaro will also use the building blocks for its own modules. The devices will be available this quarter. “We expect to see 64Gbaud optical systems being trialed in 2018 with production systems following at the end of next year and the beginning of 2019,” says Carter.
Oclaro also announced that its lithium niobate modulator supporting 400-gigabit single wavelengths is now in volume production. “Certain customers do have their preferences when it comes to first designs and particularly for long-reach systems,” says Carter. “Lithium niobate seems to be the one people go with.”
400-gigabit form factors
Oclaro did not make any announcements regarding 400-gigabit client-side modules at ECOC. At the OFC show held earlier this year, it detailed two CFP8-based 400-gigabit designs based on eight wavelengths with reaches of 10km and 40km.
“We are sampling the 400-gigabit 10km product right now,” says Carter. “The product is being tested at the system level and will go through various qualification runs.”
The 40km CFP8 product is further out. There are customers interested in such a module as they have requirements to link IP routers that are more than 10km apart.
Carter describes the CFP8 400-gigabit modules as first-generation products. The CFP8 is similar in size to the CFP2 pluggable module and that is too large for the large-scale data centre players. They want higher aggregate bandwidth and greater front panel densities for their switches and are looking such form factors as the double-density QSFP (QSFP-DD) and the Octal Small Form Factor pluggable (OSFP).
The OSFP is a fresh design, has a larger power envelope - some 15W compared to the 12W of the QSFP-DD - and has a roadmap that supports 800-gigabit data rates. In contrast, the QSFP-DD is backward compatible with the QSFP, an attractive feature for many vendors.
But it is not only a module’s power envelope that is an issue for 400-gigabit designs but also whether a one-rack-unit box can be sufficiently cooled when fully populated to avoid thermal runaway. Some 36 QSFP-DDs can fit on the front panel compared to 32 OSFPs.
Carter stresses both form factors can’t be dismissed for 400-gigabit: “Everyone is pursuing designs that are suitable for both.” Oclaro is not an advocate of either form factor given it provides optical sub-assemblies suitable for both.
The industry really wants four-channels. When you use more lasers, you are adding more cost.
Optical formats
Oclaro’s core technology is indium phosphide and, as such, its focusses on single-mode fibre designs.
The single mode options for 400 gigabits are split between eight-wavelength designs such as the IEEE 802.3bs 2km 400GBASE-FR8 and 10km 400GBASE-LR8 and the newly announced CWDM8 MSA, and four-wavelength specifications - the 500m IEEE 802.3bs parallel fibre 400GBASE-DR4 and the 2km 100G Lambda MSA 400G-FR4 that is under development. Oclaro is a founding member of the 100 Gigabit Lambda MSA but has not joined the CWDM8 MSA.
"The industry really wants four channels," says Carter. "When you use more lasers, you are adding more cost." It is also not trivial fitting eight lasers into a CFP8 never mind into the smaller QSFP-DD and OSFP modules.
“There might be some that have the technology to do the eight-channel part and there might be customers that will use that,” says Carter. “But most of the discussions we’ve been having are around four channels.”
Challenges
The industry’s goal is to have 400-gigabit QSFP-DD and OSFP module in production by the end of next year and into 2019. “There is still some risk but everybody is driving to meet that schedule,” says Carter.
Oclaro says first samples of 100-gigabit PAM-4 chips needed for 100-gigabit single wavelengths are now in the labs. Module makers can thus add their optical sub-assemblies to the chips and start testing system performance. Four-channel PAM-4 chips will be needed for the 400-gigabit module products.
Carter also acknowledges that any further delay in four-wavelength designs could open the door for other 400-gigabit solutions and even interim 200-gigabit designs.
“As a transceiver supplier and an optical component supplier you are always aware of that,” he says. “You have to have backup plans if that comes off.”
Teraxion embraces silicon photonics for its products
Teraxion has become a silicon photonics player with the launch of its compact 40 and 100 Gigabit coherent receivers.
The Canadian optical component company has long been known for its fibre Bragg gratings and tunable dispersion compensation products. But for the last three years it has been developing expertise in silicon photonics and at the recent European Conference on Optical Communications (ECOC) exhibition it announced its first products based on the technology.
"You don't have this [fabless] model for indium phosphide or silica, while an ecosystem is developing around silicon photonics"
Martin Guy, Teraxion
"We are playing mainly in the telecom business, which accounts for 80% of our revenues," says Martin Guy, vice president, product management & technology at Teraxion. "It is clear that our customers are going to more integration and smaller form-factors so we need to follow our customers' requirements."
Teraxion assessed several technologies but chose silicon photonics and the fabless model it supports. "We are using all our optical expertise that we can apply to this material but use a process already developed for the CMOS industry, with the [silicon] wafer made externally," says Guy. "You don't have this [fabless] model for indium phosphide or silica, while an ecosystem is developing around silicon photonics."
The company uses hybrid integration for its coherent receiver products, with silicon implementing the passive optical functions to which the active components are coupled. Teraxion is using externally-supplied photo-detectors which are flip-chipped onto the silicon for its coherent receiver.
"We need to use the best material for the function for this high-end product," says Guy. "Our initial goal is not to have everything integrated in silicon."
Coherent receiver
A coherent receiver comprises two inputs - the received optical signal and the local oscillator - and four balanced receiver outputs. Also included in the design are two polarisation beam splitters and two 90-degree hybrid mixers.
Several companies have launched coherent receiver products. These include CyOpyics, Enablence, NEL, NeoPhotonics, Oclaro and u2t Photonics. Silicon photonics player Kotura has also developed the optical functions for a coherent receiver but has not launched a product.
One benefit of using silicon photonics, says Teraxion, is the compact optical designs it enables.
The Optical Internetworking Forum (OIF) has specified a form factor for the 100 Gigabit-per-second (Gbps) coherent receiver. Teraxion has developed a silicon photonics-based product that matches the OIF's form factor sized 40mmx32mm. This is for technology evaluation purposes rather than a commercial product. "If customers want to evaluate our technology, they need to have a compatible footprint with their design," explains Guy. This is available in prototype form and Teraxion has customers ready to evaluate the product.
Teraxion will come to market with a second 100 Gigabit coherent receiver design that is a third of the size of the OIF's form factor, measuring 23mmx18mm (0.32x the area of the OIF specification). The compact coherent receivers for 40 and 100Gbps will be available in sample form in the first quarter of 2013.
Teraxion's OIF-specification 100 Gig coherent receiver (left) for test purposes and its compact coherent receiver product. Source: Teraxion
"We match the OIF's performance with this design but there are also other key requirements from customers that are not necessarily in the OIF specification," says Guy.
The compact 100Gbps design is of interest to optical module and system vendors but there is no one view in terms of requirements or the desired line-side form-factor that follows the 5x7-inch MSA. Indeed there are some that are interested in developing a 100 Gigabit CFP module for metro applications, says Guy.
Roadmap
Teraxion's roadmap includes further integration of the coherent receiver's design. "We are using hybrid integration but eventually we will look at having the photo-detectors integrated within the material,” says Guy.
The small size of the coherent design means there is scope for additional functionality to be included. Teraxion says that customers are interested in integrating variable optical attenuators (VOAs). The local oscillator is another optical function that can be integrated within the coherent receiver.
In 2005 Teraxion acquired Dicos Technologies, a narrow line-width laser specialist. Teraxion's tunable narrow line-width laser product - a few kiloHertz wide - is available in the lab. "The purpose of this product is not to be deployed on the line card - right now," says Guy. "We believe this type of performance will be required for next-generation 100 Gig, 400 Gig, 1 Terabit coherent communication systems where you will need a very 'clean' local oscillator."
Teraxion is also working on developing a silicon-photonics-based modulator. The company has been exploring integrating Bragg gratings within silicon waveguides for which it has applied for patents. This is several years out, says Guy, but has the potential to enable high-speed modulators suited for short-reach datacom applications.