Deutsche Telekom explains its IP-over-DWDM thinking
Telecom operators are always seeking better ways to run their networks. In particular, operators regularly scrutinise how best to couple the IP layer with their optical networking infrastructure.
The advent of 400-gigabit coherent modules that plug directly into an IP router is one development that has caught their eye.
Placing dense wavelength division multiplexing (DWDM) interfaces directly onto an IP router allows the removal of a separate transponder box and its interfacing.
IP-over-DWDM is not a new concept. However, until now, operators have had to add a coherent line card, taking up valuable router chassis space.
Now, with the advent of compact 400-gigabit coherent pluggables developed for the hyperscalers to link their data centres, telecom operators have realised that such pluggables also serve their needs.
BT will start rolling out IP-over-DWDM in its network this year, while Deutsche Telekom has analysed the merits of IP-over-DWDM.
“The adoption of IP-over-DWDM is the subject of our techno-economical studies,” says Werner Weiershausen, senior architect for the transport network at Deutsche Telekom.
Network architecture
Deutsche Telekom’s domestic network architecture comprises 12 large nodes where IP and OTN backbones align with the underlying optical networking infrastructure. These large nodes – points of presence – can be over 1,000km apart.
Like many operators, Deutsche Telekom has experienced IP annual traffic growth of 35 per cent. The need to carry more traffic without increasing costs has led the operators to adopt coherent technology, with the symbol rate rising with each new generation of optical transport technology.
A higher channel bit rate sends more data over an optical wavelength. The challenge, says Weiershausen, is maintaining the long-distance reaches with each channel rate hike.
Deutsche Telekom’s in-house team forecasts that IP traffic growth will slow down to a 20 per cent annual growth rate and even 16 per cent in future.
Weiershausen says this is still to be proven but that if annual traffic growth does slow down to 16-20 per cent, bandwidth growth issues will remain; it is just that they can be addressed over a longer timeframe.
Bandwidth and reach are long-haul networking issues. Deutsche Telekom’s metro networks, which are horse-shoe-shaped, have limited spans overall.
“For metro, our main concern is to have the lowest cost-per-bit because we are fibre- and spectrum-rich, and even a single DWDM fibre pair per metro horseshoe ring offer enough bandwidth headroom,” says Weiershausen. “So it’s easy; we have no capacity problem like the backbone. Also there, we are fibre-rich but can avoid the costly activation of multiple parallel fibre trunks.”
IP-over-DWDM
IP-over-DWDM is increasingly associated with adding pluggable optics onto an IP core router.
“This is what people call IP-over-DWDM, or what Cisco calls it hop-by-hop approach,” says Dr Sascha Vorbeck, head of strategy and architecture IP-core & transport networks at Deutsche Telekom.
Cisco’s routed optical networking – its term for the hop-by-hop approach – uses the optical layer for point-to-point connections between IP routers. As a result, traffic switching and routing occur at the IP layer rather than the optical layer, where optical traffic bypass is performed using reconfigurable optical add/drop multiplexers (ROADMs).
Routed optical networking also addresses the challenge of the rising symbol rate of coherent technology, which must maintain the longest reaches when passing through multiple ROADM stages.
Deutsche Telekom says it will not change its 12-node backbone network to accommodate additional routing stages.
“We will not change our infrastructure fundamentally because this is costly,” says Weiershausen. “We try to address this bandwidth growth with technology and not with the infrastructure change.”
Deutsche Telekom’s total cost-of-ownership analysis highlights that optical bypass remains attractive compared to a hop-by-hop approach for specific routes.
However, the operator has concluded that the best approach is to have both: some hop-by-hop where it suits its network in terms of distances but also using optical bypass for longer links using either ROADM or static bypass technology.
“A mixture is the optimum from our total cost of ownership calculation,” says Weiershausen. “There was no clear winner.”
Strategy
Deutsche Telecom favours coherent interfaces on its routers for its network backbone because it wants to simplify its network. In addition, the operator wants to rid its network of existing DWDM transponders and their short reach – ‘grey’ – interfaces linking the IP router to the DWDM transponder box.
“They use extra power and are an extra capex [capital expenditure] cost,” says Weiershausen. “They are also an additional source of failures when you have in-line several network elements. That said, heat dissipation of long-reach coherent optical DWDM interfaces limited the available IP router interfaces that could have been activated in the past.
For example, a decade ago, Deutsche Telecom tried to use IP-over-DWDM for its backbone network but had to step back to use an external DWDM transponder box due to heat dissipation problems.
The situation may have changed with modern router and optical interface generations, but this is under further study by Deutsche Telecom and is an essential prerequisite for its evolution roadmap.
Deutsche Telecom is still using traditional DWDM equipment between the interconnection of IP routers with grey interfaces. Deutsche Telecom undertook an evaluation in 2020 and calculated a traditional DWDM network versus a hop-by-hop approach. Then, the hop-by-hop method was 20 per cent more expensive. Deutsche Telecom plans to redo the calculations to see if anything has changed.
The operator has yet to decide whether to adopt ZR+ coherent pluggable optics and a hop-by-hop approach or use more advanced larger coherent modules in its routers. “This is not decided yet and depends on pricing evolution,” says Weiershausen.
With the volumes expected for pluggable coherent optics, the expectation is they will have a notable price advantage compared to traditional high-performance coherent interfaces.
But Deutsche Telekom is still determining, believing that conventional coherent interfaces may also come down markedly in price.
SDN controller
Another issue for consideration with IP-over-DWDM is the software-defined networking (SDN) controller.
IP router vendors offer their SDN controllers, but there also is a need for working with third-party SDN controllers.
For example, Deutsche Telekom is a member of the OpenROADM multi-source agreement and has pushed for IP-over-DWDM to be a significant application of the MSA.
But there are disaggregation issues regarding how a router’s coherent optical interfaces are controlled. For example, are the optical interfaces overseen and orchestrated by the OpenROADM SDN controller and its application programming interface (API) or is the SDN controller of each IP router vendor responsible for steering the interfaces?
Deutsche Telekom says that a compromise has been reached for the OpenROADM MSA whereby the IP router vendors’ SDN controllers oversee the optics but that for the solution to work, information is exchanged with the OpenROADM’s SDN controller.
“That way, the path computation engine (PCE) of the optical network layer, including the ROADMs, can calculate the right path to network the traffic. “Without information from the IP router, it would be blind; it would not work,” says Weiershausen.
Automation
Weiershausen says it is not straightforward to say which approach – IP-over-DWDM or a boundary between the IP and optical layers – is easier to automate.
“Principally, it is the same in terms of the information model; it is just that there are different connectivity and other functionalities [with the two approaches],” says Weiershausen.
But one advantage of a clear demarcation between the layers is the decoupling of the lifecycles of the different equipment.
Fibre has the longest lifecycle, followed by the optical line system, with IP routers having the shortest of the three, with new generation equipment launched every few years.
Decoupling and demarcation is therefore a good strategy here, notes Weiershausen.
Lumentum’s CTO discusses photonic trends
CTO interviews part 2: Brandon Collings
- The importance of moving to parallel channels will only increase given the continual growth in bandwidth.
- Lumentum’s integration of NeoPhotonics’ engineers and products has been completed.
- The use of coherent techniques continues to grow, which is why Lumentum acquired the telecom transmission product lines and staff of IPG Photonics.
“It has changed quite significantly given what Lumentum is engaging in,” he says. “My role spans the entire company; I’m engaged in a lot of areas well beyond communications.”
A decade ago, the main focus was telecom and datacom. Now Lumentum also addresses commercial lasers, 3D sensing, and, increasingly, automotive lidar.
Acquisitions
Lumentum was busy acquiring in 2022. The deal to buy NeoPhotonics closed last August. The month of August was also when Lumentum acquired IPG Photonics’ telecom transmission product lines, including its coherent digital signal processing (DSP) team.
NeoPhotonics’ narrow-linewidth tunable lasers complement Lumentum’s modulators and access tunable modules. Meanwhile, the two companies’ engineering teams and portfolios have now been merged.
NeoPhotonics was active in automotive lidar, but Lumentum stresses it has been tackling the market for several years.
“It’s an area with lots of nuances as to how it is going to be adopted: where, how fast and the cost dependences,” says Collings. “We have been supplying illuminators, VCSELs, narrow-linewidth lasers and other technologies into lidar solutions for several different companies.”
Lumentum gained a series of technological capabilities and some products with the IPG acquisition. “The big part was the DSP capability,” says Collings.
ROADMs
Telecom operators have been assessing IP-over-DWDM anew with the advent of coherent optical modules that plug directly into an IP router.
Cisco’s routed optical networking approach argues the economics of using routers and the IP layer for traffic steering rather than at the optical layer using reconfigurable optical add-drop multiplexers (ROADMs).
Is Lumentum, a leading ROADM technology supplier, seeing such a change?
“I don’t think there is a sea change on the horizon of moving from optical to electrical switching,” says Collings. “The reason is still the same: transceivers are still more expensive than optical switches.”
That balance of when to switch traffic optically or electrically remains at play. Since IP traffic continues to grow, forcing a corresponding increase in signalling speed, savings remain using the optical domain.
“There will, of course, be IP routers in networks but will they take over ROADMs?” says Collings. “It doesn’t seem to be on the horizon because of this growth.”
Meanwhile, the transition to more flexible optical networking using colourless, directionless, contentionless (CDC) ROADMs, is essentially complete.
Lumentum undertook four generations of switch platform design in the last decade to enable CDC-ROADM architectures that are now dominant, says Collings.
Lumentum moved from a simple add-drop to a route-and-select and a colourless, contentionless architecture.
A significant development was Lumentum’s adoption of liquid-crystal-on-silicon (LCOS) technology that enabled twin wavelength-selective switches (WSSes) per node that adds flexibility. LCOS also has enabled a flexible grid which Lumentum knew would be needed.
“We’re increasingly using MEMS technology alongside LCOS to do more complex switching functions embedded in colourless, directionless and contentionless networks today,” says Collings.
Shannon’s limit
If the last decade has been about enabling multiplexing and demultiplexing flexibility, the next challenge will be dealing with Shannon’s limit.
“We can’t stuff much more information into a single optical fibre – or that bit of the amplified spectrum of the optical fibre – and go the same distance,” says Collings. “We’ve sort of tapped out or reached that capacity.”
Adding more capacity requires amplified fibre bandwidth, such as using the L-band alongside the C-band or adding a second fibre.
Enabling such expansion in a cost- and power-efficient way will be fundamental, says Collings, and will define the next generation of optical networks.
Moreover, he expects consumer demand for bandwidth growth to continue. More sensing and more up-hauling of data to the cloud for processing will occur.
Accordingly, optical transceivers will continue to develop over the next decade.
“They are the complement requirement for scaling bandwidth, cost and power effectively,” he says.
Parallelism
Continual growth of bandwidth over the next decade will cause the industry to experience technological ceilings that will drive more parallelism in communications.
“If you look in data centres and datacom interconnects, they have long moved to parallel interface implementations because they felt that bandwidth ceiling from a technological, power dissipation or economic reason.”
Coherent systems have a symbol rate of 128 gigabaud (GBd), and the industry is working on 256GBd systems. Sooner or later, the consensus will be that the symbol rate is fast enough, and it is time to move to a parallel regime.
“In large-scale networks, parallelism is going to be the new thing over the next ten years,” says Collings.
Coherent technology
Collings segments the coherent optical market into three.
There are high-end coherent designs for long-haul transport developed by optical transport vendors such as Ciena, Cisco, Huawei, Infinera and Nokia.
Then there are designs such as 400ZR developed for data centre interconnect. Here a ‘pretty aggressive’ capability is needed but not full-scale performance.
At the lower end, there are application areas where direct-detect optics is reaching its limit. For example, inside the data centre, campus networks and access networks. Here the right solution is coherent or a ‘coherent-light’ technology that is a compromise between direct detection and full-scale coherence used for the long haul.
“So there is emerging this wide continuum of applications that need an equal continuum of coherent technology,” says Collings.
Now that Lumentum has a DSP capability with the IPG acquisition, it can engage with those applications that need solutions that use coherent but may not need the highest-end performance.
800 gigabits and 1.6 terabits
There is also an ongoing debate about the role of coherent for 800-gigabit and 1.6-terabit transceivers, and Collings says the issues remain unclear.
There’s a range of application requirements: 500m, 2km, and 10km. A direct-detect design may meet the 500m application but struggle at 2k and break down at 10km. “There’s a grey area, just in this simple example,” he says.
Also, the introduction of coherent should be nuanced; what is not needed is a long-haul 5,000km DSP. It is more a coherent-light solution or a borrowing from coherent technologies, says Collings: “You’re still trying to solve a problem that you can almost do with direct detect but not quite.”
The aim is to use the minimum needed to accomplish the goal because the design must avoid paying the cost and power to implement the full complement coherent long-haul.
“So that’s the other part of the grey area: how much you borrow?” he says. “And how much do you need to borrow if you’re dealing with 10km versus 2km, or 800 gigabits versus 1.6 terabits.”
Data centres are already using parallel solutions, so there is always the option to double a design through parallelism.
“Eight hundred gigabit could be the baseline with twice as many lanes as whatever we’re doing at 400 gigabits,” he says. “There is always this brute force approach that you need to best if you’re going to bring in new technologies.”
Optical interconnect
Another area Lumentum is active is addressing the issues of artificial intelligence machine-learning clusters. The machine-learning architectures used must scale at an unprecedented rate and use parallelism in processors, multiple such processors per cluster, and multiple clusters.
Scaling processors requires the scaling of their interconnect. This is driving a shift from copper to optics due to the bandwidth growth involved and the distances: 100, 200 and 400 gigabits and lengths of 30-50 meters, respectively.
The transition to an integrated optical interconnect capability will include VCSELs, co-packaged optics, and much denser optical connectivity to connect the graphic processing units (GPUs) rather than architectures based on pluggables that the industry is so familiar with, says Collings.
Co-packaged optics address a power dissipation interconnect challenge and will likely first be used for proprietary interconnect in very high density GPU artificial intelligence clusters.
Meanwhile, pluggable optics will continue to be used with Ethernet switches. The technology is mature and addresses the needs for at least two more generations.
“There’s an expectation that it’s not if but when the switchover happens to co-packaged optics and the Ethernet switch,” says Collings.
Material systems
Lumentum has expertise in several material systems, including indium phosphide, silicon photonics and gallium arsenide.
All these materials have strengths and weaknesses, he says.
Indium phosphide has bandwidth advantages and is best for light generation. Silicon is largely athermal, highly parallelisable and scalable. Staff joining from NeoPhotonics and IPG have strengthened Lumentum’s silicon photonics expertise.
“The question isn’t silicon photonics or indium phosphide. It’s how you get the best out of both material systems, sometimes in the same device,” says Collings. “Sticking in one sandbox is not going to be as competitive as being agile and having the ability to bring those sandboxes together.”
BT's IP-over-DWDM move
- BT will roll out next year IP-over-DWDM using pluggable coherent optics in its network
- At ECOC 2022, BT detailed network trials that involved the use of ZR+ and XR optics coherent pluggable modules
Telecom operators have been reassessing IP-over-DWDM with the advent of 400-gigabit coherent optics that plug directly into IP routers.
According to BT, using pluggables for IP-over-DWDM means a separate transponder box and associated ‘grey’ (short-reach) optics are no longer needed.
Until now, the transponder has linked the IP router to the dense wavelength-division multiplexing (DWDM) optical line system.
“Here is an opportunity to eliminate unnecessary equipment by putting coloured optics straight onto the router,” says Professor Andrew Lord, BT’s head of optical networking.
Removing equipment saves power and floor space too.
DWDM trends
Operators need to reduce the cost of sending traffic, the cost-per-bit, given the continual growth of IP traffic in their networks.
BT says its network traffic is growing at 30 per cent a year. As a result, the operator is starting to see the limits of its 100-gigabit deployments and says 400-gigabit wavelengths will be the next capacity hike.
Spectral efficiency is another DWDM issue. In the last 20 years, BT has increased capacity by lighting a new fibre pair using upgraded optical transport equipment.
Wavelength speeds have gone from 2.5 to 10, then to 40, 100, and soon 400 gigabits, each time increasing the total traffic sent over a fibre pair. But that is coming to an end, says BT.
“If you go to 1.2 terabits, it won’t go as far, so something has to give,” says Lord. ”So that is a new question we haven’t had to answer before, and we are looking into it.”
Fibre capacity is no longer increasing because coherent optical systems are already approaching the Shannon limit; send more data on a wavelength and it occupies a wider channel bandwidth.
Optical engineers have improved transmission speeds by using higher symbol rates. Effectively, this enables more data to be sent using the same modulation scheme. And keeping the same modulation scheme means existing reaches can still be met. However, upping the symbol rate is increasingly challenging.
Other ways of boosting capacity include making use of more spectral bands of a fibre: the C-band and the L-band, for example. BT is also researching spatial division multiplexing (SDM) schemes.
IP-over-DWDM
IP-over-DWDM is not a new topic, says BT. To date, IP-over-DWDM has required bespoke router coherent cards that take an entire chassis slot, or the use of coherent pluggable modules that are larger than standard QSFP-DD client-side optics ports.
“That would affect the port density of the router to the point where it’s not making the best use of your router chassis,“ says Paul Wright, optical research manager at BT Labs.
The advent of OIF-defined 400ZR optics has catalysed operators to reassess IP-over-DWDM.
The 400ZR standard was developed to link equipment housed in separate data centres up to 120km apart. The 120km reach is limiting for operators but BT’s interest in ZR optics stems from the promise of low-cost, high-volume 400-gigabit coherent optics.
“It [400ZR optics] doesn’t go very far, so it completely changes our architecture,” says Lord. “But then there’s a balance between the numbers of [router] hops and the cost reduction of these components.”
BT modelled different network architectures to understand the cost savings using coherent ZR and ZR+ optics; ZR+ pluggables have superior optical performance compared to 400ZR.
The networks modelled included IP routers in a hop-by-hop architecture where the optical layer is used for point-to-point links between the routers.
This worked well for traffic coming into a hub site but wasn’t effective when traffic growth occurred across the network, says Wright, since traffic cascaded through every hop.
BT also modelled ZR+ optics in a reconfigurable optical add-drop multiplexer (ROADM) network architecture, as well as a hybrid arrangement using both ZR+ and traditional coherent optics. Traditional coherent optics, with its superior optical performance, can pass through a string of ROADM stages where ZR+ optics falls short.
BT compared the cost of the architectures assuming certain reaches for the various coherent optics and published the results in a paper presented at ECOC 2020. The study concluded that ZR and ZR+ optics offer significant cost savings compared to coherent transponders.
ZR+ pluggables have since improved, using higher output powers to better traverse a network’s ROADM stages. “The [latest] ZR+ optics should be able to go further than we predicted,” says Wright.
It means BT is now bought into IP-over-DWDM using pluggable optics.
BT is doing integration tests and plans to roll out the technology sometime next year, says Lord.
XR optics
BT is a member of the Open XR Forum, promoting coherent optics technology that uses optical sub-carriers.
Dubbed XR optics, if all the subs-carriers originate at the same point and are sent to a common destination, the technology implements a point-to-point communication scheme.
Sub-carrier technology also enables traffic aggregation. Each sub-carrier, or a group of sub-carriers, can be sent from separate edge-network locations to a hub where they are aggregated. For example, 16 endpoints, each using a 25-gigabit sub-carrier, can be aggregated at a hub using a 400-gigabit XR optics pluggable module. Here, XR optics is implementing point-to-multipoint communication.
Lord views XR optics as innovative. “If only we could find a way to use it, it could be very powerful,” he says. “But that is not a given; for some applications, XR optics might be too big and for others it may be slightly too small.”
ECOC 2022
BT’s Wright shared the results of recent trial work using ZR+ and XR optics at the recent ECOC 2022 conference, held in Basel in September.
The 400ZR+ were plugged into Nokia 7750 SR-s routers for an IP-over-DWDM trial that included the traffic being carried over a third-party ROADM system in BT’s network. BT showed the -10dBm launch-power ZR+ optics working over the ROADM link.
For Wright, the work confirms that 0dBm launch-power ZR+ optics will be important for network operators when used with ROADM infrastructures.
BT also trialled XR optics where traffic flows were aggregated.
“These emerging technologies [ZR+ and XR optics] open up for the first time the ability to deploy a full IP-over-DWDM solution,” concluded Wright.
Telecoms embraces 400ZR optics for IP-over-DWDM
Verizon Media has trialled 400-gigabit coherent pluggable optics to improve the delivery of video content to subscribers.
Verizon Media added a 400ZR QSFP-DD module from Inphi to a switch already using 100-gigabit optics.
Adding dense wavelength-division multiplexing (DWDM) optics to a switch enables it to send IP traffic (IP-over-DWDM) directly without needing a separate DWDM data centre interconnect box and additional client-side optics to link the two platforms (see diagram).
“Verizon Media, showing leadership outside the hyperscalers, is moving to IP-over-DWDM,” says Tomas Maj, senior director, marketing, optical interconnect at Inphi. “It shows the maturity of the ecosystem and the confidence of more and more operators in IP-over-DWDM and 400ZR.”
Content distribution network
Inphi cites three applications driving traffic growth between data centres: cloud network virtualisation, content distribution and edge analytics, and data mirroring and backup.
The primary users of these applications are the hyperscalers – it is the hyperscalers that spurred the creation of the OIF’s 120km 400ZR standard – but these applications increasingly apply to the telcos.
Verizon Media uses its content delivery network to share and back-up video between its data centres dubbed super PoPs (points-of-presence). Video is also sent to smaller outlying sites, closer to subscribers, where the most popular content is hosted.
ColorZ II
Verizon Media’s network uses Inphi’s existing 100-gigabit ColorZ QSFP28 pluggable optics.
The ColorZ is a direct-detect module that uses 4-level pulse amplitude modulation (PAM-4) to convert 4×25-gigabit electrical signals to two 50-gigabit PAM-4 optical wavelengths that fit within a 100GHz channel.
The ColorZ module, of which Inphi has now shipped over 100,000 units, has an 80km reach.
Inphi’s second-generation ColorZ II uses the OIF’s 400ZR coherent standard. Both generations employ an silicon photonics chip to implement the optics.
“As you go up in PAM-4 speed, you are taking hits in optical signal-to-noise ratio and receiver sensitivity and the design becomes costly,” says Maj. “At some point, you look at coherent and you have better yield and optical performance.”
For Verizon Media’s trial, the ColorZ II 400ZR QSFP-DD was added to switches from Arista Networks. Using ColorZ II optics in the same 100GHz channels quadruples fibre capacity from 4 to 16 terabits while halving the transmission cost-per-bit.
Nitin Batta, principal infrastructure architect at Verizon Media, said in a press release that the ColorZ II was chosen to enable it to “rapidly, easily and cost-effectively add terabits of capacity in response to customer demand.”
The 400ZR standard ensures interoperability and gives customers confidence by having several module companies to choose from, says Maj. Adopting the module also provides important diagnostic information regarding a link’s performance.
All the elements for a 400-gigabit ecosystem are coming together, says Inphi.
Four-hundred-gigabit client-side optical modules are leading the way and now 400-gigabit coherent pluggables are at the testing and validation stage before volume deployment.
The ColorZ II will be generally available at the year’s end.
P-OTS 2.0: 60s interview with Heavy Reading's Sterling Perrin
Q: Heavy Reading claims the metro packet optical transport system (P-OTS) market is entering a new phase. What are the characteristics of P-OTS 2.0 and what first-generation platform shortcomings does it address?
A: I would say four things characterise P-OTS 2.0 and separate 2.0 from the 1.0 implementations:
- The focus of packet-optical shifts from time-division multiplexing (TDM) functions to packet functions.
- Pure-packet implementations of P-OTS begin to ramp and, ultimately, dominate.
- Switched OTN (Optical Transport Network) enters the metro, removing the need for SONET/SDH fabrics in new elements.
- 100 Gigabit takes hold in the metro.
The last two points are new functions while the first two address shortcomings of the previous generation. P-OTS 1.0 suffered because its packet side was seen as sub-par relative to Ethernet "pure plays" and also because packet technology in general still had to mature and develop - such as standardising MPLS-TP (Multiprotocol Label Switching - Transport Profile).
Your survey's key findings: What struck Heavy Reading as noteworthy?
The biggest technology surprise was the tremendous interest in adding IP/MPLS functions to transport. There was a lot of debate about this 10 years ago. Then the industry settled on a de facto standard that transport includes layers 0-2 but no higher. Now, it appears that the transport definition must broaden to include up to layer 3.
A second key finding is how quickly SONET/SDH has gone out of favour. Going forward, it is all about packet innovation. We saw this shift in equipment revenues in 2012 as SONET/SDH spend globally dropped more than 20 percent. That is not a one-time hit - it's the new trend for SONET/SDH.
Heavy Reading argues that transport has broadened in terms of the networking embraced - from layers 0 (WDM) and 1 (SONET/SDH and OTN) to now include IP/MPLS. Is the industry converging on one approach for multi-layer transport optimisation? For example, IP over dense WDM? Or OTN, Carrier Ethernet 2.0 and MPLS-TP? Or something else?
We did not uncover a single winning architecture and it's most likely that operators will do different things. Some operators will like OTN and put it everywhere. Others will have nothing to do with OTN. Some will integrate optics on routers to save transponder capital expenditure, but others will keep hardware separate but tightly link IP and optical layers via the control plane. I think it will be very mixed.
You talk about a spike in 100 Gigabit metro starting in 2014. What is the cause? And is it all coherent or is a healthy share going to 100 Gigabit direct detection?
Interest in 100 Gigabit in the metro exceeds interest in OTN in the metro - which is different from the core, where those two technologies are more tightly linked.
Cloud and data centre interconnect are the biggest drivers for interest in metro 100 Gig but there are other uses as well. We did not ask about coherent versus direct in this survey, but based on general industry discussions, I'd say the momentum is clearly around coherent at this stage - even in the metro. It does not seem that direct detect 100 Gig has a strong enough cost proposition to justify a world with two very different flavours of 100 Gig.
What surprised you from the survey's findings?
It was really the interest-level in IP functionality on transport systems that was the most surprising find.
It opens up the packet-optical transport market to new players that are strongest on IP and also poses a threat to suppliers that were good at lower layers but have no IP expertise - they'll have to do something about that.
Heavy Reading surveyed 114 operators globally. All those surveyed were operators; no system vendors were included. The regional split was North America - 22 percent, Europe - 33 percent, Asia Pacific - 25 percent, and the rest of the world - Latin America mainly - 20 percent.
OTN processors from the core to the network edge
The latest silicon design announcements from PMC and AppliedMicro reflect the ongoing network evolution of the Optical Transport Network (OTN) protocol.
"There is a clear march from carriers, led in particular by China, to adopt OTN in the metro"
Scott Wakelin, PMC
The OTN standard, defined by the telecom standards body of the International Telecommunication Union (ITU-T), has existed for a decade but only recently has it emerged as a key networking technology.
OTN's growing importance is due to the enhanced features being added to the protocol coupled with developments in the network. In particular, OTN enhances capabilities that operators have long been used to with SONET/SDH, while also supporting packet-based traffic. Moreover chip vendors are unveiling OTN designs that now span the core to the network edge.
"OTN switching is a foundational technology in the network"
Michael Adams, Ciena
OTN supports 1 Gigabit Ethernet (GbE) with ODU0 framing alongside ODU1 (2.5G), ODU2 (10G), ODU3 (40G) and ODU4 (100G). The standard packs efficiently client signals such as SONET/SDH, Ethernet, video and Fibre Channel, at the various speed increments up to 100Gbps prior to transmission over lightpaths. Meanwhile, the Optical Internetworking Forum (OIF) has recently developed the OTN-over-Packet-Fabric standard that allows OTN to be switched using packet fabrics.
"OTN switching is a foundational technology in the network," says Michael Adams, Ciena’s vice president of product & technology marketing.
Operator benefits
Whereas 10Gbps services matched 10Gbps lightpaths only a few years ago, transport speeds have now surged ahead. Common services are at 1 and 10 GbE while transport is now at 40Gbps and 100Gbps speeds. OTN switching allows client signals to be combined efficiently to fill the higher capacity lightpaths and avoid stranded bandwidth in the network.
OTN also benefits network connectivity changes. With AT&T's Optical Mesh Service, for example, customers buy a total capacity and, using a web portal, can adapt connectivity between their sites as requirements change. "It [OTN] can manage GbE streams and switch them through the network in an efficient manner," says Adams.
The ability to adapt connectivity is also an important requirement for cloud computing, with OTN switching and a mesh control plane seen as a promising way to enable dynamic networking that provides guaranteed bandwidth when needed, says Ciena.
OTN also offers an alternative to IP-over-DWDM, argues Ciena. By adding a 100Gbps wavelength, service routers can exploit OTN to add 10G services as needed rather than keep adding a 10Gbps wavelength for each service using IP-over-DWDM. "To enable service creation quickly, why not put your router network on top of that network versus running it directly?" says Adams.
OTN hardware announcements
The latest OTN chip announcements from PMC and Applied Micro offer enhanced capacity when aggregating and switching client signals, while also supporting the interfacing to various switch fabrics.
PMC has announced two metro OTN processors, dubbed the HyPHY 20Gflex and 10Gflex. The devices are targeted at compact "pizza boxes" that aggregate residential, enterprise and mobile backhaul traffic, as well as packet-optical and optical transport platforms.
AppliedMicro's TPACK unit has unveiled two additions to its OTN designs: a 100Gbps chipset and the TPO134. The company also announced the general availability of its 100Gbps muxponder and transponder OTN design, now being deployed in the network.
Source: AppliedMicro
"OTN has long had a home in the core of the network," says Scott Wakelin, product manager for HyPHY flex at PMC. "But there is a clear march from carriers, led in particular by China, to adopt OTN in the metro, whether layer-zero or layer-one switched."
Using various market research forecasts, PMC expects the global OTN chip market to reach US $600 million in 2015, the bulk being metro.
PMC and AppliedMicro offer application-specific standard product (ASSP) OTN ICs while AppliedMicro also offers FPGA-based OTN designs.
The benefits of using an FPGA, says AppliedMicro, include time-to-market, the ability to reprogramme the design to accommodate standards’ tweaks, and enabling system vendors to add custom logic elements to differentiate their designs. PMC develops ASSPs only, arguing that such chips offer superior integration, power efficiency and price points.
Both companies, when developing an ASSP, know that the resulting design will be adopted by end customers. When PMC announced its original HyPhy family of devices, seven of the top nine OEMs were developing board designs based on the chip family.
PMC's metro OTN processors
The HyPHY 20Gflex has 16 SFP (up to 5Gbps) and two 10Gbps XFP/SFP+ interfaces, whose streams it can groom using the device's 100Gbps cross-connect. The cross-connect can manipulate streams down to SONET/SDH STS-1/ STM-0 rates and ODU0 (1GbE) OTN channels.
Both ODU0 and ODUflex channels are supported. Before adding ODU0, a Gigabit Ethernet channel could only sit in a 2.5Gbps (ODU1) container, which wastes half the capacity. Similarly by supporting ODUflex, signals such as video can be mapped into frames made up of increments of 1.25Gbps. "For efficient use of resources from the metro into the core, you need to start at the access," said Wakelin.
Source: PMC
The chip also supports the OTN-over-Packet-Fabric protocol. The devices can interface to OTN, SONET/SDH and packet switch fabrics.
The 20Gflex offers 40Gbps of OTN framing and a further 20Gbps of OTN mapping. The OTN mapping is used for those client signals to be fitted into ODU frames. With the additional 40Gbps interfaces that connect to the switch fabric, the total interface throughput is 100Gbps, matching the device's cross-connect capacity.
Other chip features include Fast Ethernet, Gigabit Ethernet and 10GbE MACs for carrier Ethernet transport, and support for timing over packet standards, including IEEE 1588v2 over OTN, used to carry mobile backhaul timing information.
The 10Gflex variant has similar functionality to the 20Gflex but with lower throughput.
PMC is now sampling the HyPHY Gflex devices to lead customers.
AppliedMicro's OTN designs
AppliedMicro's TPACK unit has unveiled two OTN designs: a TPO415/C415 OTN multiplexer chipset for use in 100Gbps packet optical transport line cards, and the TPO134 device used at the network edge.
The two devices combined - the TPO415 and TPOC415 - are implemented using FPGAs, what AppliedMicro dubs softsilicon. The two devices interface between the 100Gbps line side and the switch fabric. The TPO415 takes the OTU4 line side OTN signal and demultiplexes it to the various channel constituents. These can be ODU0, ODU1, ODU2, ODU3, ODU4 and ODUflex - capacity from 1Gbps to 100Gbps.
"The [100Gbps muxponder] design comes with an API that makes it look like one component"
Lars Pedersen, AppliedMicro
The TPOC415 has a 100Gbps, 80-channel segmentation and reassembly function (SAR) compliant with the OIF OTN-over-Packet-Fabric standard. The TPOC415 also has a 100Gbps, 80 channel packet mapper function for the transport of Ethernet and MPLS-TP over ODUk or ODUflex. The device's 100Gbps Interlaken interface is used to connect to the switch fabric for packet switching and ODU cross-connection. The devices can also be used in a standalone fashion for designs where the switch fabric does not use Interlaken, or when working with integrated switches and network processors.
Source: AppliedMicro
"This is the first solution in the market for doing these hybrid functions at 100Gbps," says Lars Pedersen, CTO of AppliedMicro's TPACK.
The second design is the softsilicon TPO134, a 10Gbps add/drop multiplexer that can take in up to 16 clients signals and has two OTU2 interfaces. In between is the cross-connect that supports ODU0, ODU1 and ODUflex channels. Two devices can be combined to support 32 client channels and four OTU2 interfaces. Such a dual-design in a pizza-box system would be used to combine multiple client streams.
Being softsilicon, the TPO134 can also be used for packet optical transport systems. Here by downloading a different FPGA image, the design can also implement the segmentation and reassembly function required for the OIF's OTN-over-Packet-Fabric standard. "The interface to the switch fabric is Interlaken again," says Pedersen.
The TPO134 design doubles the capacity of AppliedMicro's previous add/drop multiplexer designs and is the first to support the OIF standard.
AppliedMicro has also announced the general availability of its 100G muxponder design. The muxponder design is a three-device chipset based on two PQ60 ASSPs and a TPO404 softsilicon design.
The PQ60T devices map 10 and 40Gbps clients into OTN and the TPO404 performs the multiplexing to OTU4 with forward error correction. The client signals supported include SONET/SDH, Ethernet and Fibre Channel. On the line side the design also supports various FEC schemes including an enhanced FEC. The TPO404 differ from the TPOT414/424 devices that link 100GbE and 100Gbps line side.
"The [100Gbps muxponder] design comes with an API [application programming interface] that makes it look [from a software perspective] like one component with some client and line ports, similar to the TPO134 device," says Pedersen.
Further reading:
Transport processors now at 100 Gigabit