The long arm of PCI Express

Optical is being added as a second physical medium to the PCI Express (PCIe) data transfer protocol.

PCI Express is an electrical standard, but now the Peripheral Component Interconnect Special Interest Group (PCI-SIG) has created a working group to standardise PCIe’s delivery optically.

PCI-SIG is already developing copper cabling specifications for the PCI Express 5.0 and 6.0 standards.

Source: PCI-SIG

Since each generation of PCIe doubles the data transfer rate, PCI-SIG member companies want copper cabling to help with the design of high-speed PCIe interconnects on a printed circuit board (PCB), between PCBs, and between racks (see diagram).

“We've seen a lot of interest over recent months for an optical cable that will support PCI Express,” says Al Yanes, PCI-SIG president and chairperson.

He cites the trends of the decreasing cost and size of optics and how silicon photonics enables the adding of optics alongside ASICs.

“We have formed a workgroup to deliver an optical cable,” says Yanes. “There are many applications, but one is a longer-distance reach for PCI Express.”

“It is a void in the market [the lack of optical support for PCIe], and it needs to be filled,” says Bill Koss, CEO of Drut Technologies. “These efforts tend to take longer than estimated, so better to start sooner.”

Drut has developed a PCIe over photonics solution as part of its photonic direct connect fabric for the data centre.

The data centre is going photonic, says Koss, so there is a need for such working standards as photonics get closer to processors.

The PCIe protocol

PCIe is used widely across many industries.

In the data centre, PCIe is used by general-purpose microprocessors and accelerator chips, such as FPGAs, graphics processing units and AI hardware, to connect to storage and network interface cards.

The PCIe bus uses point-to-point communications based on a simple duplex scheme - serial transmissions in both directions which is referred to as a lane.

The bus can be bundled in various lane configurations - x1, x2, x4, x8, x12, x16 and x32 - with x4, x8 and x16 the configurations most used.

The first two PCIe versions, 1.0 and 2.0, delivered 2.5 and 5 giga transfers-per-second (GT/s) per lane per direction, respectively.

A transfer refers to an encoded bit. The first two PCIe versions use an 8b/10b encoding scheme such that for every ten-bit payload sent, 8 bits are data. This is why the data transfer rates per lane per direction are 2Gbps and 4Gbps (250 and 500 gigabytes per second), respectively.

With PCIe 3.0, the decision was made to increase the transfer rate to 8GT/s per lane, which assumed that no equalisation would be needed to counter inter-symbol interference at that speed. However, equalisation was required, which explains why PCIe 3.0 adopted 8GT/s and not 10GT/s.

Source: PCI-SIG

Another PCIe 3.0 decision was to move to a 128b/130b scheme to reduce the encoding overhead from 20 per cent to over 1 per cent. Now the transfer and bit rates are almost equal from the PCIe 3.0 standard onwards.

PCIe 4.0 doubles the transfer rate from 8GT/s to 16GT/s, while PCIe 5.0 is 32GT/s per lane per direction.

Since then, PCIe 6.0 has been specified, supporting 64GT/s per lane per direction. PCIe 6.0 is the first standard for 4-level pulse amplitude modulation (PAM4) signalling.

Now the PCIe 7.0 specification work is at version 0.3. PCIe 7.0 uses PAM-4 to deliver 128GT/s per lane per direction. The standard is expected to be completed in 2025, with industry adoption in 2027.

Optical cabling for PCIe

The PCI Express 5.0 and 6.0 copper cabling specifications are expected by the year-end. The expected distance using copper cabling and retimers is 5-6m. 

The reach of an optical PCIe standard will ‘go a lot further’ than that, but how far is to be determined.

Yanes says optical cables for PCIe will also save space: “An optical cable is not as bulky nor as thick as a copper cable.”

Whether the optical specification work will support all versions of PCIe is to be determined.

“There's some interest to support them all; the copper solution supports all the negotiations,” says Yanes. “It’s something that needs to be discussed, but, for sure, it will be the higher speeds.”

The working group will decide what optical options to specify. “We know that there are some basic things that we need to do to PCI Express technology to make it support optics,” says Yanes.

The working group aims to make the specification work generic enough that it is ‘optical friendly’.

"There are many optical techniques in the industry, and there is discussion as to which of these optical techniques is going to be the winner in terms of usage," says Yanes. "We want our changes to make PCI Express independent of that discussion."

The organisation will make the required changes to the base specification of PCIe to suit optical transmission while identifying which optical solutions to address and build.

PCI-SIG will use the same Flit Mode and the same link training, for example, while the potential specification enhancements include coordinating speed transitions to match the optics, making side-band signals in-band, and making the specification more power-efficient given the extended reach.

Pluggable optical modules, active optical cables, on-board optics, co-packaged optics and optical input-output are all optical solutions being considered.

An optical solution for PCIe will also benefit technologies such as Compute Express Link (CXL) and the Non-Volatile Memory Express (NVMe) protocols implemented over PCIe. CXL, as it is adopted more broadly, will likely drive new uses that will need such technology.

The PCIe optical working group will complete the specifications in 12-18 months. Yanes says a quicker working solution may be offered before then.