Although we can already buy commercial transceiver solutions that allow us to use PCIe devices like GPUs outside of a PC, these use an encapsulating protocol like Thunderbolt rather than straight PCIe. The appeal of [Sylvain Munaut]’s project is thus that it dodges all that and tries to use plain PCIe with off-the-shelf QSFP transceivers.
As explained in the intro, this doesn’t come without a host of compatibility issues, least of all PCIe device detection, side-channel clocking and for PCIe Gen 3 its equalization training feature that falls flat if you try to send it over an SFP link. Fortunately [Eli Billauer] had done much of the leg work already back in 2016, making Gen 2 PCIe work over SFP+.
The test setup involves a Raspberry Pi 5 on a PCIe breakout board and a PCIe card connected to the whole QSFP intermediate link with custom SFP module PCBs for muxing between PCIe edge connector or USB 3.0 connectors to use those cheap crypto miner adapter boards. The fiber is just simple single-mode fiber. Using this a Gen 2 x1 link can be created without too much fuss, demonstrating the basic principle.
Moving this up to Gen 3 will be challenging and will be featured in future videos, involving more custom PCBs. With Gen 5 now becoming standard on mainboards, it would be great to see this project work for Gen 3 – 5 at link sizes of x4 and even x16 so that it might be able to run external GPUs at full bandwidth unlike Thunderbolt.
Thanks to [zoobab] for the tip.
just wait until you hear about Firewire, or NVMe-oF, but RDMA will surely blow your socks off
Too bad this isn’t just a new standard architecture, so external PCIe racks with dozens of slots can connect to a single motherboard directly to an onboard fiber adapter.
Those cheap USB extenders can be hot or miss… love seeing it lose the copper in the middle
Adnaco make a reasonably priced (~$1k) commercial set of products like this. They’ve had 4x PCIe gen 2 over fibre for a while but seemed to stop there. I’ve always wondered if there was something about gen 3 and above that made it hard to do or if it was just not worth the r&d investment. Will be interesting to see where this home brew version goes!
Interesting, I didn’t see them in my previous searches and compared to other solution it’s indeed not overly overpriced.
But indeed when you go to gen3 more complex stuff pops up as I explain in the video there are equalization phases that won’t survive the optical intermediate.
This is serious andvery interesting. I can see plenty of uses where it is relevant. One obvious would be where to digital acquisition card is in a place where the control system would rather not be (because of radiations, RF or else, for example) or might be easily damaged.
Infiniband can carry PCIe x4, as it is aggregate. We’ve been using that since 2005.
DIY USB-C 4 or Thunderbolt over fiber?
USB over fiber is possible, but quite tricky to get right due to tight timing and some other issues. Especially when you also want to support USB 2 (which is required for being at least somewhat compliant).
An easier solution is often to do PCIe over fiber and then stick a regular USB host controller IC on the other end. What tnt shows here is the complicated part of that, adding an USB host controller to it once he published a finished design will be easy.