The Bus That’s Not A Bus: The Joys Of Hacking PCI Express

PCI Express (PCIe) has been around since 2003, and in that time it has managed to become the primary data interconnect for not only expansion cards, but also high-speed external devices. What also makes PCIe interesting is that it replaces the widespread use of parallel buses with serial links. Instead of having a bus with a common medium (traces) to which multiple devices connect, PCIe uses a root complex that directly connects to PCIe end points.

This is similar to how Ethernet originally used a bus configuration, with a common backbone (coax cable), but modern Ethernet (starting in the 90s) moved to a point-to-point configuration, assisted by switches to allow for dynamic switching between which points (devices) are connected. PCIe also offers the ability to add switches which allows more than one PCIe end point (a device or part of a device) to share a PCIe link (called a ‘lane’).

This change from a parallel bus to serial links simplifies the topology a lot compared to ISA or PCI where communication time had to be shared with other PCI devices on the bus and only half-duplex operation was possible. The ability to bundle multiple lanes to provide less or more bandwidth to specific ports or devices has meant that there was no need for a specialized graphics card slot, using e.g. an x16 PCIe slot with 16 lanes. It does however mean we’re using serial links that run at many GHz and must be implemented as differential pairs to protect signal integrity.

This all may seem a bit beyond the means of the average hobbyist, but there are still ways to have fun with PCIe hacking even if they do not involve breadboarding 7400-logic chips and debugging with a 100 MHz budget oscilloscope, like with ISA buses.

Continue reading “The Bus That’s Not A Bus: The Joys Of Hacking PCI Express”

NVMe Blurs The Lines Between Memory And Storage

The history of storage devices is quite literally a race between the medium and the computing power as the bottleneck of preserving billions of ones and zeros stands in the way of computing nirvana. The most recent player is the Non-Volatile Memory Express (NVMe), something of a hybrid of what has come before.

The first generations of home computers used floppy disk and compact cassette-based storage, but gradually, larger and faster storage became important as personal computers grew in capabilities. By the 1990s hard drive-based storage had become commonplace, allowing many megabytes and ultimately gigabytes of data to be stored. This would drive up the need for a faster link between storage and the rest of the system, which up to that point had largely used the ATA interface in Programmed Input-Output (PIO) mode.

This led to the use of DMA-based transfers (UDMA interface, also called Ultra ATA and Parallel ATA), along with DMA-based SCSI interfaces over on the Apple and mostly server side of the computer fence. Ultimately Parallel ATA became Serial ATA (SATA) and Parallel SCSI became Serial Attached SCSI (SAS), with SATA being used primarily in laptops and desktop systems until the arrival of NVMe along with solid-state storage.

All of these interfaces were designed to keep up with the attached storage devices, yet NVMe is a bit of an odd duck considering the way it is integrated in the system. NVMe is also different for not being bound to a single interface or connector, which can be confusing. Who can keep M.2 and U.2 apart, let alone which protocol the interface speaks, be it SATA or NVMe?

Let’s take an in-depth look at the wonderful and wacky world of NVMe, shall we?

Continue reading “NVMe Blurs The Lines Between Memory And Storage”

Adding PCIe To Your Raspberry Pi 4, The Easier Way

Ever since people figured out that the Raspberry Pi 4 has a PCIe bus, the race was on to be the first to connect a regular PCIe expansion card to a Raspberry Pi 4 SBC. Now [Zak Kemble] has created a new approach, using a bridge PCB that replaces the VL805 USB 3 controller IC. This was also how the original modification by [Tomasz Mloduchowski] worked, only now it comes in a handy (OSHPark) PCB format.

After removing the VL805 QFN package and soldering in the bridge PCB, [Zak] confirmed that everything was hooked up properly and attempted to use the Raspberry Pi 4 with a PCIe extender. This showed that the Raspberry Pi would happily talk with a VL805-based USB 3.0 PCIe expansion card, as well as a Realtek RTL8111-based Ethernet card, but not a number of other PCIe cards. Exactly why this is is still unclear at this point.

As a bonus, [Zak] also found that despite the removal of the VL805 IC from the Raspberry Pi rendering its USB 3 ports useless, one can still use the USB-C ‘power input’ on the SBC as a host controller. This way one can have both PCIe x1 and USB on a Raspberry Pi 4.

This is the third iteration we’ve seen for using PCIe with the Pi. If you’re building on the work of [Thomasz Mloduchowski], which inspired [Colin Riley] to add expanders, and now this excellent hack by [Zak], we want to hear about it!

(Thanks to Itay for the tip)

PCIe Multiplier Expands Raspberry Pi 4 Possibilities

It probably goes without saying that hardware hackers were excited when the Raspberry Pi 4 was announced, but it wasn’t just because there was a new entry into everyone’s favorite line of Linux SBCs. The new Pi offered a number of compelling hardware upgrades, including an onboard PCI-Express interface. The only problem was that the PCIe interface was dedicated to the USB 3.0 controller; but that’s nothing a hot-air rework station couldn’t fix.

We’ve previously seen steady-handed hackers remove the USB 3.0 controller on the Pi 4 to connect various PCIe devices with somewhat mixed results, but [Colin Riley] has raised the bar by successfully getting a PCIe multiplier board working with the diminutive Linux computer. While there are still some software kinks to work out, the results are very promising and he already has  a few devices working.

Getting that first PCIe port added to the Pi 4 is already fairly well understood, so [Colin] just had to follow the example set by hackers such as [Tomasz Mloduchowski]. Sure enough, when he plugged the port multiplier board in (after a bit of what he refers to as “professional wiggling”), the appropriate entry showed up in lspci.

But there was a problem. While the port multiplier board was recognized by the kernel, nothing he plugged into it showed up. Checking the kernel logs, he found messages relating to bus conflicts, and one that seemed especially important: “devices behind bridge are unusable because [bus 02] cannot be assigned for them“. To make a long story short, it turns out that the Raspbian kernel is specifically configured to only allow a single PCI bus.

Fortunately, it’s an easy fix once you know what the problem is. Using the “Device Tree Compiler” tool, [Colin] was able to edit the Raspbian Device Tree file and change the PCI “bus-range” variable from <0x0 0x1> to <0x0 0xff>. From there, it was just a matter of plugging in different devices and seeing what works. Simple things such as USB controllers were no problem, but getting ARM Linux support for the NVIDIA GTX 1060 he tried will have to be a topic for another day.

[Thanks to Paulie for the tip.]

Giving The Pi 4 PCI Express

The release of the Raspberry Pi 4 brought us a new SoC, up to 4 Gigs of memory, and most importantly, got away from that janky USB to USB and Ethernet solution. The Raspberry Pi 4 has a PCI Express interface buried under some chips, and if you’re very good at soldering you can add a PCIe x1 device to the new best single board computer.

[Thomasz] took a look at the Raspberry Pi 4 and realized the new USB 3.0 chip is attached to the PCI Express interface on the SoC. That is, if you remove this chip and you have some very fine wires, you can patch in a real PCI Express slot. Removing the chip is easy enough with a hot air gun, although a few caps did get messed up. Throw that in an ultrasonic cleaner, and you have a blank canvas to work PCI magic.

This hack requires six wires, or three differential pairs, there’s a reference clock, a lane 0 transmit, and a lane zero receive. Working backwards from a PCI Express riser, [Thomasz] traced out these connections and soldered a few wires in. On the Pi side, a few capacitors were required to be compliant with the PCI Express spec, but the soldering isn’t too bad. You can do a lot with a small tip on an iron and a microscope.

The Pi was successfully wired up to a PCI Express riser card, along with the lines for ground, 5V, link reactivation, and a power good signal. The only thing left to do was to plug in a PCI card and test. This didn’t go as well as expected, because the PCI Express adapter didn’t like being enumerated by the Raspberry Pi kernel. In subsequent experiments, an Adaptec SAS controller worked. Does this mean external graphics cards for the Pi? No, not quite; this is only one lane of PCIe, where modern graphics cards require an x16 slot for the best performance. Still, if you’ve ever wanted a SCSI card for a Pi, this is the best option yet.

Catching The (PCIe) Bus

If you are trying to learn about FPGAs, there is only so far you can go with the usual blinking lights and VGA outputs. Eventually, you want to do something more. Although not terribly cheap, you can get FPGA boards in a PCIe form-factor and use them directly with PC software. Is it easy? Well, it isn’t flashing an LED, but there are tools to help. [Angelos Kyriakos] did a Master’s thesis on the very subject and used a project known as RIFFA to help with the task.

RIFFA (Reusable Integration Framework for FPGA Accelerators) is a simple framework for communicating data from a host CPU to an FPGA via a PCI Express bus. The framework requires a PCIe enabled workstation and an FPGA on a board with a PCIe connector. RIFFA supports Windows and Linux, Altera and Xilinx, with bindings for C/C++, Python, MATLAB, and Java. With proper design, RIFFA can transfer quite a bit of data in a short period of time between your computer and your FPGA.

Continue reading “Catching The (PCIe) Bus”

IPhone NVMe Chip Reversed With Custom Breakout Boards

Ever so slowly, the main storage in our computers has been moving from spinning disks, to SSDs over SATA, to Flash drives connected to a PCI something or other. The lastest technology is NVMe — Non-Volitile Memory Express — a horribly named technology that puts a memory controller right on the chip. Intel has a PCI-based NVMe drive out, Samsung recently released an M.2 NVMe drive, and the iPhone 6S and 6S Plus are built around this storage technology.

New chips demand a reverse engineering session, and that’s exactly what [Ramtin Amin] did. He took a few of these chips out of an iPhone, created a board that will read them, and managed to analize the firmware.

Any reverse engineering will begin with desoldering the chip. This is easy enough, with the real trick being getting it working again outside whatever system it was removed from. For this, [Ramtin] built his own PCIe card with a ZIF socket. This socket was custom-made, but the good news is you can buy one from ITEAD. Yes, it is expensive — that’s what you get with a custom-made ZIF socket.

With the chip extracted, a custom PCIe card, and a bit of work with the NVMe implementation for Linux, [Ramtin] had just about everything working. Eventually, he was able to dump the entire file system on the chip, allowing anyone to theoretically back up the data on their iPhone or MacBook Air. Of course, and especially for the iPhone, this data is encrypted. It’s not possible to clone an iPhone using this method, but it is a remarkably deep dive into the hardware that makes our storage tick.”