Network Booting The Pi 4

We’ve talked about PXE booting the Raspberry Pi 3B+, and then looked at the Raspberry Pi 4 as a desktop replacement. But there’s more! The Pi 4 sports a very useful new feature, the flashable bootloader. Just recently a beta version of that bootloader was released that supports PXE  — booting up over the network — which has become a must-have for those of us who have had consistently bad experiences with root filesystems on SD cards.

Pi with no SD CardWhat are the downsides, I hear you ask? You might see slower speeds going across the network compared to a high quality SD card, particularly with the Pi 4 and its improved SD card slot. PXE does require an Ethernet cable; WiFi is not enough, so you have that restriction to contend with. And finally, this isn’t a portable option — you are tethered to that network cable while running, and tethered to your network to boot at all.

On the other hand, if you’re doing a permanent or semi-permanent install of a Pi, PXE is absolutely a winner. There are few things worse than dragging a ladder out to access a Pi that’s cooked its SD card, not to mention the possibility that you firewalled yourself out of it. Need to start over with a fresh Raspbian image? Easy, just rebuild it on the PXE server and reboot the Pi remotely.

Convinced PXE is for you? Let’s get started!

To boot a Raspberry Pi 4 using PXE, there are a few steps required, starting with updating that bootloader firmware. This means installing Raspbian to an SD card and booting the Pi off of it at least once. From there, we turn to the PXE server to build the remote filesystem and set up the NFS and dnsmasq services. This article draws from a pair of official Pi network boot guides.

Bootloader Update

At the time of writing, the eeprom firmware that supports PXE boot is still in beta. We have to grab that firmware, change the boot order configuration, and then flash it to the onboard chip. Once the Pi 4 is booted off your Raspbian SD card, we can do the following to get the firmware updated:

sudo apt-get update
sudo apt-get upgrade
wget https://github.com/raspberrypi/rpi-eeprom/raw/master/firmware/beta/pieeprom-2019-10-16.bin
rpi-eeprom-config pieeprom-2019-10-16.bin > bootconf.txt
sed -i s/0x1/0x21/g bootconf.txt
rpi-eeprom-config --out pieeprom-2019-10-16-netboot.bin --config bootconf.txt pieeprom-2019-10-16.bin
sudo rpi-eeprom-update -d -f ./pieeprom-2019-10-16-netboot.bin
cat /proc/cpuinfo

That last command should output some information on the Pi itself. We’re interested in the entry for the Pi’s serial number. Take note of the last 8 characters of that code, as we’ll use it later. In my case, it is “0c4c21e5”. That’s all the setup needed for the Pi itself.

Building the Filesystem

Last time we set up a PXE server, we used Centos and a dedicated network. This time, let’s use Debian, and see if we can get PXE working on an existing home network. These instructions should apply for a Raspbian server as well.

The following simply builds the Pi’s filesystem as an NFS share. The Raspbian image we’re using does need two files manually updated, which is why we’re deleting and re-downloading start4.elf and fixup4.dat

sudo apt-get install unzip kpartx dnsmasq nfs-kernel-server
sudo mkdir -p /nfs/raspi1
wget https://downloads.raspberrypi.org/raspbian_latest
unzip raspbian_latest
sudo kpartx -a -v 2019-09-26-raspbian-buster.img
mkdir rootmnt
mkdir bootmnt
sudo mount /dev/mapper/loop0p2 rootmnt/
sudo mount /dev/mapper/loop0p1 bootmnt/
sudo cp -a rootmnt/* /nfs/raspi1/
sudo cp -a bootmnt/* /nfs/raspi1/boot/
cd /nfs/raspi1/boot
sudo rm start4.elf
sudo rm fixup4.dat
sudo wget https://github.com/Hexxeh/rpi-firmware/raw/master/start4.elf
sudo wget https://github.com/Hexxeh/rpi-firmware/raw/master/fixup4.dat

Remember that serial number from earlier? Here’s where it comes into play. The first place a Pi attempts to PXE boot from is a folder named the last 8 characters of that serial number. You’ll want to replace each “0c4c21e5” below with the serial number you found earlier. We’re also using a bind mount so the /boot folder is accessible as part of the tftp service, as well as being mounted as part of the NFS share. The advantage here is that the kernel and rest of the boot code can be updated using apt.

sudo mkdir -p /tftpboot/0c4c21e5
echo "/nfs/raspi1/boot /tftpboot/0c4c21e5 none defaults,bind 0 0" | sudo tee -a /etc/fstab
sudo mount /tftpboot/0c4c21e5
sudo chmod 777 /tftpboot

We’ll create a file in the boot folder to enable SSH, modify the Pi’s fstab so it won’t look for filesystems on the SD card, and finally replace the boot command in cmdline.txt. Be sure to replace nfsroot IP address with the IP of the Debian machine serving as the PXE server.

sudo touch /nfs/raspi1/boot/ssh
sudo sed -i /UUID/d /nfs/raspi1/etc/fstab
echo "console=serial0,115200 console=tty root=/dev/nfs nfsroot=192.168.2.209:/nfs/raspi1,vers=3 rw ip=dhcp rootwait elevator=deadline" | sudo tee /nfs/raspi1/boot/cmdline.txt

Configuring Services

Setting up the NFS share is as easy as adding a line to /etc/exports and starting the services. Do note that we’re not setting up a particularly secure NFS service. This isn’t a problem on a home network where you trust all the computers, but don’t run this setup exposed to the public internet.

echo "/nfs/raspi1 *(rw,sync,no_subtree_check,no_root_squash)" | sudo tee -a /etc/exports
sudo systemctl enable rpcbind
sudo systemctl enable nfs-kernel-server
sudo systemctl restart rpcbind
sudo systemctl restart nfs-kernel-server

We need to add our settings to the dnsmasq config file, which is where most of the magic happens. Let’s talk about that “proxy” setting. What we’re asking dnsmasq to do is watch for DHCP requests, and rather than respond to those requests directly, wait for the primary DHCP server to assign an IP address. If dnsmasq sees a request for PXE information, it will send additional information to inform the PXE-capable device of the PXE server information. The upside is that this approach lets us support PXE booting without modifying the primary DHCP server.

You will need to adjust the dhcp-range setting to match your network. Usually it can be set to the broadcast address of your network, which can be identified using ip addr, looking at the brd entry.

echo 'dhcp-range=192.168.2.255,proxy' | sudo tee -a /etc/dnsmasq.conf
echo 'log-dhcp' | sudo tee -a /etc/dnsmasq.conf
echo 'enable-tftp' | sudo tee -a /etc/dnsmasq.conf
echo 'tftp-root=/tftpboot' | sudo tee -a /etc/dnsmasq.conf
echo 'pxe-service=0,"Raspberry Pi Boot"' | sudo tee -a /etc/dnsmasq.conf

Success!


The final step is to start dnsmasq and watch the log for connections.

sudo systemctl enable dnsmasq
sudo systemctl restart dnsmasq
sudo tail -f /var/log/daemon.log

At this point, you can take the Pi 4 we configured, remove the SD card, and attempt booting it over PXE. You should be able to see the tftp requests for boot files in the log, and finally an NFS mount request. Congratulations, we’ve made it work!

33 thoughts on “Network Booting The Pi 4

  1. The formatting of this article shows that the site goofed again. There are words broken when it should be sentences. The command strings are broken on to two lines when they should nee all on one line.

      1. Yes it does. And I saw that my comment needed work. It seems this keyboard goofed. Normal for it. It’s a USB keyboard from the early part of the century, and this laptop is new….

  2. “compared to a high quality SD card, particularly with the Pi 4 and its improved SD card slot.”
    What has been improved with the SD card performance on the Pi 4? Searching this online has turned up short

  3. Afaik apt-get is old, you should use apt instead. No need for sudo when using wget and personally i would check the SHA256 (or at least SHA1) of the bootloader before flashing anything.

    1. Fair point about checking the SHA256. apt-get and apt seem to be equally recommended in the current Debian documentation. We need to use “sudo wget” because we’re replacing files in the boot folder, which is all owned by root.

      1. Ok for wget, did not notice that.
        For apt-get/apt, you are right. apt-get is older but i can’t find any sign that it’s no longer supported. apt is newer and easier to use apparently. See for example https://itsfoss.com/apt-vs-apt-get-difference/ . Well, i’m not a Linux-Guru, i just repeated what i read on some forum somewhere… Next time i will check before repeating… I will stick with apt, it’s shorter…

    2. for what its worth, I find some vendors that release custom linux distributions only freeze their packages using apt-get. In other words, I’ve found that apt can brick some embedded devices since apt and apt-get dont seen to share the same package freeze db. Looking at you Gemini computer. Of course, this isn’t a reason to use one over the other…but it is a word of caution.

    1. Maybe one of these days I’ll figure out how to run a dual-head setup and write that up. 1 Pi for two workstations. It has plenty of CPU power for it, if you’re just doing educational software.

    2. Oh, is it meant to have dual screens? I thought they put on two micro HDMI connectors so that there would be a spare when the first one breaks.

      I haven’t broken one (yet), but it doesn’t strike me as the most robust connector.

  4. Next steps:

    – Set a second Pi as PXE server for the first Pi; Put PXE boot on it
    – Set a third Pi as PXE server for the second Pi; Put PXE boot on it
    while(true) {
    – Set a ${N} Pi as PXE server for the ${N – 1} Pi; Put PXE boot on it
    – N++
    }

    Jokes aside, my “permanent pi” has a 128MB SD card with the boot partition only, and the root partition is on a USB HDD…

    1. So here’s the funny thing, once the Pi boots, you can connect the wifi to the same network, manually remove the Ethernet route, and run the nfs root over WiFi. It’s not super reliable, but does work.

  5. Hi, Have managed to netowrk boot, except on mine:
    systemctl status rpi-eeprom-update.service
    the service doesn’t start during bootup.
    my fstab is:

    proc /proc proc defaults 0 0
    192.168.0.7:/pxe/9ed78903 /boot nfs defaults,vers=3 0 0

    my cmdline.txt is:

    selinux=0 dwc_otg.lpm_enable=0 console=tty1 rootwait rw nfsroot=192.168.0.7:/pxe/9ed78903/rootfs,v3 ip=dhcp root=/dev/nfs elevator=deadline modprobe.blacklist=bcm2835_v4l2

    is the problem with them? or something else.
    Thanks for your how to, very helpful
    Chris

    1. It looks right. If it’s booting, then the PX stuff is probably all correct. I would troubleshoot the service not starting as being unrelated to network booting.

      I assume you’ve done the basics, like install updates, disable and re-enable the service, etc.

  6. This reminds me of the LTSP project (http://www.ltsp.org/) plus use of EtherBoot (http://etherboot.org/wiki/)

    I have no practical experience here mind you, but could it be possible to bring the WiFi link up early enough during boot-up to somehow redirect the PXE generated ethernet frames over an EtherIP tunnel (https://tools.ietf.org/html/rfc3378) using the WiFi link to a gateway device that de-cap’s and forwards the ethernet frame?

    I’m guessing someone will reply that my idea won’t work given PXE is a BIOS function and not geared for this, but just curious if something similar to the kexec comment someone made could make an equivalent to PXE viable, but without installing a bunch of ethernet/WiFi bridge devices.

    Is it possible to virtualize Raspbian within Raspbian and still retain all the hardware access features? For example, take the earlier super minimal image idea someone mentioned and make that your wireless bridge; booting the 2nd image as a VM with a software defined VLAN between it and the minimal image. A quick search suggests PXE on VMware VMs may be possible at least (https://pubs.vmware.com/vsphere-50/topic/com.vmware.vsphere.vm_admin.doc_50/GUID-ABDA2AC1-9799-4C9C-B2A0-97CBB5E78D68.html)

    Again, no experience here. Just enjoyed the article and throwing out ideas and questions.

      1. Assuming iPXE will work with RPi, then I think the iPXE bootchaining option looks promising. Assuming the Pi4 DHCP request options are the same as the Pi3 (https://www.raspberrypi.org/forums/viewtopic.php?t=209247), configure the DHCP server to serve up the iPXE rom for this vendor class (other options may be required or need to be tested):

        Option 60 (vendor-class-identifier) [sic] – needs to be set to text value “PXEClient”

        Then setup the DHCP server to serve up the stuff in the article for this user class:

        Option 77 (user-class) – needs to be set to text value “iPXE”

        Anyway, this looks promising to me if iPXE is compatible, since it supports WiFi booting. Note: the Pi3 thread above quotes option 60 as vendor-class-identifier; however, IANA lists that as simply “class id”, listing option 124 as “Vendor-Identifying Vendor Class” instead. Just pointing out the name of the DHCP option in that Pi3 solution may be incorrectly referenced, but assume the option listed was what was successfully used.

        https://www.iana.org/assignments/bootp-dhcp-parameters/bootp-dhcp-parameters.xhtml#options

        Hope this helps someone with a Pi4 take the next steps.

  7. Hi Jonathan,

    I’m trying to get PXE working for testing with no success. I read some articles/tutorials and the setup mostly is RPi (server) to RPi (client). With this, can I just ask if PXE will work if the setup is RPi 4 (client) to PC with Rasbian installed (server)? Sorry for newbie question. I’m trying to learn as well.

    Thanks for putting this article.

    Cheers.

      1. Yup, Debian on the PC. Thanks for correcting me.

        Okay. I’ll try again. I already thought of getting another RPi4 for testing in case it’s not supported. I hope to figure out what’s missing on my setup/config.

        Thanks for the feedback, Jonathan.

        All the best!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.