Network Booting The Pi 4

We’ve talked about PXE booting the Raspberry Pi 3B+, and then looked at the Raspberry Pi 4 as a desktop replacement. But there’s more! The Pi 4 sports a very useful new feature, the flashable bootloader. Just recently a beta version of that bootloader was released that supports PXE  — booting up over the network — which has become a must-have for those of us who have had consistently bad experiences with root filesystems on SD cards.

Pi with no SD CardWhat are the downsides, I hear you ask? You might see slower speeds going across the network compared to a high quality SD card, particularly with the Pi 4 and its improved SD card slot. PXE does require an Ethernet cable; WiFi is not enough, so you have that restriction to contend with. And finally, this isn’t a portable option — you are tethered to that network cable while running, and tethered to your network to boot at all.

On the other hand, if you’re doing a permanent or semi-permanent install of a Pi, PXE is absolutely a winner. There are few things worse than dragging a ladder out to access a Pi that’s cooked its SD card, not to mention the possibility that you firewalled yourself out of it. Need to start over with a fresh Raspbian image? Easy, just rebuild it on the PXE server and reboot the Pi remotely.

Convinced PXE is for you? Let’s get started!

To boot a Raspberry Pi 4 using PXE, there are a few steps required, starting with updating that bootloader firmware. This means installing Raspbian to an SD card and booting the Pi off of it at least once. From there, we turn to the PXE server to build the remote filesystem and set up the NFS and dnsmasq services. This article draws from a pair of official Pi network boot guides.

Bootloader Update

At the time of writing, the eeprom firmware that supports PXE boot is still in beta. We have to grab that firmware, change the boot order configuration, and then flash it to the onboard chip. Once the Pi 4 is booted off your Raspbian SD card, we can do the following to get the firmware updated:

sudo apt-get update
sudo apt-get upgrade
wget https://github.com/raspberrypi/rpi-eeprom/raw/master/firmware/beta/pieeprom-2019-10-16.bin
rpi-eeprom-config pieeprom-2019-10-16.bin > bootconf.txt
sed -i s/0x1/0x21/g bootconf.txt
rpi-eeprom-config --out pieeprom-2019-10-16-netboot.bin --config bootconf.txt pieeprom-2019-10-16.bin
sudo rpi-eeprom-update -d -f ./pieeprom-2019-10-16-netboot.bin
cat /proc/cpuinfo

That last command should output some information on the Pi itself. We’re interested in the entry for the Pi’s serial number. Take note of the last 8 characters of that code, as we’ll use it later. In my case, it is “0c4c21e5”. That’s all the setup needed for the Pi itself.

Building the Filesystem

Last time we set up a PXE server, we used Centos and a dedicated network. This time, let’s use Debian, and see if we can get PXE working on an existing home network. These instructions should apply for a Raspbian server as well.

The following simply builds the Pi’s filesystem as an NFS share. The Raspbian image we’re using does need two files manually updated, which is why we’re deleting and re-downloading start4.elf and fixup4.dat

sudo apt-get install unzip kpartx dnsmasq nfs-kernel-server
sudo mkdir -p /nfs/raspi1
wget https://downloads.raspberrypi.org/raspbian_latest
unzip raspbian_latest
sudo kpartx -a -v 2019-09-26-raspbian-buster.img
mkdir rootmnt
mkdir bootmnt
sudo mount /dev/mapper/loop0p2 rootmnt/
sudo mount /dev/mapper/loop0p1 bootmnt/
sudo cp -a rootmnt/* /nfs/raspi1/
sudo cp -a bootmnt/* /nfs/raspi1/boot/
cd /nfs/raspi1/boot
sudo rm start4.elf
sudo rm fixup4.dat
sudo wget https://github.com/Hexxeh/rpi-firmware/raw/master/start4.elf
sudo wget https://github.com/Hexxeh/rpi-firmware/raw/master/fixup4.dat

Remember that serial number from earlier? Here’s where it comes into play. The first place a Pi attempts to PXE boot from is a folder named the last 8 characters of that serial number. You’ll want to replace each “0c4c21e5” below with the serial number you found earlier. We’re also using a bind mount so the /boot folder is accessible as part of the tftp service, as well as being mounted as part of the NFS share. The advantage here is that the kernel and rest of the boot code can be updated using apt.

sudo mkdir -p /tftpboot/0c4c21e5
echo "/nfs/raspi1/boot /tftpboot/0c4c21e5 none defaults,bind 0 0" | sudo tee -a /etc/fstab
sudo mount /tftpboot/0c4c21e5
sudo chmod 777 /tftpboot

We’ll create a file in the boot folder to enable SSH, modify the Pi’s fstab so it won’t look for filesystems on the SD card, and finally replace the boot command in cmdline.txt. Be sure to replace nfsroot IP address with the IP of the Debian machine serving as the PXE server.

sudo touch /nfs/raspi1/boot/ssh
sudo sed -i /UUID/d /nfs/raspi1/etc/fstab
echo "console=serial0,115200 console=tty root=/dev/nfs nfsroot=192.168.2.209:/nfs/raspi1,vers=3 rw ip=dhcp rootwait elevator=deadline" | sudo tee /nfs/raspi1/boot/cmdline.txt

Configuring Services

Setting up the NFS share is as easy as adding a line to /etc/exports and starting the services. Do note that we’re not setting up a particularly secure NFS service. This isn’t a problem on a home network where you trust all the computers, but don’t run this setup exposed to the public internet.

echo "/nfs/raspi1 *(rw,sync,no_subtree_check,no_root_squash)" | sudo tee -a /etc/exports
sudo systemctl enable rpcbind
sudo systemctl enable nfs-kernel-server
sudo systemctl restart rpcbind
sudo systemctl restart nfs-kernel-server

We need to add our settings to the dnsmasq config file, which is where most of the magic happens. Let’s talk about that “proxy” setting. What we’re asking dnsmasq to do is watch for DHCP requests, and rather than respond to those requests directly, wait for the primary DHCP server to assign an IP address. If dnsmasq sees a request for PXE information, it will send additional information to inform the PXE-capable device of the PXE server information. The upside is that this approach lets us support PXE booting without modifying the primary DHCP server.

You will need to adjust the dhcp-range setting to match your network. Usually it can be set to the broadcast address of your network, which can be identified using ip addr, looking at the brd entry.

echo 'dhcp-range=192.168.2.255,proxy' | sudo tee -a /etc/dnsmasq.conf
echo 'log-dhcp' | sudo tee -a /etc/dnsmasq.conf
echo 'enable-tftp' | sudo tee -a /etc/dnsmasq.conf
echo 'tftp-root=/tftpboot' | sudo tee -a /etc/dnsmasq.conf
echo 'pxe-service=0,"Raspberry Pi Boot"' | sudo tee -a /etc/dnsmasq.conf

Success!


The final step is to start dnsmasq and watch the log for connections.

sudo systemctl enable dnsmasq
sudo systemctl restart dnsmasq
sudo tail -f /var/log/daemon.log

At this point, you can take the Pi 4 we configured, remove the SD card, and attempt booting it over PXE. You should be able to see the tftp requests for boot files in the log, and finally an NFS mount request. Congratulations, we’ve made it work!

106 thoughts on “Network Booting The Pi 4

  1. The formatting of this article shows that the site goofed again. There are words broken when it should be sentences. The command strings are broken on to two lines when they should nee all on one line.

      1. Yes it does. And I saw that my comment needed work. It seems this keyboard goofed. Normal for it. It’s a USB keyboard from the early part of the century, and this laptop is new….

  2. “compared to a high quality SD card, particularly with the Pi 4 and its improved SD card slot.”
    What has been improved with the SD card performance on the Pi 4? Searching this online has turned up short

  3. Afaik apt-get is old, you should use apt instead. No need for sudo when using wget and personally i would check the SHA256 (or at least SHA1) of the bootloader before flashing anything.

    1. Fair point about checking the SHA256. apt-get and apt seem to be equally recommended in the current Debian documentation. We need to use “sudo wget” because we’re replacing files in the boot folder, which is all owned by root.

      1. Ok for wget, did not notice that.
        For apt-get/apt, you are right. apt-get is older but i can’t find any sign that it’s no longer supported. apt is newer and easier to use apparently. See for example https://itsfoss.com/apt-vs-apt-get-difference/ . Well, i’m not a Linux-Guru, i just repeated what i read on some forum somewhere… Next time i will check before repeating… I will stick with apt, it’s shorter…

        1. I have to commend you for admitting you made mistakes, and also thanks for looking up whether or not apt-get is dead. I assumed it was as well, but like you I’ll stick with apt.

    2. for what its worth, I find some vendors that release custom linux distributions only freeze their packages using apt-get. In other words, I’ve found that apt can brick some embedded devices since apt and apt-get dont seen to share the same package freeze db. Looking at you Gemini computer. Of course, this isn’t a reason to use one over the other…but it is a word of caution.

    1. Maybe one of these days I’ll figure out how to run a dual-head setup and write that up. 1 Pi for two workstations. It has plenty of CPU power for it, if you’re just doing educational software.

    2. Oh, is it meant to have dual screens? I thought they put on two micro HDMI connectors so that there would be a spare when the first one breaks.

      I haven’t broken one (yet), but it doesn’t strike me as the most robust connector.

    3. Dual screen is the new standard for most people that can afford two screens. It really makes lots of stuff easier, including coding while looking up the API references on the other screen.

  4. Next steps:

    – Set a second Pi as PXE server for the first Pi; Put PXE boot on it
    – Set a third Pi as PXE server for the second Pi; Put PXE boot on it
    while(true) {
    – Set a ${N} Pi as PXE server for the ${N – 1} Pi; Put PXE boot on it
    – N++
    }

    Jokes aside, my “permanent pi” has a 128MB SD card with the boot partition only, and the root partition is on a USB HDD…

    1. So here’s the funny thing, once the Pi boots, you can connect the wifi to the same network, manually remove the Ethernet route, and run the nfs root over WiFi. It’s not super reliable, but does work.

  5. Hi, Have managed to netowrk boot, except on mine:
    systemctl status rpi-eeprom-update.service
    the service doesn’t start during bootup.
    my fstab is:

    proc /proc proc defaults 0 0
    192.168.0.7:/pxe/9ed78903 /boot nfs defaults,vers=3 0 0

    my cmdline.txt is:

    selinux=0 dwc_otg.lpm_enable=0 console=tty1 rootwait rw nfsroot=192.168.0.7:/pxe/9ed78903/rootfs,v3 ip=dhcp root=/dev/nfs elevator=deadline modprobe.blacklist=bcm2835_v4l2

    is the problem with them? or something else.
    Thanks for your how to, very helpful
    Chris

    1. It looks right. If it’s booting, then the PX stuff is probably all correct. I would troubleshoot the service not starting as being unrelated to network booting.

      I assume you’ve done the basics, like install updates, disable and re-enable the service, etc.

  6. This reminds me of the LTSP project (http://www.ltsp.org/) plus use of EtherBoot (http://etherboot.org/wiki/)

    I have no practical experience here mind you, but could it be possible to bring the WiFi link up early enough during boot-up to somehow redirect the PXE generated ethernet frames over an EtherIP tunnel (https://tools.ietf.org/html/rfc3378) using the WiFi link to a gateway device that de-cap’s and forwards the ethernet frame?

    I’m guessing someone will reply that my idea won’t work given PXE is a BIOS function and not geared for this, but just curious if something similar to the kexec comment someone made could make an equivalent to PXE viable, but without installing a bunch of ethernet/WiFi bridge devices.

    Is it possible to virtualize Raspbian within Raspbian and still retain all the hardware access features? For example, take the earlier super minimal image idea someone mentioned and make that your wireless bridge; booting the 2nd image as a VM with a software defined VLAN between it and the minimal image. A quick search suggests PXE on VMware VMs may be possible at least (https://pubs.vmware.com/vsphere-50/topic/com.vmware.vsphere.vm_admin.doc_50/GUID-ABDA2AC1-9799-4C9C-B2A0-97CBB5E78D68.html)

    Again, no experience here. Just enjoyed the article and throwing out ideas and questions.

      1. Assuming iPXE will work with RPi, then I think the iPXE bootchaining option looks promising. Assuming the Pi4 DHCP request options are the same as the Pi3 (https://www.raspberrypi.org/forums/viewtopic.php?t=209247), configure the DHCP server to serve up the iPXE rom for this vendor class (other options may be required or need to be tested):

        Option 60 (vendor-class-identifier) [sic] – needs to be set to text value “PXEClient”

        Then setup the DHCP server to serve up the stuff in the article for this user class:

        Option 77 (user-class) – needs to be set to text value “iPXE”

        Anyway, this looks promising to me if iPXE is compatible, since it supports WiFi booting. Note: the Pi3 thread above quotes option 60 as vendor-class-identifier; however, IANA lists that as simply “class id”, listing option 124 as “Vendor-Identifying Vendor Class” instead. Just pointing out the name of the DHCP option in that Pi3 solution may be incorrectly referenced, but assume the option listed was what was successfully used.

        https://www.iana.org/assignments/bootp-dhcp-parameters/bootp-dhcp-parameters.xhtml#options

        Hope this helps someone with a Pi4 take the next steps.

  7. Hi Jonathan,

    I’m trying to get PXE working for testing with no success. I read some articles/tutorials and the setup mostly is RPi (server) to RPi (client). With this, can I just ask if PXE will work if the setup is RPi 4 (client) to PC with Rasbian installed (server)? Sorry for newbie question. I’m trying to learn as well.

    Thanks for putting this article.

    Cheers.

      1. Yup, Debian on the PC. Thanks for correcting me.

        Okay. I’ll try again. I already thought of getting another RPi4 for testing in case it’s not supported. I hope to figure out what’s missing on my setup/config.

        Thanks for the feedback, Jonathan.

        All the best!

  8. For anyone attempting this and hanging for about a minute after acquiring an IP address (during which time the green LED periodically flashes) before getting the following messages:

    VFS: Unable to mount root fs via NFS, trying floppy.
    VFS: Cannot open root device “nfs” or unknown-block(2,0): error -6
    Please append a correct “root=” boot option; here are the available partitions:
    Kernel panic – not syncing : VFS: Unable to mount root fs on unknown-block(2,0)

    followed by a kernel panic, this may be caused by the Pi attempting to mount the NFS share via UDP when the server is configured to only accept TCP connections (the default for the CentOS 8 nfs-utils package). There are two possible solutions: either add udp=y under the [nfsd] section in /etc/nfs.conf, or (preferably) add proto=tcp to the nfsroot= options in /boot/cmdline.txt

    Hope that helps!

  9. Thank you!
    I just reproduced this using a Pi2 as the server. Interesting setup. Cleaner than the Pi-Desktop, allowing real usage.
    I’m going to try to set this up for a motionEyeOS setup to obviate the need for the SDCard boot.
    So many possibilities.
    I would suggest something larger than a 16GB storage device on the server, especially if you are going to have multiple machines served off of it.

  10. Additionally, Raspbian now supports NFSv4.1 (NFSv4.2 is not supported yet), which resulted in a 15 second reduction in boot time for me. Here is my current cmdline.txt:

    console=serial0,115200 console=tty root=/dev/nfs nfsroot=192.168.2.209:/srv/nfs/raspi1,vers=4.1,proto=tcp rw ip=dhcp rootwait elevator=deadline

  11. One more gotcha: if the version of Linux you are using has SELinux enabled, you will need to set the appropriate context for the tftp files. You can do this with the following command:

    sudo chcon -R unconfined_u:object_r:tftpdir_rw_t:s0 /path/to/boot/dir

  12. I don’t know what changed, but something did.
    RPi2, Raspbian Lite base for TFTP server.
    RPi4, updated to latest beta bootloader, and downgraded back to the 10/16/19.
    Imported the Raspbian Latest, followed all the instructions above. (as I did earlier this month)
    In log on Pi2

    Jan 23 12:10:24 rpi2-tftp dnsmasq-tftp[2184]: error 0 Early terminate received from 192.168.2.112
    Jan 23 12:10:24 rpi2-tftp dnsmasq-tftp[2184]: failed sending /tftpboot/2ba2d181/start4.elf to 192.168.2.112
    Jan 23 12:10:24 rpi2-tftp dnsmasq-tftp[2184]: sent /tftpboot/2ba2d181/config.txt to 192.168.2.112
    Jan 23 12:10:27 rpi2-tftp dnsmasq-tftp[2184]: sent /tftpboot/2ba2d181/start4.elf to 192.168.2.112
    Jan 23 12:10:27 rpi2-tftp dnsmasq-tftp[2184]: sent /tftpboot/2ba2d181/fixup4.dat to 192.168.2.112
    Jan 23 12:24:02 rpi2-tftp dnsmasq-tftp[2184]: error 0 Early terminate received from 192.168.2.112
    Jan 23 12:24:02 rpi2-tftp dnsmasq-tftp[2184]: failed sending /tftpboot/2ba2d181/start4.elf to 192.168.2.112
    Jan 23 12:24:02 rpi2-tftp dnsmasq-tftp[2184]: sent /tftpboot/2ba2d181/config.txt to 192.168.2.112
    Jan 23 12:24:05 rpi2-tftp dnsmasq-tftp[2184]: sent /tftpboot/2ba2d181/start4.elf to 192.168.2.112
    Jan 23 12:24:05 rpi2-tftp dnsmasq-tftp[2184]: sent /tftpboot/2ba2d181/fixup4.dat to 192.168.2.112
    
    
    pi@rpi4-1:~ $ sudo rpi-eeprom-update
    BCM2711 detected
    *** UPDATE REQUIRED ***
    BOOTLOADER: update required
    CURRENT: Wed 16 Oct 2019 05:00:03 PM UTC (1571245203)
     LATEST: Fri 17 Jan 2020 05:37:11 PM UTC (1579282631)
    VL805: up-to-date
    CURRENT: 000137ad
     LATEST: 000137ad
    

    Any ideas?

    1. Updated /boot/cmdline.txt to
      console=serial0,115200 console=tty root=/dev/nfs nfsroot=192.168.2.209:/nfs/raspi1,vers=4.1,proto=tcp rw ip=dhcp rootwait elevator=deadline

      Jan 23 12:43:08 rpi2-tftp dnsmasq-tftp[400]: sent /tftpboot/2ba2d181/start4.elf to 192.168.2.112
      Jan 23 12:43:08 rpi2-tftp dnsmasq-tftp[400]: sent /tftpboot/2ba2d181/fixup4.dat to 192.168.2.112
      Jan 23 12:43:41 rpi2-tftp systemd[1]: dev-serial1.device: Job dev-serial1.device/start timed out.
      Jan 23 12:43:41 rpi2-tftp systemd[1]: Timed out waiting for device /dev/serial1.
      Jan 23 12:43:41 rpi2-tftp systemd[1]: Dependency failed for Configure Bluetooth Modems connected by UART.
      Jan 23 12:43:41 rpi2-tftp systemd[1]: hciuart.service: Job hciuart.service/start failed with result 'dependency'.
      Jan 23 12:43:41 rpi2-tftp systemd[1]: dev-serial1.device: Job dev-serial1.device/start failed with result 'timeout'.
      Jan 23 12:43:41 rpi2-tftp systemd[1]: Reached target Multi-User System.
      Jan 23 12:43:41 rpi2-tftp systemd[1]: Reached target Graphical Interface.
      Jan 23 12:43:41 rpi2-tftp systemd[1]: Starting Update UTMP about System Runlevel Changes...
      Jan 23 12:43:42 rpi2-tftp systemd[1]: systemd-update-utmp-runlevel.service: Succeeded.
      Jan 23 12:43:42 rpi2-tftp systemd[1]: Started Update UTMP about System Runlevel Changes.
      Jan 23 12:43:42 rpi2-tftp systemd[1]: Startup finished in 4.751s (kernel) + 1min 32.394s (userspace) = 1min 37.146s.
      

      and hangs.

      1. Thanks. I’ve built it 3 times today, clean installs, on a Pi2 and a PI3 Rasbian Buster Lite & fully updated. Same result each time.
        Currently building a Debian Buster VBox image. It’s using 4 processors, 4GB RAM, 50GB HD. I will also try an Ubuntu 16.04 image in a bit, to confirm not a RASPBIAN issue.

          1. That fixed it, thanks!
            Now to auto-install a RAMDisk (RPi4-4GB) and add NAS connection.
            There are some advantages to this over PiServer, but I think both are going to be in my tool box.

        1. With the unique serial numbers for the file systems, and using a 1TB USB drive or filesystem, giving each a 25GB filesystem, I can get 40 on a single share. Not the best way, maybe, but there are alternatives, Pi-Net, and Raspbian for Desktops, that also work.

  13. Has anyone else had an error in their /var/log/syslog.log saying:

    dnsmasq-tftp[31420]: file /tftpboot/bootcode.bin not found
    dnsmasq-tftp[31420]: file /tftpboot/bootsig.bin not found

    The only difference I made to the above instructions was to use the latest pieeprom file “pieeprom-2020-01-17.bin”.

    I found the following RPi guide which mentions network boot looking for bootcode.bin too – https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net.md

    Here are the rest of the logs if it helps:
    dnsmasq-dhcp[31420]: 653460281 available DHCP subnet: 192.168.1.255/255.255.255.0
    dnsmasq-dhcp[31420]: 653460281 vendor class: PXEClient:Arch:00000:UNDI:002001
    dnsmasq-dhcp[31420]: 653460281 PXE(enp3s0) b8:27:eb:22:a3:19 proxy
    dnsmasq-dhcp[31420]: 653460281 tags: enp3s0
    dnsmasq-dhcp[31420]: 653460281 broadcast response
    dnsmasq-dhcp[31420]: 653460281 sent size: 1 option: 53 message-type 2
    dnsmasq-dhcp[31420]: 653460281 sent size: 4 option: 54 server-identifier 192.168.199.104
    dnsmasq-dhcp[31420]: 653460281 sent size: 9 option: 60 vendor-class 50:58:45:43:6c:59:55:5e:34
    dnsmasq-dhcp[31420]: 653460281 sent size: 17 option: 97 client-machine-id 00:39:a0:02:b1:39:a0:02:b1:39:a0:02:b1:39…
    dnsmasq-dhcp[31420]: 653460281 sent size: 32 option: 43 vendor-encap 06:01:03:0a:04:00:50:58:45:09:14:00:00:11…
    dnsmasq-tftp[31420]: file /tftpboot/bootcode.bin not found
    dnsmasq-tftp[31420]: file /tftpboot/bootsig.bin not found

    TIA

    1. Or, if all else fails, you can put a copy of bootcode.bin in /tftpboot/. I have 4 booting right now, a Pi4 (no SDCard), a Pi3B+ (no SDCard) and 2 Pi3Bs, one of which refuses to boot without a local copy of bootcode.bin on a SDCard, but the other boots if I include the bootcode.bin in the tftpboot folder in my server. All 4 the proceed to boot fine. I don’t know what the difference is in the Pi3Bs, (am willing to look if someone tells me where) both report
      $ vcgencmd otp_dump | grep 17:
      17:3020000a
      both boot USB normally. I’ve swapped PS, network cables, locations. One netboots clean, the other requires the SDCard bootcode.bin
      I’m doing this for 2 friends, one a school teacher, (who doesn’t like the limitations of Pi-Net or RPiDesktop) and an HR/IT Instructor at a corporation. I’m using a RPi2 with /tftpboot & /nfs on an external USB 1TB drive. Working well over a 1GB Ethernet network. All are almost as fast as native.

  14. Thanks for this tutorial! I am not sure what my issue is. I followed the instructions and double-checked my work. I triple-checked the serial number of the pi and can confirm I have an empty ssh file in /tftpboot/a9bf2d66

    Here is the contents of the output of tail command when I power up the raspberry pi. Does anything stand out to you experts that I am doing something wrong? I can ping the IP address that is assigned but nothing else such as ssh.

    Mar 1 08:45:00 desktop-v33oins dnsmasq-tftp[503]: failed sending /tftpboot/a9bf2d66/kernel8.img to 192.168.86.250
    Mar 1 08:45:00 desktop-v33oins dnsmasq-tftp[503]: file /tftpboot/a9bf2d66/kernel8-32.img not found
    Mar 1 08:45:00 desktop-v33oins dnsmasq-tftp[503]: error 0 Early terminate received from 192.168.86.250
    Mar 1 08:45:00 desktop-v33oins dnsmasq-tftp[503]: failed sending /tftpboot/a9bf2d66/kernel7l.img to 192.168.86.250
    Mar 1 08:45:00 desktop-v33oins dnsmasq-tftp[503]: file /tftpboot/a9bf2d66/armstub8-32-gic.bin not found
    Mar 1 08:45:01 desktop-v33oins dnsmasq-dhcp[503]: 1583070300 available DHCP subnet: 192.168.86.255/255.255.255.0
    Mar 1 08:45:01 desktop-v33oins dnsmasq-tftp[503]: sent /tftpboot/a9bf2d66/kernel7l.img to 192.168.86.250
    Mar 1 08:45:09 desktop-v33oins dnsmasq-dhcp[503]: 3257136223 available DHCP subnet: 192.168.86.255/255.255.255.0
    Mar 1 08:45:09 desktop-v33oins dnsmasq-dhcp[503]: 3257136223 available DHCP subnet: 192.168.86.255/255.255.255.0
    Mar 1 08:45:31 desktop-v33oins dnsmasq-dhcp[503]: 1583070330 available DHCP subnet: 192.168.86.255/255.255.255.0
    Mar 1 08:45:43 desktop-v33oins dnsmasq-dhcp[503]: 2575688061 available DHCP subnet: 192.168.86.255/255.255.255.0
    Mar 1 08:45:43 desktop-v33oins dnsmasq-dhcp[503]: 2575688061 vendor class: PXEClient:Arch:00000:UNDI:002001
    Mar 1 08:45:43 desktop-v33oins dnsmasq-dhcp[503]: 2575688061 PXE(enp3s0) dc:a6:32:64:e8:67 proxy
    Mar 1 08:45:43 desktop-v33oins dnsmasq-dhcp[503]: 2575688061 tags: enp3s0
    Mar 1 08:45:43 desktop-v33oins dnsmasq-dhcp[503]: 2575688061 broadcast response
    Mar 1 08:45:43 desktop-v33oins dnsmasq-dhcp[503]: 2575688061 sent size: 1 option: 53 message-type 2
    Mar 1 08:45:43 desktop-v33oins dnsmasq-dhcp[503]: 2575688061 sent size: 4 option: 54 server-identifier 192.168.86.24
    Mar 1 08:45:43 desktop-v33oins dnsmasq-dhcp[503]: 2575688061 sent size: 9 option: 60 vendor-class 50:58:45:43:6c:69:65:6e:74
    Mar 1 08:45:43 desktop-v33oins dnsmasq-dhcp[503]: 2575688061 sent size: 17 option: 97 client-machine-id 00:66:2d:bf:a9:66:2d:bf:a9:66:2d:bf:a9:66…
    Mar 1 08:45:43 desktop-v33oins dnsmasq-dhcp[503]: 2575688061 sent size: 32 option: 43 vendor-encap 06:01:03:0a:04:00:50:58:45:09:14:00:00:11…
    Mar 1 08:45:43 desktop-v33oins dnsmasq-dhcp[503]: 2575688061 available DHCP subnet: 192.168.86.255/255.255.255.0
    Mar 1 08:45:43 desktop-v33oins dnsmasq-tftp[503]: error 0 Early terminate received from 192.168.86.250
    Mar 1 08:45:43 desktop-v33oins dnsmasq-tftp[503]: failed sending /tftpboot/a9bf2d66/start4.elf to 192.168.86.250
    Mar 1 08:45:43 desktop-v33oins dnsmasq-tftp[503]: sent /tftpboot/a9bf2d66/config.txt to 192.168.86.250
    Mar 1 08:45:43 desktop-v33oins dnsmasq-tftp[503]: file /tftpboot/a9bf2d66/recover4.elf not found
    Mar 1 08:45:43 desktop-v33oins dnsmasq-tftp[503]: file /tftpboot/a9bf2d66/recovery.elf not found
    Mar 1 08:45:44 desktop-v33oins dnsmasq-tftp[503]: sent /tftpboot/a9bf2d66/start4.elf to 192.168.86.250
    Mar 1 08:45:44 desktop-v33oins dnsmasq-tftp[503]: sent /tftpboot/a9bf2d66/fixup4.dat to 192.168.86.250
    Mar 1 08:45:44 desktop-v33oins dnsmasq-tftp[503]: file /tftpboot/a9bf2d66/recovery.elf not found
    Mar 1 08:45:44 desktop-v33oins dnsmasq-tftp[503]: sent /tftpboot/a9bf2d66/config.txt to 192.168.86.250
    Mar 1 08:45:44 desktop-v33oins dnsmasq-tftp[503]: file /tftpboot/a9bf2d66/dt-blob.bin not found
    Mar 1 08:45:44 desktop-v33oins dnsmasq-tftp[503]: file /tftpboot/a9bf2d66/recovery.elf not found
    Mar 1 08:45:44 desktop-v33oins dnsmasq-tftp[503]: sent /tftpboot/a9bf2d66/config.txt to 192.168.86.250
    Mar 1 08:45:44 desktop-v33oins dnsmasq-tftp[503]: file /tftpboot/a9bf2d66/bootcfg.txt not found
    Mar 1 08:45:44 desktop-v33oins dnsmasq-tftp[503]: sent /tftpboot/a9bf2d66/bcm2711-rpi-4-b.dtb to 192.168.86.250
    Mar 1 08:45:45 desktop-v33oins dnsmasq-tftp[503]: sent /tftpboot/a9bf2d66/config.txt to 192.168.86.250
    Mar 1 08:45:45 desktop-v33oins dnsmasq-tftp[503]: sent /tftpboot/a9bf2d66/overlays/vc4-fkms-v3d.dtbo to 192.168.86.250
    Mar 1 08:45:45 desktop-v33oins dnsmasq-tftp[503]: sent /tftpboot/a9bf2d66/cmdline.txt to 192.168.86.250
    Mar 1 08:45:47 desktop-v33oins dnsmasq-tftp[503]: file /tftpboot/a9bf2d66/recovery8.img not found
    Mar 1 08:45:47 desktop-v33oins dnsmasq-tftp[503]: file /tftpboot/a9bf2d66/recovery8-32.img not found
    Mar 1 08:45:47 desktop-v33oins dnsmasq-tftp[503]: file /tftpboot/a9bf2d66/recovery7l.img not found
    Mar 1 08:45:47 desktop-v33oins dnsmasq-tftp[503]: file /tftpboot/a9bf2d66/recovery7.img not found
    Mar 1 08:45:47 desktop-v33oins dnsmasq-tftp[503]: file /tftpboot/a9bf2d66/recovery.img not found
    Mar 1 08:45:47 desktop-v33oins dnsmasq-tftp[503]: file /tftpboot/a9bf2d66/kernel8-32.img not found
    Mar 1 08:45:47 desktop-v33oins dnsmasq-tftp[503]: error 0 Early terminate received from 192.168.86.250
    Mar 1 08:45:47 desktop-v33oins dnsmasq-tftp[503]: failed sending /tftpboot/a9bf2d66/kernel8.img to 192.168.86.250
    Mar 1 08:45:47 desktop-v33oins dnsmasq-tftp[503]: file /tftpboot/a9bf2d66/armstub8-32-gic.bin not found
    Mar 1 08:45:47 desktop-v33oins dnsmasq-tftp[503]: error 0 Early terminate received from 192.168.86.250
    Mar 1 08:45:47 desktop-v33oins dnsmasq-tftp[503]: failed sending /tftpboot/a9bf2d66/kernel7l.img to 192.168.86.250
    Mar 1 08:45:48 desktop-v33oins dnsmasq-tftp[503]: sent /tftpboot/a9bf2d66/kernel7l.img to 192.168.86.250

    1. I think I figured out my problem. The serial number was one digit off, (although there was some confusion on this from the tutorial that perhaps I didn’t understand). It appears that it needs to be the last 8 digits of the mac address on the NIC? When I ran the command to get the serial number, it was one digit off.

      1. Thanks. I got it working. What are you all using to copy custom images to the pxe server (I want to Pxe boot a pi running unbound for dns). I built an an image with everything I want and I want to copy it somehow to the tftp folder that’s used when a new pi comes online.

        1. I use the Pi-Gen and have it create the Noobs images.
          You should be able to copy the /root and /boot folders to the /tftpboot/ folder just like the process here.

          1. Ugh. I have a newb request here…. so, on the Pi that is running the way I want, what files/folders do I copy and where exactly do they belong on the pxe server? I am so confused where they actually get served from. I’ve wasted just about the whole day re-doing things and now my pxe server is broken again. Thanks. hope you can reply.

          2. Yes. Precisely. What I ended up doing was following this tutorial (which gave me a base Raspbian image) and then I just installed Unbound DNS onto the Pi that was running on PXE boot. This worked great. I had a disconnect in my brain thinking that if I installed anything on the pxe booted Pi that those “files” would only be in memory somehow and not on the PXE server itself. So, I think I’m all set now! Thank you!

  15. So, I am so glad that the author chose to write this tutorial. I was successful! Yea!

    So I am wondering, how does one go about making the NUC set up so that all I have to do is plug it into the switch and have it load the correct image via pxe boot? In other words, the next time I want to add a Pi to the rack, all I need to do is . Do I need to go through this whole process again?

  16. So, not 100% related to your PXE boot tutorial (thank you btw). I got it all running in my test lab on my local router’s LAN. In my local LAN, I had dhcp-range=192.168.86.255,proxy This all worked great in the test lab. Then, I go to put this into production and now have a situation where I need to be in a different subnet. Problem is, I need the NUC to be a dhcp-relay agent to my “real” dhcp server. So, in dnsmasq.conf, I put dhcp-relay=,. I cannot get this to work and there are no firewall rules that would block dhcp traffic. So, since that not work, I try to just make dnsmasq be the dhcp server by putting dhcp-range=66.54.97.3,66.54.97.4,255.255.255.224,24h

    So, doing that allows the two PI I have to obtain a lease (from the NUC). The NUC can ping them but I cannot ping them from outside the NUC switch area. All my routes look ok.

    Do you have any tips to get this to work properly with a dhcp-relay option? I would rather not have the NUC be a dhcp server itself.

    Thanks!

      1. Nuc only has one port. Both the NUC and PI are connected to the same switch which is attached to a router that is the default gateway for both the NUC and PI. The Dhcp server hangs off of the same router but on a different interface and subnet

        1. OK, I think you’ll want to comment out dhcp-relay, and add the proxy option back onto your dhcp-range option. I *think* dhcp-relay would be used if your NUC had two network ports, upstream network on one, and PIs on the other.

          I’d try to break the problem into discrete chunks. Start by getting the NUC to pull an IP over DHCP from your real DHCP server. Then get the dhcp proxying working with the PXE PIs. After that, you can tackle the routing.

          Doing the routing is going to be the tricky part. If all the devices are using the same device as their default gateway, it’s much easier, but can still be a pain to get working.

          1. So, I did solve my first problem (with the help of a colleague). I’ll share it here. I added this line into the dnsmasq.conf file. The option 3 and the IP after is the gateway. So, now the dhcp works and I do not have to use relay either.

            dhcp-option=3,66.54.97.1

            Now, something else is going on though. It is very puzzling. The PI do the PXE boot and they are pingable. However, I cannot SSH into them (there’s no firewall in place, so port 22 is not being blocked), and (this is the really weird part), after about 5 minutes, I can’t ping the PI any more! They are still powered up but the PI don’t show up in the switch’s MAC table any more.

            Any ideas?

  17. Has anyone seen this error in /var/log/syslog:
    pr 13 12:43:56 node03 kernel: [ 6398.040763] mmc0: Timeout waiting for hardware cmd interrupt.
    Apr 13 12:43:56 node03 kernel: [ 6398.040774] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
    Apr 13 12:43:56 node03 kernel: [ 6398.040787] mmc0: sdhci: Sys addr: 0x00000000 | Version: 0x00001002
    Apr 13 12:43:56 node03 kernel: [ 6398.040797] mmc0: sdhci: Blk size: 0x00000000 | Blk cnt: 0x00000000
    Apr 13 12:43:56 node03 kernel: [ 6398.040807] mmc0: sdhci: Argument: 0x00000000 | Trn mode: 0x00000000
    Apr 13 12:43:56 node03 kernel: [ 6398.040817] mmc0: sdhci: Present: 0x1fff0001 | Host ctl: 0x00000001
    Apr 13 12:43:56 node03 kernel: [ 6398.040826] mmc0: sdhci: Power: 0x0000000f | Blk gap: 0x00000080
    Apr 13 12:43:56 node03 kernel: [ 6398.040835] mmc0: sdhci: Wake-up: 0x00000000 | Clock: 0x0000f447
    Apr 13 12:43:56 node03 kernel: [ 6398.040844] mmc0: sdhci: Timeout: 0x00000000 | Int stat: 0x00000000
    Apr 13 12:43:56 node03 kernel: [ 6398.040854] mmc0: sdhci: Int enab: 0x00ff1003 | Sig enab: 0x00ff1003
    Apr 13 12:43:56 node03 kernel: [ 6398.040863] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
    Apr 13 12:43:56 node03 kernel: [ 6398.040873] mmc0: sdhci: Caps: 0x45ee6432 | Caps_1: 0x0000a525
    Apr 13 12:43:56 node03 kernel: [ 6398.040882] mmc0: sdhci: Cmd: 0x00000502 | Max curr: 0x00080008
    Apr 13 12:43:56 node03 kernel: [ 6398.040891] mmc0: sdhci: Resp[0]: 0x00000000 | Resp[1]: 0x00000000
    Apr 13 12:43:56 node03 kernel: [ 6398.040900] mmc0: sdhci: Resp[2]: 0x00000000 | Resp[3]: 0x00000000
    Apr 13 12:43:56 node03 kernel: [ 6398.040908] mmc0: sdhci: Host ctl2: 0x00000000
    Apr 13 12:43:56 node03 kernel: [ 6398.040917] mmc0: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x00000000
    Apr 13 12:43:56 node03 kernel: [ 6398.040925] mmc0: sdhci: ============================================

  18. Hi, I have some troubles understanding the command: sudo mount /tftpboot/0c4c21e5
    It gives me an error. Shouldn’t it be: sudo mount –bind /boot /tftpboot/0c4c21e5
    (Yes, replace serial with the one from my Pi)

    Your thoughts please.

    Thx. Marco

    1. Hey Marco.

      The command just above that one adds an entry to your system’s fstab, giving the system the details about what to mount in that folder. When you run mount with just the single argument like that, it should automatically pull the rest of the details from fstab. If you’re getting an error, I would first check that the fstab entry is correct.

    1. setup_iscsiroot (){
      if mount | grep /dev/sda > /dev/null; then
      echo “”
      else
      #sudo apt-get -qy install open-iscsi initramfs-tools rsync
      iscsiadm -m discovery -t sendtargets -p 192.168.1.2
      source /etc/iscsi/initiatorname.iscsi
      echo ‘ISCSI_INITIATOR=’$(echo $InitiatorName) > /etc/iscsi/iscsi.initramfs
      echo ‘ISCSI_TARGET_NAME=iqn.2000-01.com.:NAS.Target-‘$(hostname) >> /etc/iscsi/iscsi.initramfs
      echo ‘ISCSI_TARGET_IP=192.168.1.2’ >> /etc/iscsi/iscsi.initramfs
      iscsiadm -m node -l -T iqn.2000-01.com.:NAS.Target-$(hostname) -p 192.168.1.2
      [ -d “/mnt/iscsi” ] || mkdir -p /mnt/iscsi
      [ -d “/mnt/iscsi/usr/bin” ] || mkfs.ext4 /dev/sda;
      mount /dev/sda /mnt/iscsi;
      mkdir -p /mnt/iscsi/dev;
      rsync -avhP –exclude /boot –exclude /proc –exclude /sys –exclude /dev –exclude /mnt / /mnt/iscsi
      cp /boot/cmdline.txt /boot/cmdline.txt.bak
      sed -i ‘1 s/^/#/’ /boot/cmdline.txt
      echo ‘console=serial0,115200 console=tty1 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait ip=dhcp root=/dev/sda ISCSI_INITIATOR=’$(echo $InitiatorName)’ ISCSI_TARGET_NAME=iqn.2000-01.com.:NAS.Target-‘$(hostname)’ ISCSI_TARGET_IP=192.168.1.2 rw’ >> /boot/cmdline.txt
      cd /boot && update-initramfs -c -k $(uname -r)
      grep -qxF ‘initramfs initrd.img-‘$(uname -r)’ followkernel’ /boot/config.txt || echo ‘initramfs initrd.img-‘$(uname -r)’ followkernel’ >> /boot/config.txt
      touch /var/lib/iscsi-complete
      sudo reboot
      exit
      fi < hope this helps

    1. I have 24 Raspberry Pi 4Bs using this method to boot through the network. The last 8 added to the cluster are the new 8GB version running Raspberry OS 64 bit:

      Output from /etc/os-release:

      PRETTY_NAME=”Debian GNU/Linux 10 (buster)”
      NAME=”Debian GNU/Linux”
      VERSION_ID=”10″
      VERSION=”10 (buster)”
      VERSION_CODENAME=buster
      ID=debian

      The cluster is crunching COVID-19 work units from Rosetta@Home. All nodes run all four processors at 100% for 100% of the time and they are doing great. The NFS server is a Dell XPS 8700 running debian as well.

      The Cluster at work:

      https://boinc.bakerlab.org/rosetta/hosts_user.php?userid=2138265

  19. Very nice approach to this. I made a few tweaks – my main router provides the dnsmasq service tied to dhcp – so it provides the persistent mac -> ip mappings. I set aside one raspberry pi with attached ssd to provide both the tftp and nsf shares to my net-booting cluster nodes (I am using tftpd-hpa). I have configured the pi’s in my cluster to use the bootloader tftp_ip setting to point to my tftp server (which simplifies the router dhcp configuration.)

  20. Hey i tried to run:

    sudo kpartx -a -v 2020-02-13-raspbian-buster.img

    and git this error:

    “””/dev/mapper/control: open failed: No such device Failure to communicate with kernel device-mapper driver. Check that device-mapper is available in the kernel. Incompatible libdevmapper 1.02.155 (2018-12-18) and kernel driver (unknown version). device mapper prerequisites not met”””

    any sugestions

  21. Hi,
    I wonder why my Pi doesn’t get the TFTP address from dnsmasq. When I set it on my Pi’s eprom configuration, it find the server and attempts to download the files, but it never gets it when relying on dnsmasq. It does get the pie-service title (“Raspberry Pi boot”), but not the TFTP address.
    What could be wrong here?
    Best,
    Francis

      1. I’d probably try to fire up Wireshark to catch the DHCP packets, to look for a clue there. Since DHCP is sent as broadcast, you don’t have to do any fancy port duplication to see the packets.

  22. Hi!

    I would like a description in the next step of how to load at least the core system into RAM so that i can do what in the event of a network outage. :-/

    Thank you!!! :-)

      1. Thank You! :-)

        Yes, it is not trivial. :-D

        My problem is that when the connection to the server is lost, I can’t restart raspberry because the reboot command doesn’t work. :-(

  23. Hi,

    I have my hard drive connected to the USB port on my router. I have a single RPi 4B. Is it possible, to network boot my RPi 4B from this hard drive; or do I need to have an additional RPi, to act as a server?

    Many thanks in advance.

    Regards,
    Tony.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.