Hack My House: Running Raspberry Pi Without an SD Card

Many of us have experienced the pain that is a Raspberry Pi with a corrupted SD card. I suspect the erase-on-write nature of flash memory is responsible for much of the problem. Regardless of the cause, one solution is to use PXE booting with the Raspberry Pi 3. That’s a fancy way to say we’ll be booting the Raspberry Pi over the network, instead of from an SD card.

What does this have to do with Hacking My House? As I discussed last time, I’m using Raspberry Pi as Infrastructure by building them into the walls of every room in my house. You don’t want to drag out a ladder and screwdriver to swap out a misbehaving SD card, so booting over the network is a really good solution. I know I promised we’d discuss cabling and cameras. Think of this as a parenthetical article — we’ll talk about Ethernet and ZoneMinder next time.

So let’s dive in and see what the Preboot Execution Environment (PXE) is all about and how to use PXE with Raspberry Pi.

Getting Familiar with Raspberry Pi’s PXE Boot Feature

New Raspberry Pis ship with PXE boot enabled, allowing the Pi to load its file system from a server on the same network. My experience is that the Pi 3 model B+ boots more reliably than the older version. Either way, it’s a good idea to begin by running rpi-update. Additionally, you may need to manually enable PXE by booting once off an SD card with an option added to config.txt. The official write-up is a great guide, and one of the resources I used when I stumbled through the PXE process for the first time. However, I will be doing things just a bit differently.

Take a moment to consider your network layout. Will your Raspberry Pi connect to your existing network, or to a new, dedicated network? To use your primary network, your router must support custom DHCP options. An OpenWrt router or similar device has this capability, but a stock D-Link or Linksys probably does not.

You will need a server on your home network to provide the file systems for the PXE Pis. There are multiple approaches, and I’ll be using CentOS 7. The hardware for this could be an old desktop, a virtual machine, or even another Raspberry Pi. We’ll assume a server with two interfaces — one connected to your primary network, and the second, with a static IP of 192.168.2.1, dedicated to the Raspberry Pi network. (If you plan to use a single network, skip the dnsmasq steps below, and use your server’s existing IP address instead of 192.168.2.1.)

Housekeeping and Preparation

The code block below takes care of the initial dependencies and firewall settings, and enables the needed services. On the server, we need three primary services: Dnsmasq for DHCP, nfs-utils for NFS, and tftp-server for TFTP. Xinetd is a helper service for tftp-server, and we need unzip and wget for downloading and extracting the Raspbian disk image.

Install your favorite text editor (I’m using nano), and the policycoreutils package in order to work with SELinux. Then, allow the services through the firewall and set them to run automatically on boot. Lastly in this block of instructions, we’re setting SELinux to a more relaxed stance concerning the TFTP service.

sudo yum install -y dnsmasq tftp-server nfs-utils xinetd unzip wget nano policycoreutils-python
sudo firewall-cmd --permanent --add-service nfs3
sudo firewall-cmd --permanent --add-service mountd
sudo firewall-cmd --permanent --add-service rpc-bind
sudo firewall-cmd --permanent --add-service tftp
sudo firewall-cmd --permanent --add-service dhcp
sudo systemctl enable xinetd
sudo systemctl enable dnsmasq
sudo systemctl enable rpcbind
sudo systemctl enable nfs-server
sudo setsebool -P tftp_home_dir 1

Set up the directory structure and a pair of temporary mount points, and then download and extract the latest Raspbian image.

sudo mkdir /tftpboot sudo mkdir -p /nfs/raspi1 mkdir ~/bootmnt mkdir ~/rootmnt wget http://director.downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2018-06-29/2018-06-27-raspbian-stretch-lite.zip unzip 2018-06-27-raspbian-stretch-lite.zip

Booting with PXE starts with DHCP option 66 (not order 66, that’s something different), which points to the TFTP server. We’ll configure dnsmasq to add that option, as well as the listen interface and the IP address range. Change these values as is appropriate in your case, and add to /etc/dnsmasq.conf.

interface=ens9
dhcp-range=192.168.2.50,192.168.2.150,12h
dhcp-option=66,192.168.2.1

The configuration for NFS is simple. We specify what directory we want exported by editing /etc/exports.

/nfs/raspi1 *(rw,sync,no_subtree_check,no_root_squash)

Xinetd stands for “eXtended InterNET Daemon.” This one service controls several other services, including TFTP. We’ll configure xinetd’s config file at /etc/xinetd.d/tftp looking for the lines “server_args” and “disable”. We’re adding the two “-v” flags to increase the verbosity, as we will need to see what files the Raspberry Pi is looking for as it boots.

server_args = -v -v -s /tftpboot
disable = no

Building the Pi’s Boot Image

Kpartx is an invaluable tool to keep in your Linux-fu toolbox. We’ll use it to mount the partitions contained in this disk image, and then copy the file system to the new PXE root. The Pi looks first for bootcode.bin, so we also copy that file into place.

sudo kpartx -a -v 2018-06-27-raspbian-stretch-lite.img
sudo mount /dev/mapper/loop0p2 ~/rootmnt/
sudo mount /dev/mapper/loop0p1 ~/bootmnt/
sudo cp -a ~/rootmnt/* /nfs/raspi1
sudo cp -a ~/bootmnt/* /nfs/raspi1/boot/
sudo cp -a /nfs/raspi1/boot/bootcode.bin /tftpboot/

We can also customize the filesystem for the Pi. First, to enable ssh access.

sudo touch /nfs/raspi1/boot/ssh

Next, we’ll modify the kernel boot line, at /nfs/raspi1/boot/cmdline.txt. Here we’re informing the kernel that it shouldn’t look for filesystem on a local disk, but to mount an NFS share as its root.

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=192.168.2.1:/nfs/raspi1,udp,v3 rw ip=dhcp rootwait elevator=deadline rootfstype=nfs

And finally, we remove a couple lines from the Pi’s fstab, so it doesn’t try to mount filesystems from the non-existent SD card. Edit /nfs/raspi1/etc/fstab and remove the last two lines, that mount / and /boot.

Network Sleuthing and Bind Mounts

At this point, we’re ready to see if our hard work has paid off. Reboot the server so the pending changes are applied. (Yes, it’s fairly easy to apply them by hand, without the reboot, but this step also tests that everything comes back up correctly on power loss.) Once it’s back, watch the system log while powering the Pi connected to the new network. (Al Williams just ran an article on the versatility of the tail command that’s worth a look.)

$ sudo tail -f /var/log/messages
Sep 24 11:05:15 PXE xinetd[1043]: START: tftp pid=4824 from=192.168.2.103
Sep 24 11:05:15 PXE in.tftpd[4825]: RRQ from 192.168.2.103 filename bootcode.bin
Sep 24 11:05:15 PXE in.tftpd[4825]: Client 192.168.2.103 finished bootcode.bin
Sep 24 11:05:15 PXE in.tftpd[4826]: RRQ from 192.168.2.103 filename bootsig.bin
Sep 24 11:05:15 PXE in.tftpd[4826]: Client 192.168.2.103 File not found bootsig.bin
Sep 24 11:05:15 PXE in.tftpd[4826]: sending NAK (1, File not found) to 192.168.2.103
Sep 24 11:05:15 PXE in.tftpd[4827]: RRQ from 192.168.2.103 filename 57b3548e/start.elf

The date will differ, as will the “PXE”– the hostname of this server. You’re looking for the RRQ from an IP address. If that shows up, then everything is going great. If not, time for some troubleshooting.

You may notice that your Pi is not actually booting. That’s because we’re not quite done. Look back at the TFTP log lines, because we need the folder name your Pi is looking for, which happens to be its serial number. Create that folder inside the tftpboot folder. Replace 57b354e with the folder name from your log.

sudo mkdir /tftpboot/57b3548e

We need to talk about bind mounts, and why we’re not just copying files into this folder. Raspbian has a built-in procedure to update the kernel, through apt-get. PXE transfers the kernel stored in the TFTP folder, while the rest of the system is mounted as an NFS share. This arrangement prevents the Pi from updating its own kernel. There are a couple ways to fix this, but today we’re using the bind mount. Think of it as a hard link for directories. We’ll add a line to the server’s /etc/fstab, once again using the Pi’s serial number as the folder name.

/nfs/raspi1/boot /tftpboot/57b3548e none defaults,bind 0 0

And then use the mount command to make the bind mount live.

sudo mount /tftpboot/57b3548e

Now, power cycle the Pi, and it should come to life!

The Power of PXE and Pi

The number one problem with Raspberry Pi systems is SD card problems. Booting over the network will get around this weakness (assuming your network and your server are both stable systems). For my purposes this is a perfect choice, since I plan to use a Raspberry Pi in a three-gang electrical box, connected via Ethernet and using Power Over Ethernet. That said, any application where you will have an Ethernet connection, and need your Raspberry Pi to be reliable over the long term, is a good candidate for PXE.

Where do we go from here? Next time we’ll set up Zoneminder, and finally put these Pis to use. Let us know what else you want to see, or what we missed this time. Until then, happy hacking!

59 thoughts on “Hack My House: Running Raspberry Pi Without an SD Card

  1. Thank you for the tutorial!

    > we’ll talk about [..] ZoneMinder next time.

    Please consider covering MotionEyeOS instead, it’s been nothing but a blessing so far – set it up on 5 RPIs in our hackerspace (both dedicated and multi-use), it’s reliable, powerful and easy-to-setup. In contrast, those three don’t apply that well to Zoneminder, in my experience.

      1. Well, that depends on the stream – I do use Pi2 or better on Raspberry camera streams (which can be streamed high-res), but a single B+ can easily run two USB webcams with resolution just good enough to have some kind of picture. I don’t do motion detection on those same Pis, though – there’s a separate server for that, with storage and stuff.

    1. Neither Motion or zone minder are really that great. Pikrellcam does h254 compression so HD stream at high framerate and quite advanced motion detection with speed and size parameters and external gpio triggers. It can run on a RPI B only using 15% processor. SD card degradation is not a problem if you make the distro read only and save videos on NAS.

      1. Pikrellcam is very smart, it reuses motion vectors generated by hardware h264 compression block – makes whole motion part of the algorithm basically free. This can be done with any mpeg4 stream on low speed cpu.

      2. I used to use zoneminder. With 10 analog cameras. I upgraded to 11 1080p cameras and very quickly realized that zone minder was not built for that much data at once. Blueiris was the best that I found

    2. If your focus is video for security primarily, then consider proper CCTV cameras and corresponding NVR.
      It aint expensive and you’ll save a lot of time compared to configuring software like zone minder etc and more importantly it’ll be reliable and generally maintenance free.

      If your focus is on a project to make your own video system then by all means fill your boots and roll your own but bear in mind the limitations compared to hardware designed specifically for the task.
      This goes without saying for anything, but expect people will pipe in to say how perfectly their RPi works for CCTV without getting the point.

      1. One advantage a raspberry pi CCTV camera has over a normal IP camera is that you can add in functionality that is either locked behind license keys or doesn’t exist at all in a commercial setup. You could interface the Pi that watches the front door to open the lock and turn off the security system when it sees your phone pop up on bluetooth. You could have a doggy door unlock automatically when the camera detects something an object with a similar color and height as your dog approach the door. Trigger the coffee maker when a hallway camera sees you walk out of the bedroom anytime between 6 and 8am weekday mornings. Coffee maker timers are great but I don’t always get ready at the same time everyday. Detect when no one has been home for awhile and trigger a vacation mode where lights and TVs turn on and off to mimic someone being home. The CCTV aspect is the primary use but I’m not looking for something with 100% uptime and an SLA to prove it. I would rather have something much more flexible and customizable than I can integrate with a home automation system.

  2. I’m trying to get my feet wet with OpnSense on a PC (VPNFilter was the final bit of motivation to go with a software solution decoupled from the hardware). I assume it can be configured to be a PXE server as it is built on BSD, but has anyone tried with the Pi and found the pitfalls yet?

    As a side note, I do miss the PC mentality of booting from a variety of targets dynamically. As a former BIOS engineer, I know what a pain in the ass it can be to initialize all possible boot vectors too. That said, I am thrilled with USB mass storage boot mode on the Pi3 and even happier that the Pi3+ dies not require OTP programming to support it.

      1. Must be something like Leet, I just read over it. It’s well known that technical people aren’t te best in language. My native language isn’t even English, so there could be a lot of typo’s in here.
        And indeed, booting from various devices should be easier. When I see almost a page full of sudo’s, it must be something that I realy realy want before I start trying it.

        1. That’s well known? There seems to be a certain scorn for correct spelling, punctuation and grammar these days, but I find technical people are amongst the most linguistically dexterous. Language is just another system.

    1. The pain of (multi)booting and the “one environment” is one of the reasons why people moved to virtualization. Still PXE is great for a (multi)thin-client environment. e.g. POS with versioning and configuration control on the server.

    1. At work, we recently switched to the industrial SanDisk SD cards you’re referring to for all our products that require SD. The feature set is excellent and they appear to be robust so far. We’re mainly using them for Boot/OS on Intel FPGAs (Cyclone 5 and Arria 10 SoC), but there are a couple of our products that have integrated Raspberry Pi boards and they’ll be used there as well. So far, so good.

      I think the 8GB is only $5 or $6 in the quantities we’ve bought so far, maybe? I’m not in purchasing, but I recall hearing that number at one point.

      1. Forgot to mention… things are typically mounted read-only. Depending upon need for the product, we occasionally store information in a separate partition that’s mounted read/write. The expectation is that it might get hosed at some point, so system-critical things aren’t kept there.

      2. What i found you can buy 8GB for 9€/pcs on mouser. That’s pretty nice price for industrial grade :)
        Of course it’s about twice price as “normall” SDcard, but when you calculate the time and nerves with crashed SDcard, the money goes back quickly :)
        Read only is also very helpful.

        1. Sandisk don’t give TBW values for their ‘standard’ cards, so comparing even the provided stats is impossible. Read speed 20% lower, write speed 45% lower – continuous throughput is still bottlenecked by the pi’s controller, but typically one would expect the seek times to be vastly higher as well, dramatically reducing performance.

          There may be a few rare use cases for these cards (if they’re not snake oil) like cubesats or data logger buoys, but even then, ‘normal’ flash running read only is still safer – and for every other use case, netbooting or taking the longevity hit is likely to be a better choice for performance reasons.

          Still running a 256MB openwrt original-SD card in a Pi somewhere round here, and a 4GB raspbian Jessie SDHC card in another pi (most recent uptime 157 days, but it’s been running untouched but for occasional relocation and apt-get dist-upgrades since at least 2016) so unless you’re really ragging the card, most people will be fine.

          1. Of course it’s not for everything :)
            But e.g. we have Lora gateways with RPi and they are in a cases on a masts on a roof and there is the higher temp range pretty useful :)

          2. That makes sense, if you’re talking about using something like a 0W and a solar panel to provide the service fully wireless. Like with the cubesats and buoys I suggested – these (if they’re not snake oil) would be useful for any device which is inaccessible *and* physically disconnected from infrastructure. :)

            Buuuut, if you’re running ethernet or mains cable to it – then netbooting would still be preferable.

    2. I’ve been using consumer SanDisk SD cards for a couple of years in media player/NAS Pis. Not a single corruption even though unplanned power outages happen and I don’t bother shutting down the system properly on planned ones. Maybe it’s because I’m just lucky, maybe it’s because I never buy SD cards nor pendrives online.

  3. CentOS 7 is certainly a great choice for old hardware, but it won’t support a system that was released after it was (2015). I tried to install it on a new Intel NUC, and it was a disaster, nothing worked.

    1. Latest Centos7 ISO was spun June 2018. Single board computers and cutting edge processors are always “interesting” to get running. The Intel NUC seems to be an odd duck for CentOS support.

  4. The root of the problem is not SD cards in most cases. Any linux system that is just powered off without an orderly shutdown has a good chance of seeing filesystem corruption, even if running on a hard drive. The real solution for an embedded linux system is a read only disk — so that in effect each reboot discards all changes (the only writeable filesystem should be a ram drive) and boots from scratch. Some of the linux router targeted distributions addressed this properly.

    1. I’ve run a bunch of Linux systems over the years, and Raspberry Pis are particularly bad at killing file systems– far worse than a desktop that I regularly hardlock by playing with graphics drivers. My experience might be an outlier, though.

    2. Using SD cards/flash memory introduces a whole extra layer of risk on power failure, well beyond OS-level “clean shutdown” issue. If it didn’t, a journaling file system would mitigate the problem.

      SD cards store data in really large blocks, 4MB or so in size. In order to write to one of these, the entire block has to be erased and re-written. If power is removed during this procedure, the entire block is corrupted, and will probably take out quite a few OS-level clusters beyond the ones the OS is trying to write. In the worst case, if the SD metadata block (which stores all the wear-leveling information) is corrupted the entire SD card may become permanently unusable.

      This research paper gives more details. https://cseweb.ucsd.edu/~swanson/papers/DAC2011PowerCut.pdf

      NOTE: SSD drives, I believe, contain either supercaps or batteries which mitigate this issue.

        1. That would be interesting idea to research, I really don’t know enough about the low-level protocol(s) used to talk to SD cards (and what happens under the hood) to say. My guess would be “not entirely”, because the wear leveling and error correction might happen well after a write, unless the card has been instructed specifically to shut down.

          Correction to my post above: Many Industrial SSDs contain capacitors and special circuitry to implement whats called Power Loss Protection (PLP). Apparently many/most consumer SSDs do not. :-(

          Intel, Micron, and other companies have white papers on the subject, e.g. https://www.micron.com/~/media/documents/products/white-paper/ssd_power_loss_protection_white_paper_lo.pdf

    3. Perhaps it would be helpful to go through Pi distros with the “How to setup Linux for SSD” tips and tricks pages and make sure all the boxes are checked for disabling write behinds and file tables in RAM and other volatile things and sources of write thrashing.

    4. Well, while I’m using a specially crafted distro (Formilux) which runs off a read-only FS on my appliances, I don’t really agree with you, because since I’ve started to use journaled file systems ~18 years ago (reiserfs initially, then xfs, then ext3 and ext4), I don’t recall having typed “shutdown” or “halt” to stop a linux machine, neither desktop nor server. I simply type “sync” and I power it off. All those of us with laptops who experience power outages after draining batteries are used not to worry whether the PC will still boot once plugged.

  5. I’ve found the network boot feature to be quite fragile in certain cases.

    An R&D project I was working on used 16 compute modules (CM3) with an Intel Cyclone 5 network booting the CM3s via a LAN9512 USB attached to each compute module. If I released the resets to all CM3s simultaneously, I would only get a few booted successfully. In order to successfully boot all the CM3s, I had to start the first, let it finish booting, then move onto the next. It would take approximately 30 seconds per CM3. The issue seemed to be related to the flood of BOOTP requests on the network. The CM3s seemed to hear one another talking and would wedge. Very frustrating.

    I’ve had a bug ticket open with the RPi folks since March or April of this year.

    I suspect this *could* be a problem on a network with many Pi boards if, say, the power went out and all the Pi boards tried to boot at the same time once power was restored.

    A possible workaround, which I have yet to try on the development platform, is to put each Pi on a separate VLAN (we have several smart switches onboard) so the each CM3 essentially sits on an isolated network.

    1. I’ve also found the older Pis to be fragile, but the newer 3 b+ devices have been much more reliable. At some point, I’ll try rebooting all of them at once and see how many come back up.

  6. Of course if you are going to have a real computer powered up to boot the rpi from you might as well just run your house from the real computer. Why one would wanna pxe boot a rpi is beyond me. I can see doing it perhaps for deploying a local OS but that is about it.

    1. It’s just a pain to try to run a boatload of gpio and i2c ports off a single central machine, not to mention getting a usb port on the opposite side of the house to hang a webcam from, for instance.

      As we dig further into their uses in coming articles, hopefully the impetus will make more sense.

  7. In addition to what I wrote above, wanted to point out that my personal go-to method for making robust embeddable raspi’s is to setup the OS to boot read-only by default. This procedure is well documented and works well. If the SD card isn’t being written to, it won’t get corrupted on power fail. I’ve stress tested this pretty heavily and it seems pretty solid.

    https://learn.adafruit.com/read-only-raspberry-pi/overview

    You’re still able to *temporarily* write to the filesystem by using the built-in ramdisk mount points like /tmp and /dev/shm , but of course those are lost on powerfail or reboot.

    Later updates can still be made by temporarily remounting the filesystem as read-write.

    1. I hadn’t seen that article before, thanks! A read-only Pi has also been on my todo list, but I’ve never gotten around to actually setting it up. Their script looks like a slick shortcut to getting one set up.

      1. Just used it the less than a week ago on raspbian-stretch-lite (2018-06-27), and it worked like a champ. Previous to that I’ve done it all manually, but its a bit laborious and the script is a bit time saver and seems to be well-maintained.

        Only thing that is missing is it should really add some remount aliases to .bashrc. I use the following:

        alias ro=’sudo mount -o remount,ro / ; sudo mount -o remount,ro /boot’
        alias rw=’sudo mount -o remount,rw / ; sudo mount -o remount,rw /boot’

        That way I can “rw”, make some changes and then “ro” and put it back in “safe” mode (read-only)

        1. Thank you so much for this! I’m not super confortable with how mounting works on Linux and was looking for those commands before using the Adafruit script. This is great.

        2. Thank you so much for this! I’m not super confortable with how mounting works on Linux and was looking for those commands before using the Adafruit script. This is great.

  8. Can anybody remind me the exact underlying technical reason why PXE booting is not supported in earlier RPi models? SoC incompatibility? Deprecated/unmantained bootloader? No OTP section to set the usb bootmode flag(s)…?

    1. Strictly speaking, it’s because there isn’t support for loading bootcode.bin over tftp. You can boot off an sdcard that chains to PXE for the rest of the boot process.

  9. Got to ask – are you going to take advantage of the fact that you’ll have a dual-band linux wi-fi device in each room, and set up the Pi’s as hostapd access points? Benchmarks suggest that you should be able to get about 100mpbs throughput (less than half the ethernet bandwidth, so plenty of room for the OS uplink, too) – maybe not enough for the thinkpad or macbook in your lab, but plenty for the bathroom, kitchen, utility room, bedroom – stuff like epicurious, RSS readers and panic-googling stain removal don’t take much bandwidth after all.

  10. Thanks for the post. Network boot was commonplace back when hard drives cost a fortune. My first drive was $650 dollars for 20 megabyte drive, about $1400 in today’s dollars. That’s $1400 for 20 megabytes, not 20 gigabytes.

    Anyway, i kept about 10 sun workstations running with netboot starting in 1989, almost exactly the same way you did here: tftp, nfs, bind, multiple CPU architectures, etc. The PXE equiv was a feature of the Ethernet card back then. I’ve saved the article to netboot deploy RPIs around here.

    ps. You didn’t mention that netbooting permits file system backups and lowers MTTR, meantime to repair. You fry your pi? pull it out, put another one in and turn it on, done.

    1. I worked at a couple of places in the mid-late 90s that had hundreds of old sun stations netbooted. Some did have hard drives, but they were ignored for MTTR reasons as you noted above. A dead station could be replaced in minutes by updating DHCP with the new MAC and powering it up (though they were tremendously reliable in the first place). Software updates only required changing the shared image. Not having persistent local storage made sure users put their documents in their home directory where they belonged, so no heartburn if something did happen to a station.
      It was a real blessing to have so many easy-to-maintain systems. Especially since the NT workstations at the facility more than made up for the time savings…

    1. So does the 3B – an SD card is needed for a one-time-setup on the 3B, but afterwards the SD can be removed and it will always boot from USB.

      The 3B+ just avoids the need to do this. Though, the CPU cooler alone makes the 3B+ a great replacement, before the clock increase, quarter-gigabit ethernet and dramatically improved wireless. So, that’s nice!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.