Understanding a national budget can be difficult, with political interests and distorted talking points obscuring the truth. There’s no substitute for diving into the hard data yourself, but it can be difficult to know where to start. [D. Scott Williamson] wanted to do just that, and took an unconventional path – using file system tools to analyse the 2019 US budget.
The basic idea is simple. A file is created for every line item on the budget, and this file is filled with a $ for every million dollars that item costs. These files are sorted into folders for the relevant bureaus and agencies responsible, creating a Fiscal File System representation of the entire budget. The file system can then be analysed with standard tools to get a look at the overall make up of the budget – in this case, WinDirStat does a particularly good job.
It’s a novel way to go about budget analysis, and fills a gap in the market now that Time Plots aren’t producing their Death and Taxes infographics anymore. Pre-cooked file systems covering budgets back to 1977 are available for your perusal, should you need to investigate how things have changed over time.
Unsurprisingly, we haven’t covered the national budget before, but we’re always down to talk file systems.
Filesystems for computers are not the best bet for embedded systems. Even those who know this fragment of truth still fall into the trap and pay for it later on while surrounded by the rubble that once was a functioning project. Here’s how it happens.
The project starts small, with modest storage needs. It’s just a temperature logger and you want to store that data, so you stick on a little EEPROM. That works pretty well! But you need to store a little more data so the EEPROM gets paired with a small blob of NOR flash which is much larger but still pretty easy to work with. Device settings go to EEPROM, data logs go to NOR. That works for a time but then you remember that people on the Internet are all about the Internet of Things so it’s time to add WiFi. You start serving a few static pages with that surprisingly capable processor and bump into storage problems again so the NOR flash gets replaced with an SD card and now the logs go there too. Suddenly you’re dealing with multiple files and want access on a computer so a real filesystem is in order. FAT is easy, so the card grows a FAT filesystem. Everything is great, but you start to notice patches missing from the logs. Then the SD card gets totally corrupted. What’s going on? Let’s take a look at the problem, and how to reach embedded file nirvana.
Continue reading “Cool Tools: A Little Filesystem That Keeps Your Bits On Lock”
[Matthew’s] recent blog post does a good job explaining the basics of the Raspberry Pi’s file system. The Linux operating system installed on a Pi is generally installed on two different partitions on an SD card. The first partition is a small FAT partition. All of the files on this partition are used for the initial booting of the Pi. This partition also includes the kernel images. The second partition is the root file system and is generally formatted as ext4. This partition contains the rest of the operating system, user files, installed programs, etc.
With that in mind you can deduce that in order to backup your Pi, all you really need to do is backup all of these files. [Matt] has written some scripts to make this a piece of cake (or pie). The first script will simply copy all of the files into a gzipped archive. You can save this to an external SD card, USB drive, or network share.
The second script is perhaps more interesting. This script requires that you have one free USB port and a USB SD card reader. The script will automatically format the extra SD card to contain the two critical partitions. It will then copy the “boot” files to the new boot partition and the root file system files to the new SD card’s root partition. When all is said and done, you will end up with an SD card that is an exact copy of your current running file system.
This can be very handy if you have multiple Pi’s that you want to run the same software, such as in a Pi cluster. Another good example is if you have spent a lot of time tweaking your Pi installation and you want to make a copy for a friend. Of course there are many ways to skin this cat, but it’s always fun to see something custom-built by a creative hacker.
Have you ever thought a particular project could be better if you could just control the file access directly? [Stavros Korokithakis] did, specifically for a backup program he was working on. What followed was the realization that writing a FUSE filesystem, particularly in Python, isn’t as complicated as it may seem. Really, through the power of open source, the heavy lifting has already been done for us. If you’d like to try it yourself, you’ll need to install fusepy. From that point, you simply need to define the filesystem methods you will be using.
Python isn’t going to win any speed contests in the filesystem space, but that isn’t really the point. Using this technology opens up a huge opportunity for new ways of accessing data. If you let your mind wander, you can conceive of encrypted filesystems, seamless remote data access, new key-value storage designs, etc. Perhaps even more interesting is the idea of using Python to communicate with a physical device… maybe a proc filesystem to keep track of your robot telemetry? We’d love to hear your ideas in the comments.
We had success using [Stavros’] example script on Linux and OSX. (Fair warning if you’re on a Mac, the pip version of fusepy seems to be linked against fuse4x rather than OSXFUSE, but once you’ve got the prerequisites installed, you’re golden.) We didn’t have a Windows machine to test. Can anyone confirm if the same is possible there?