There was a time when version control was an exotic idea. Today, things like Git and a handful of other tools allow developers to easily rewind the clock or work on different versions of the same thing with very little effort. I’m here to encourage you not only to use version control but also to go even a step further, at least for important projects.
My First Job
I remember my first real job back in the early 1980s. We made a particular type of sensor that had a 6805 CPU onboard and, of course, had firmware. We did all the development on physically big CP/M machines with the improbable name of Quasar QDP-100s. No, not that Quasar. We’d generate a hex file, burn an EPROM, test, and eventually, the code would make it out in the field.
Of course, you always have to make changes. You might send a technician out with a tube full of EPROMs or, in an emergency, we’d buy the EPROMs space on a Greyhound bus. Nothing like today.
I was just getting started, and the guy who wrote the code for those sensors wasn’t much older than me. One day, we got a report that something was misbehaving out in the field. I asked him how we knew what version of the code was on the sensor. The blank look I got back worried me.
Seat of the Pants
Turns out, he’d burn however many EPROMs were required and then plow forward developing code. We had no idea what code was really running in the field. After we fixed the issue, I asked for and received a new rule. Every time we shipped an EEPROM, it got a version number sticker, and the entire development directory went on an 8″ floppy. The floppy got a write-protect tab and went up on the shelf.
I was young. I realize now that I needed to back those up, too, but it was still better than what we had been doing.
Enter Meta Version Control
Today, it would have been easy to label a commit and, later, check it back out. But there is still a latent problem. Your source code is only part of the equation when you are writing code. There’s also your development environment, including the libraries, the compiler, and anything else that can add to or modify your code. How do you version control that? Then there’s the operating system, which could interact with your code or development tools too.
Maybe it is a call back to my 8″ floppy days, but I have taken to doing serious development in a virtual machine. It doesn’t matter if you use QEMU or VirtualBox or VMWare. Just use it. The reason is simple. When you do a release, you can backup the entire development environment.
When you need to change something five years from now, you might find the debugger no longer runs on your version of the OS. The compiler fixed some bugs that you rely on or added some that you now trip over. But if you are in your comfy five-year-old virtual environment, you won’t care. I’ve had a number of cases where I wish I had done that because my old DOS software won’t run anymore. Switched to Linux? Or NewOS 2100tm? No problem, as long as it can host a virtual machine.
Can’t decide on which one to use? [How to Simple] has some thoughts in the video below.
How About You?
How about it? Do you or will you virtualize and save? Do you use containers for this sort of thing? Or do you simply have faith that your version-controlled source code is sufficient? Let us know in the comments.
If you think Git is just for software, think again.
We use docker / podman containers that contain all the environment including external code and libraries which obviously could also disappear in the future.
The docker file can be versioned as well, the resulting containers stored and backuped.
And every code version is linked to one single container version which has been used to build it.
Virtual machines don’t really fix the problem.
When you upgrade your host operating system, the old guest extensions often stop working. If your toolchain (FPGA vendors are especially bad with this!) requires a license, even a free one, that will have expired and will try to call out to servers that have changed. The operating system will do its best to force you to upgrade things. And virtual machines just make development so much more miserable to do.
Instead try to make the build system reproducible, without resorting to just archiving everything. CI builds go a long way towards this, but only for a few years until the runner image is deprecated and old dependency versions disappear from repositories. You need a shareable, documented build environment anyway if you expect to ever have more than one developer in the project.
As a last resort, have your whole system backup solution keep a few snapshots that you can boot and run if you really have to.
I agree with you on the documentation front, but would argue that containerized or virtualized build environments are still superior to manually setup ones in terms of being reproducible. Manual recreation from documentation should be more of a last resort. I like the CI approach best too — a docker container is a lot closer to a well documented environment than a simple archived VM is.
If you are really desperate, or REALLY want to make sure you can build later, a day of engineering time buys at least a couple of laptops. Air gapping and storing (or shipping!) a couple copies of a workstation is always an option, and I have seen it done. It’s gross but it works.
While it’s not always possible, especially with things like FPGA tools, the license issue is the single biggest reason I stringently advocate for the use of open tool chains. Commercial tool support just won’t be there long term. Past versions of GCC or clang will, and going back to a previous OS release is generally doable as well in the worst case.
The bottom line to all of this is really complexity. That’s something that doesn’t get nearly enough attention these days. Did you get rid of all the dependencies you don’t REALLY need? Did you minimize the number of oddball third party tools you use to automate build steps in favor of long lived and popular tools that will still be around in a decade? Documentation is a piece of the puzzle, but in a lot of cases we don’t document systems because we don’t even understand all the variables anymore.
“If your toolchain (FPGA vendors are especially bad with this!) requires a license, even a free one, that will have expired and will try to call out to servers that have changed.”
At least for all the FPGA vendors I’ve worked with, the software doesn’t try to verify a license over the network unless it’s a floating license (obviously). And at least for Xilinx those don’t actually ‘expire’ (they’re not like maintenance stuff, that obviously sucks), they just expire for new versions.
Which is a good thing, because the real downside to the vendors is that upgrading the software isn’t always ‘safe’ – it’s not remotely backwards compatible and there’s no way to make it so. As in, a core that you were using in a prior version might just up and disappear. So if you’ve got a design it’s usually smart to just lock it to a specific version.
Of course the second downside to that it is that when they started doing the Linux versions using Yocto derivatives… now everything blew up, because the Yocto stuff pulls files from the network, and there’s no guarantee those will exist. So you’re mostly required to match toolchain to yocto/oe stuff, which means… pray?
Sigh.
Nix (yes, I know this is the new “I use Arch, BTW”) is a great way to have dev env version control. It’s fairly easy to get into without resorting to the Nix wizardry that many in the community love. The only place where it isn’t great is for peripheral access, EG if you need to interface with an external programmer, debugger, serial adapter, etc, but even then it’s not any worse than a container or vm.
I use Guix, BTW :)
I have been intrigued by the Nix concept, but this is the first time I have seen it mentioned in the context of build environment reproduction. Your comment might have finally pushed me over the edge of giving it a try. Thanks for that!
Setting up my first new computer in 5 years last week, I VERY much wanted to put my entire dev environment in a VM. I had nothing but issues. VirtualBox should’ve been fine, but wasn’t. And something in the host OS (Windows 11) is now messed up and my only solution is a complete reinstall. Which I don’t really have time for right now, but eventually will have to be done.
To be clear, the dev environment is also Windows. (Because our servers are Windows, yes, yes, I know.
Not my choice.)
new computer setup is always pain. part of my motivation for moving to VMs was that I was really quite tired of setting up new machines, and finding relevant drivers (which may not even exist for modern OSs), and all the other concomitant platform-specific stuff.
So in a way you can view the pain of the setup experience for your whatever VM solution as an effort concentrator: get your system working for your hypervisor, and then after that profit! by being able to use your archive of maybe 100 or so dev VMs that will work in that hardware abstracted platform.
What am I missing? What if the relevant drivers continue not even existing? is your client forced to forever run the product in a VM? With the added pain of setting up and learning how to use the VM. Yes, I know “pain” is a strong word. I really feel I’m missing something here but I now nothing of VMs except that they exist.
“know nothing” THAT’s pain.
” is your client forced to forever run the product in a VM?”
Oh. I’m there already. Xilinx’s dev environment for chips that they’re literally still selling requires you to install a VM because they don’t support any modern OS. There are hacks to try to work around that but they periodically fail as the new OSes get updated and trying to keep ahead of it is exhausting.
the VM is for your dev environment. let me give a concrete example and temporarily disregard some anachronisms, because this real-world example is from the past.
* part of my product integrates with 3rd party Point-Of-Sale systems. They typically run on Windows systems, so to develop that component I need a deployed POS. I support 8 or so POS’s from different vendors. Some are C#, some are VB, some are Delphi (no I’m not kidding). One is Java.
* part of my product runs on mobile phones — an iPhone and an Android variety. iPhone’s dev env (xcode) requires a real-deal MacOS. Android (at the time) requires Linux.
* part of my product runs on on RackSpace virtualized servers. These are CentOS and occasionally Ubuntu.
* I am in possession of one Macbook Pro.
So, by having VMs for all those disparate machine configurations, I am able to develop the entire system on one computer.
Think of a VM as a computer-in-a-file. You can store your configured computer in a (very large) file and put it on a hard drive and give a copy to your coworkers. It has all the relevant dev tools installed. It also has only the relevant dev tools installed, so you don’t have conflicts in that Micro requires C# 3.5x and Aloha requires C# 4.x. Etc.
If I hire a new dev, I can hand them a copy of the dev env and they are productive on the first day.
In other cases this can also be vital if the vendor of the dev tools moves on and doesn’t, say, support the version that only runs on Windows 7 but in fact you really need because you’re supporting some old chips in the field.
hth!
In 2025 I am coming to view Windows like I viewed Linux in 2001 — it’s works, but it’s usually the less stable and more painful path to a system you want to do real work on. Never thought that day would come.
It’s painful to do the first time, but I have gotten Win 11 up and running in a Linux host environment. The trick is making sure to use a UEFI bios image and make sure to include a virtual TPM.
I occasionally get stuck working with Windows too… but only when the job market is bad.
Oh! the days before version control. I came up in hardware, so I didn’t even know about version control other than manually PKZIP’ing stuff with a timestamped name. I distinctly remember when my spell correcting fingers caused me to unzip rather than zip and I looked on in paralyzed horror as my day’s work was reverted before my eyes in a scrolling death list of “here’s some work you can kiss goodbye!”
And VC systems were expensive! (e.g. PVCS) But my life was changed in 1994 when I broke down and bought a copy of SourceSafe from (then) One Tree Software for the low price of $600 or so. Being able to move through time and also diff’ing was such a revelation. Obviously I have moved on to git and for personal projects I really like Fossil. I don’t think folks use ‘distributed’ version control for the ‘distributed’ part, but rather having to support ‘distributed’ development forced creating diff algorithms that were much smarter. I mean, you could branch and merge just fine with CVS and Subversion, but woe be unto thee if you have a merge conflict.
Regarding dev envs, these days I virtualize VMWare, and have been doing so since the early 2000s. When I was in the bay area in 2000 a friend was telling me about this new company that made a thing that allows you to run two OS’s on your machine at the same time, and maybe I should join them. “I can dual boot already”. “No, they’re running atthesametime*!” “OK, but who would want to do that?” lol. It was super useful when I was doing malware detection because it was like a condom for the computer, and ‘snapshots’ were vaguely like version control for the system. Later I would use the VMs as ‘dev kit in a box’ and just hand them out to new hires. No more “I’m installing my dev env” and following woefully outdated “how to set up your box” wiki docs. Also the work we did involved 3rd party integrations, so the various base images could have just the relevant tools for whatever system was being worked on. No conflicts with other system’s requirements. And you didn’t have to worry about whether the host system was Windows, Mac, or Linux.
These days I would like to use containers. You can do that for a lot of stuff, but not everything. Hope you like Linux because there you are, and UI… well. A lot has progressed to browser based so maybe. But I personally have too much invested in archives of dev system images now that it’s unlikely I will change from VMWare anytime soon. I did explore free things like VirtualBox but I find the USB support lacking and it to be a bit slow. At the same time VMWare is not at all free, and frankly seems to be on a downward trajectory in terms of reliability and speed. So I’m open to revisiting the strategy.
But time travel is definitely the wave of the future. Or the past. Or in branching off into a parallel universe.
BT,DT. In Windows it’s ‘pkunzip’ instead of ‘pkzip’, in Unix it’s ‘tar xv’ instead of ‘tar cv’, which is a REALLY easy mistake.
and all my hand-crafted asm for a DSP routines that ran ~ 80x faster than the C versions because I avoided segment register loads on a 386 scrolled past my eyes to their doom.
:sad-trombone:
I did a similar thing a long time ago. Created my build machine VMs, then exported the VM system, burnt it onto a CD and the Document Controlled the disk. That way, if the goverment agencies that regulated us asked “how or where was this made?”, I could point directly to a specific part in our control.
Now, Github action runners achieve that same thing. The github action workflow defines out exactly what will be installed to build, release, and deploy the code. When a commit or PR is created, a container VM spins up, installs the specific libraries needed, then the whole app builds, wraps up and spits out the artifacts and provides the whole documentation for how we got from point A to point B.
It would be nice if more projects did the same. Irks me sometimes to find a project that I want to try out, but they only have the code designed for some weird very specific version of a compiler with a weird very specific version of a library and their instructions for setting up to build the code is “install these dependencies” but don’t say exactly which versions and when you do so half don’t play nice with different versions of other dependencies. So you finally figure out that you need libraryA 5.13 because it is the only version that works with libraryB 1.222.222q which only works with libraryC 1.2 or 1.4, but not 1.3 and then you find out that libraryD only works with libraryB 1.222.222p…. ugh.
Nah, it should have a workflow script that builds a container that slots all the exact tools needed to build in.
Prior to using git I created periodic zip files with version numbers. If two people were working on a project we would manually merge with win merge. This took too much storage and time so we moved to git. Different work folders per developer were replaced by branches and zip files by commits (optionally with version tags). Submodules allow us to reuse parts of the code in multiple projects in a way that could be versioned too. .gitignore prevented storing build-artifacts and local tool chain artifacts from being committed. Commit timestamps have helped me many times to figure out how much time I roughly spend on each project.
When building I also save the commit hash in firmware and give a warning or error if there are uncommitted changes (which also happens at an incorrect commit for a submodule). Even if you forget to bump the version number the commit hash doesn’t lie.
Binaries of released versions are archived.
I also version hardware. Often by using spare MCU pins. The firmware is aware of the board revision. This allows backwards compatible firmware to be written.
Tool chain is archived too. Every I download a new version of a part of the tool chain I archive it first and also document the new version in readme.
It sucks that applications keep getting larger, because companies stopped caring about download sizes. If you archive different versions of tools and binaries it can easily take up a lot of drive space.
I routinely use git and target containers these days.
It’s imperfect, but it is highly repeatable.
My actual pipeline is very limited though, no massive build chain, just simple build scripts.
Using podman for containers, their yml files (kube play) to create pods (collections of containers) and occasional, but forced rebuilds from scratch to prove the pipeline is functioning as-expected.
So can save a built container and transport it to production and know it’ll run.
i haven’t found any solution better than simply writing the code to a reasonably high standard (many different aspects to that metric) and then it’s only moderately hard to deal with bitrot.
recent thorn in my side is modern gcc rejects (error diagnostic) a lot of things that used to be just warnings, like unprototyped functions and pointers to mismatched types. i know there must be a convenient commandline option to get gcc to be accepting of both old and new idioms at the same time. but instead, for the old versions i care about, i have simply fixed the problems that really should have been done right in the first place.
i can only imagine what kind of nightmare it is to work with code developed targetting a 6 hour old version of rust. or basically anything that builds on top of a package manager. android, for example…every single time i sit down to it, everything has been obsoleted. i do kind of wish i had a VM with the 2012 android dev kit still on it
I am there with versions of gcc. Over time the (new) warnings and errors have caught a few bugs (none critical) each time a new gcc version released. Fixed. This last one found a few old style function declarations (from the 80s) that we had missed to convert which the newest compiler no longer accepts. Simple to correct. I like my code to compile ‘clean’ with no warnings. I did have to suppress one warning as it would be real pain to address … and there was no problem. The problem was code reusing big buffers for small strings and it thought the code needed as big a buffer for the result string it was being strcat’ed too. Obviously not a bug. Suppressed the warning (after evaluation of them all that were kicked out).
hahah yeah i’ve got parts of my life where there are pages upon pages of sprintf warnings. the other new warnings are pretty nice!
Dear god I hate the moving target that is Android. Even if I could compile my old code, it won’t run on a modern Android phone.
For work, we use git. BT/IT automatically backs up systems periodically in case have to go back. At home for my projects, I use git for the important ones (to me) and usually copy off to another system with the date before changing code. Like projectX_20250505_1152 . I always do this anyway before I make major changes … just in case. Never failed me yet… And disk space is cheap.
I remember the EPROM days. Everyone that went out had a label on top with name/revision and position (if multiple sockets). Of course the label also covered the erase window. And of course FedEx Next Day to job site. Finally Flash came along … and email to send zips of needed programs… Until zips got blocked by IT, and then setup ftp site to grab updated software from home base and so on. Fun!
Also learned to always put a header on each code file. Every change to file is documented in the header with who and when. 1.001 RJC 05/05/2025 Blah blah blah …
OMG. “in an emergency, we’d buy the EPROMs space on a Greyhound bus.”
Flashback to where I was at the destination, digging through a pile of boxes at Greyhound to find the one with my name on it.
But THAT was the good old days. Flash forward to hand-carrying spare boards, a 1200′ magtape and a toolbox on a plane Seattle to Vancouver BC to Denver to Calgary because that was the only route the travel agent could find that would get me there within 24 hours, customs and immigration both ways looking at me trying to figure out what kind of scam I was running and putting me in a small room..
Your travel agent was a clown.
You book the flight to your destination with a connecting flight to some expensive far destination.
The airline computer bumps some schmo for your super profitable last minute trip.
You then cancel to leg to Dubai/Tokyo/Sydney and try not to smile when the schmo is yelling at the gate agent.
I knew one dude that repeatedly overbooked flights on purpose just to collect enough travel vouchers to take his family to Hawaii.
Most guys with that many miles (to be able to last minute overbook and then collect the price as a voucher) don’t have time.
But he had a half day wasted anyhow w a long connection later, so he got on, then ‘begrudgingly’ volunteered to get off of 3 or 4 flights in a row, only danger was one wouldn’t be full.
For version control of development environments, I’d suggest looking into Nix and/or Guix. They allow for development environments to be defined in configuration files and are designed to lead to reproducible environments, no need to backup a whole VM, you can just check the required config files for Nix / Guix into version control along with your code.
This is similar to something like Vagrant, isn’t it?
Downside to stuff like that is that you’re relying on the Internet to keep things available for you. This works for a few years (hopefully) but once you start getting to like, 4-5+ years it’s increasingly likely something breaks. Once you get to a decade it starts becoming a lot harder.
And if a decade sounds like overkill – the issue is that if you don’t control the build environment or the development time is long, you essentially can get “latencies” that build up and cut that ‘decade’ down to a lot shorter. For instance: you start working with a tool in 2022 that used other builds from 2020, it takes you 2-3 years in development, now you’re trying to download releases and tools from 5 years ago.
“How about it? Do you or will you virtualize and save? Do you use containers for this sort of thing? Or do you simply have faith that your version-controlled source code is sufficient?”
God, I wish it was possible with FPGA development. At least with Xilinx, version control is basically “pray.” There’s no real way to fully recreate a build because of the randomness involved in the place & route (you can kinda-sorta try, but it’s not perfect). You just have to save all the actual build checkpoints, which can be around 0.5 GB per build. Oh, and that’s already compressed and binary, so yeah. You’re just screwed.
Saving the build environment and tools also eats up space faster than you can imagine. Those at least compress somewhat, but you’re still talking multiple gigabytes. Yes, storage space is cheap: it’s not that cheap. Even multiple terabytes get chewed up fast.
A while ago I just took the path of saying look, it’s just not possible to have a perfectly version-controlled system that multiple people can use, so you make it so that other people can at least logically recreate the build, even if it won’t be exactly the same.
“I’m here to encourage you not only to use version control but also to go even a step further, at least for important projects.”
From writers…
https://opensource.com/article/19/4/write-git
To Graphic artists…
https://stackoverflow.com/questions/29292/version-control-for-graphics
“There’s also your development environment, including the libraries, the compiler, and anything else that can add to or modify your code. How do you version control that?”
Just don’t do that part. Those are build-time concerns
Put everything needed to get to a running state in source control. No system configs, no by hand actions.
It should require only one command/button press/keystroke to go from a new environment with nothing set up to having a running debug session
Great idea!
I recently had to service a system I developed in 2012. It was based on legacy hardware at the time, so the development environment required Visual Studio 4 and associated SDKs for Windows Mobile. I think I got all this to run on Windows 7 back then, but I don’t even use a Windows machine anymore.
I built a VM using Windows 2000 and eventually got it all working again. I hope I saved it :-)
For hobby projects I keep a gitea instance on the home nas. Useful stuff goes to GitHub. Most of the old android 4.x stuff is unbuildable these days, which is a bit sad, but the hardware is also mostly gone. Stuff like libreelec/Lakka/coreboot ports usually also bitrot. Sometimes a git rebase main and few flash / debug cycles will get it current again, but most hardware ends up with the last build that ‘worked enough’ forevermore.