Setting up a cluster of computers used to be a high-end trick used in big data centers and labs. After all, buying a bunch of, say, VAX computers runs into money pretty quickly (not even counting the operating expense). Today, though, most of us have a slew of Raspberry Pi computers.
Because the Pi runs Linux (or, at least, can run Linux), there are a wealth of tools out there for doing just about anything. The trick is figuring out how to install it. Clustering several Linux boxes isn’t necessarily difficult, but it does take a lot of work unless you use a special tool. One of those tools is Docker, particularly Docker Swarm Mode. [Alex Ellis] has a good video (see below) showing the details of a 28 CPU cluster.
It is easy to set up a swarm using the instructions on the Docker website. If you aren’t familiar with Docker, it is almost (but not quite) a light-weight virtual machine manager. A true virtual machine manager pretends to be a piece of hardware so that Linux (or another operating system) can boot on it and run applications. Docker is a container manager, which means it doesn’t pretend to be a piece of hardware, it pretends to be a running operating system. Programs see their own file system and other resources, but in reality, there is only one kernel running on the host hardware.
The idea is similar to running something in a chroot jail. The program can make changes to its file system without upsetting the rest of the system. Docker also provides other kinds of isolation. The real draw, though, is that it can automatically load images of predefined environments. This allows developers to provide packages that are essentially preinstalled in their own private operating system.
That’s important because it means that a service can run on any node in a cluster. That lets you do tricks like balancing load across multiple nodes. You can also do rolling updates and dynamically add or remove computers from the cluster.
We’ve seen clusters before, of course. Even the little Pi Zero can get in on the act. If you want to understand more about Docker, you can always read the official documentation. Given the prevalence of Linux in embedded systems, it might be an interesting way to deploy preconfigured applications.