Clustering A Lot Of Raspberry Pi Zeros

It became something of a cliché a few years ago in online discussions, whenever a new single board computer was mentioned someone would pop up and say something like “Imagine a Beowulf cluster…“. Back then it was said largely in jest, but with the current generation of boards it’s a distinct possibility. Who hasn’t looked at a Raspberry Pi and idly thought about a cluster of them, or even created one!

[Electronoob] did just that, creating a variety of Raspberry Pi cluster configurations, the most impressive of which is a stack of 32 Pi Zeros mounted together with stand-offs. The plan was to network it via USB, for which he initially considered building a backplane, but was put off by the cost of vertical USB connectors and instead went for a wired approach. If there is a lesson to be learned from his experiences it is that buying very cheap USB cables is a minefield: his pile of eBay specials turned out to have significant numbers of faults. He’s now faced with a stark choice, solder  32 sets of USB pads on the base of each Zero or buy better cables.

The stack of Zeros is pretty impressive, but so what, you think. It’s still not working properly. But the Zero cluster isn’t his only work. He’s also created a set of very nicely executed Ethernet clusters using the larger Pi boards, and the way he’s mounted them on top of compact Ethernet switches sets them apart from some of the more spaghetti-like Pi clusters.

It’s true a Pi cluster won’t cut it in the world of supercomputers, you could almost certainly buy more bang for your buck without too much effort. But it does represent a very accessible way to learn about cluster computing, and you have to admit it a stack of Zeros does look rather impressive.

We’ve seen quite a few Pi clusters here since 2012, the biggest of which is probably this 120 node behemoth, complete with screens.

51 thoughts on “Clustering A Lot Of Raspberry Pi Zeros

    1. Probably like 3 or 4 Raspberry Pi 3s if optimized correctly. Real world probably like 2 Raspberry Pi 3s. But, as always needs to be mentioned in these threads, building a cluster out of Raspberry Pis is not about achieving high performance. They do it because they can and because it’s awesome, not because it’s practical.

      1. It is practical.But not for performance reasons. Such cluster is a great parallel programming learning tool. You get 32 completely separate nodes, with network communication between them This way you can practice programming with MPI, PVM, or any other distributed computing tool, using isolated nodes communicating only over the network.
        This is probably the easiest, cheapest and quickest way to gain experience which will be applicable in real, high-performance cluster computing environments.

      2. All I can think is – realtime non-linux-core or bare metal programming on each of those. 32 completely separate hard real time units. Can’t beat realtime performance of that with no Raspi 3…

  1. I was working on a stacked cluster until recently. Got a couple of Orange Pi + 2Es and all the gear to stack them and pass power through the standoffs. Turns out though there’s no mainline kernel support for the allwinner H3 yet which puts me in a pickle as I wanted to use the nodes for a ceph cluster and those need XFS or BTRFS and therefore a modern kernel. Looks like there won’t be support any time soon either so it’s find a different ARM board with GigE, plenty of USB bandwidth etc. for a cheap price or go x86… but x86 boards are expensive. $35 for the 2E, $80 for a lattepanda and while that gains me USB 3.0 for more bandwidth it compares unfavorably to say a HP microserver for £125 or $160. Or a homebrew server tbh.

      1. Oddly one of my secondary uses for those ARM boards is video encoding. It’s slow as all heck (about 24x slower than my workstation) but far more efficient (3x or so). So while some of the older atoms are rubbish like the 230, 450 etc. the more recent bay trail and cherry trail have been kinda impressive and may be worth considering.

          1. Yup, so dual E5-2670s (total 16 cores of sandybridge at 3GHz) consuming 230W total vs this ARM board (so RAM, networking included) consuming 3W. I could probably get a little more out of it too by turning the GPU off. So while I have no problem saying the ARM board is more efficient by a large margin there are a few too many variables to nail down that margin like the GPU in my workstation or the CPUs not drawing their absolute peak thermal rating all the time.

          2. *edit* didn’t explicitly say. Yes I was using frames per second per watt for those comparisons. So while the FPS was tiny on the ARM board, it wasn’t 230/3=76 times less than the workstation as you’d expect if both were as efficient as each other. It was only 24x less.

    1. There is mainline support for the H3. It does not include all fancy multimedia peripherals, but for a cluster you don’t need them anyway. Kernel 4.8 (at least) can boot a usable system with network and USB on the OrangePI One.

  2. Standoffs, inter-connects, and the zeros themselves x 32 puts you into the mid to high end cell-phone price range with similar overall performance numbers. And that doesn’t include the performance derating from clustering 32 cores across USB2 or the ‘what’s your time worth’ factor. The only way this qualifies as a ‘hack’ is because it is a pointless effort ‘just because.’

    1. It was fun for him to learn. That is the main point. It also looks cool to see all of them connected together. Why is there this type of comment every time someone does something like this? Are you really so daft that you can’t see the WHY?

      1. Because two pis would have been enough to learn. It is simply worthless wrt performance or an actual use case. The only benefit is you learn something. And well – you learn cluster computing as I said from more than one already. And anything beyond two does not provide more learning experience.

    2. Doesn’t that describe the majority of hacks here? I mean in reality the most of the hacks either serve no real purpose or are a more costly way of accomplishing something that could be done with less effort ‘just because.’

  3. For what exactly is a cluster good for? i searched a little bit, but don’t found a good reason? Can you crack for example wpa2 hashes? Or calculate things better/faster. I’ve seen lots of “Cluster”-posts, could you may make a post for what we could use the clusters. Thanks!

  4. The USB on the Pi Zero sucks. One can *maybe* power a mouse from it or *perhaps* a WiFi dongle; the Pi crashes when one plugs it in and reboots. So no cheating with swapping USB devices. I only got my Pi Zeros to work because of a hack which allows them to be configured offline run as USB WiFi devices; then one can ssh into them over USB from a Linux computer (info: http://blog.gbaman.info/?p=699).

    Rather than wasting time on the almost useless USB, which surely one will find have more things broken once one start poking at the paintwork, after all the present “crash right in your face”-design made it through Q&A , then maybe it would be simpler to use an Ethernet-to-RS-485 device to network the RasPi’s.

  5. “… it is impossible to sharpen a pencil with a blunt axe. It is equally vain to try to do it with ten blunt axes instead.”–Edsger Djikstra

    Or, in this case, thirty-two blunt axes made, somewhat appropriately, from Raspberry Pi Zeros.

  6. What’s most impressive about this project:

    Someone managed to get their hands on 32 Pi Zeros

    Yes, I know that they are in stock in most places now. They’re still limited to one per customer, so getting 32 of them is a real chore and a bit expensive.

  7. So there has been alot of people here complaining about no real use for for such a cluster. I was thinking that instead of clustered computing, why not turn it into some sort of low power server device. Maybe have each serve up a different website, each one belonging to a different client that doesn’t expect a large amount of traffic. Something like square space or word press but on a much smaller scale.

    Alternatively I was thinking that a pi3 cluster would be great for setting up micro bungeecord nodes for a larger Minecraft sever. Maybe individual towns, or even small world’s for paid members to get away from the larger community. They could also be good for niche styles of Minecraft.

  8. Nearly one year after the rpi zero was announced, some people do nearly useless stuff with a bunch of them and some (like me) are still waiting to get their hands on a rpi zero for the announced price! i’m pissed!

  9. If two of the holes at the edges of the board were big vias attached to the positive and negative power rails, it would be possible to power the whole array of RPis by using metallic spacers.

  10. i love how people keep saying how useless the pi zero is.. But Mine is running Pi – Hole just fine. Cuts a ton of ads out of my browsing experience from my desktop and cellphones and tablets at home..

  11. “People who are really serious about software should make their own hardware.”–Alan Kay

    The real reason for building clusters is because one of these days it will be transparently obvious to someone as to how to elegantly–simply–craft the software tools needed to make full use of these machines.

  12. You could put them all on stacking headers and write an SPI-ethernet bridge. 20 node selects using GPIO, dual SPI for 10MHz synchronous comms, i2c for issuing management commands. That way you could build 21 pi towers and only one USB connection needed.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s