It became something of a cliché a few years ago in online discussions, whenever a new single board computer was mentioned someone would pop up and say something like “Imagine a Beowulf cluster…“. Back then it was said largely in jest, but with the current generation of boards it’s a distinct possibility. Who hasn’t looked at a Raspberry Pi and idly thought about a cluster of them, or even created one!
[Electronoob] did just that, creating a variety of Raspberry Pi cluster configurations, the most impressive of which is a stack of 32 Pi Zeros mounted together with stand-offs. The plan was to network it via USB, for which he initially considered building a backplane, but was put off by the cost of vertical USB connectors and instead went for a wired approach. If there is a lesson to be learned from his experiences it is that buying very cheap USB cables is a minefield: his pile of eBay specials turned out to have significant numbers of faults. He’s now faced with a stark choice, solder 32 sets of USB pads on the base of each Zero or buy better cables.
The stack of Zeros is pretty impressive, but so what, you think. It’s still not working properly. But the Zero cluster isn’t his only work. He’s also created a set of very nicely executed Ethernet clusters using the larger Pi boards, and the way he’s mounted them on top of compact Ethernet switches sets them apart from some of the more spaghetti-like Pi clusters.
It’s true a Pi cluster won’t cut it in the world of supercomputers, you could almost certainly buy more bang for your buck without too much effort. But it does represent a very accessible way to learn about cluster computing, and you have to admit it a stack of Zeros does look rather impressive.
We’ve seen quite a few Pi clusters here since 2012, the biggest of which is probably this 120 node behemoth, complete with screens.
How much performance is it? (esp compared to Raspberry Pi 3)
Probably like 3 or 4 Raspberry Pi 3s if optimized correctly. Real world probably like 2 Raspberry Pi 3s. But, as always needs to be mentioned in these threads, building a cluster out of Raspberry Pis is not about achieving high performance. They do it because they can and because it’s awesome, not because it’s practical.
It is practical.But not for performance reasons. Such cluster is a great parallel programming learning tool. You get 32 completely separate nodes, with network communication between them This way you can practice programming with MPI, PVM, or any other distributed computing tool, using isolated nodes communicating only over the network.
This is probably the easiest, cheapest and quickest way to gain experience which will be applicable in real, high-performance cluster computing environments.
Probably easier and cheaper with some VM nodes but I don’t know how high fidelity the network latency, bandwidth, etc would be simulated that way.
Cheaper? for sure. Easier? depends. Having piece of real hardware which you can touch, or throw through the window when things get sour – priceless.
All I can think is – realtime non-linux-core or bare metal programming on each of those. 32 completely separate hard real time units. Can’t beat realtime performance of that with no Raspi 3…
If i clustered a few raspberry pi zero together could i get expanded use of the hats??? I want to build a droid with a pi zero controlling the physical movement and another conducting gps and another running a chatbot application but all integrated. Any advice???
I am wondering with 2 nos parallel x Pi Zero W RasPi s , can we do a better imaging program using python open CV and dlib
I was working on a stacked cluster until recently. Got a couple of Orange Pi + 2Es and all the gear to stack them and pass power through the standoffs. Turns out though there’s no mainline kernel support for the allwinner H3 yet which puts me in a pickle as I wanted to use the nodes for a ceph cluster and those need XFS or BTRFS and therefore a modern kernel. Looks like there won’t be support any time soon either so it’s find a different ARM board with GigE, plenty of USB bandwidth etc. for a cheap price or go x86… but x86 boards are expensive. $35 for the 2E, $80 for a lattepanda and while that gains me USB 3.0 for more bandwidth it compares unfavorably to say a HP microserver for £125 or $160. Or a homebrew server tbh.
Careful of those x86 boards if you really want to crunch something, some of the atoms have original pentium like ipc and horrendous fpu.
Oddly one of my secondary uses for those ARM boards is video encoding. It’s slow as all heck (about 24x slower than my workstation) but far more efficient (3x or so). So while some of the older atoms are rubbish like the 230, 450 etc. the more recent bay trail and cherry trail have been kinda impressive and may be worth considering.
Is the efficiency normalized against the computing power?
Or against say a 15W i5
Yup, so dual E5-2670s (total 16 cores of sandybridge at 3GHz) consuming 230W total vs this ARM board (so RAM, networking included) consuming 3W. I could probably get a little more out of it too by turning the GPU off. So while I have no problem saying the ARM board is more efficient by a large margin there are a few too many variables to nail down that margin like the GPU in my workstation or the CPUs not drawing their absolute peak thermal rating all the time.
*edit* didn’t explicitly say. Yes I was using frames per second per watt for those comparisons. So while the FPS was tiny on the ARM board, it wasn’t 230/3=76 times less than the workstation as you’d expect if both were as efficient as each other. It was only 24x less.
Not sure I understand, Atom isn’t ARM based
The leap is that if some cheap low power ARMs beat big x86 for efficiency maybe cheap low power x86s can beat big x86 too. Software compatibility maintained and more bang for your watt, everyone wins potentially.
There is mainline support for the H3. It does not include all fancy multimedia peripherals, but for a cluster you don’t need them anyway. Kernel 4.8 (at least) can boot a usable system with network and USB on the OrangePI One.
I just went through armbian’s compiler script which got me an image with kernel version 4.7.4. Networking doesn’t work on that so maybe it does need 4.8 which isn’t out just yet.
Actually Newegg has an AMD C-70 mini itx motherboard for only $40 regular price. Two GB of ram and a power supply puts it at around $80. For that you will get get GigE. You even have a PCIe slot
Standoffs, inter-connects, and the zeros themselves x 32 puts you into the mid to high end cell-phone price range with similar overall performance numbers. And that doesn’t include the performance derating from clustering 32 cores across USB2 or the ‘what’s your time worth’ factor. The only way this qualifies as a ‘hack’ is because it is a pointless effort ‘just because.’
It was fun for him to learn. That is the main point. It also looks cool to see all of them connected together. Why is there this type of comment every time someone does something like this? Are you really so daft that you can’t see the WHY?
Because two pis would have been enough to learn. It is simply worthless wrt performance or an actual use case. The only benefit is you learn something. And well – you learn cluster computing as I said from more than one already. And anything beyond two does not provide more learning experience.
2 pis would not be enough to learn – dividing a process in half is relatively easy; splitting a process 32 ways and getting 32x speedup is completely non-trivial.
Doesn’t that describe the majority of hacks here? I mean in reality the most of the hacks either serve no real purpose or are a more costly way of accomplishing something that could be done with less effort ‘just because.’
For what exactly is a cluster good for? i searched a little bit, but don’t found a good reason? Can you crack for example wpa2 hashes? Or calculate things better/faster. I’ve seen lots of “Cluster”-posts, could you may make a post for what we could use the clusters. Thanks!
Yeah I’ve been struggling to find a reason to combine machines into one cluster. A lot of machines on one task? Eh. A lot of machines on separate but related tasks? Yes please. Why convert one video at a time when you can convert an entire season at once!
Massive database operations for one.
Sadly most of the cluster experts don’t understand the application for parallel computing except configuring one !! You need a Rasp i OS that supports parallel computing else common sense tells us how do we instruct the parallel computer for running an application !! :))
That “imagine a Beowulf cluster…“ cliché can be traced even further back to the heyday of Slashdot, before SBCs were really a thing.
Yeah I think it was a cliche by the time the duron morgan to MP hack was figured.
If I’m reading this correctly, it’s currently just 32 Pi Zero’s placed next to each other? I look forward to seeing how he solves the networking problem.
Heh heh yeah it’s almost just a “look I can do lego!” at the moment.
Wish there was a board that combined several of the usb-to-ethernet chips with an ethernet switch chip.
The USB on the Pi Zero sucks. One can *maybe* power a mouse from it or *perhaps* a WiFi dongle; the Pi crashes when one plugs it in and reboots. So no cheating with swapping USB devices. I only got my Pi Zeros to work because of a hack which allows them to be configured offline run as USB WiFi devices; then one can ssh into them over USB from a Linux computer (info: http://blog.gbaman.info/?p=699).
Rather than wasting time on the almost useless USB, which surely one will find have more things broken once one start poking at the paintwork, after all the present “crash right in your face”-design made it through Q&A , then maybe it would be simpler to use an Ethernet-to-RS-485 device to network the RasPi’s.
I am wondering with 2 nos parallel x Pi Zero W RasPi s , can we do a better imaging program using open CV and dlib. Is there a separate Raspi OS for parallel computing operations.
“… it is impossible to sharpen a pencil with a blunt axe. It is equally vain to try to do it with ten blunt axes instead.”–Edsger Djikstra
Or, in this case, thirty-two blunt axes made, somewhat appropriately, from Raspberry Pi Zeros.
I think that analogy fails. It’s more like trying to sharpen a pencil with a very short piece of blade.
What’s most impressive about this project:
Someone managed to get their hands on 32 Pi Zeros
Yes, I know that they are in stock in most places now. They’re still limited to one per customer, so getting 32 of them is a real chore and a bit expensive.
That was my first thought, Getting my hands on two (1.0 & 1.5) was a nearly herculean task.
Microcenter will sell you one a day for 99 cents.
MicroCenter will be giving them away in six months.
Six months after that, they’ll be giving away RPi 2s.
So there has been alot of people here complaining about no real use for for such a cluster. I was thinking that instead of clustered computing, why not turn it into some sort of low power server device. Maybe have each serve up a different website, each one belonging to a different client that doesn’t expect a large amount of traffic. Something like square space or word press but on a much smaller scale.
Alternatively I was thinking that a pi3 cluster would be great for setting up micro bungeecord nodes for a larger Minecraft sever. Maybe individual towns, or even small world’s for paid members to get away from the larger community. They could also be good for niche styles of Minecraft.
If this is a “Hack” it is a pointless as hacking off your ear.
http://www.newyorker.com/wp-content/uploads/2010/01/100104_r19185_p646-1200-925-23181757.jpg
At 100mA to 150mA each, that’s 3.2 to 4.8 amps. A 25W power supply should do the trick.
Nearly one year after the rpi zero was announced, some people do nearly useless stuff with a bunch of them and some (like me) are still waiting to get their hands on a rpi zero for the announced price! i’m pissed!
got mine this week for £4! pimonio
If two of the holes at the edges of the board were big vias attached to the positive and negative power rails, it would be possible to power the whole array of RPis by using metallic spacers.
But, how this usb network between the Raspi0 will be done?
i love how people keep saying how useless the pi zero is.. But Mine is running Pi – Hole just fine. Cuts a ton of ads out of my browsing experience from my desktop and cellphones and tablets at home..
Forget compute clusters — what I would use this for is to practice HPC techniques in building out xCat and Moab setups.
I’m working on a cluster with the rPi 3, the plan is to fill a 1u rack mount case with around 40 of them. Already got power sorted, just need $$$ https://www.tindie.com/products/NSBoA/5v-usb-hub-switched-pdu-16-port
“People who are really serious about software should make their own hardware.”–Alan Kay
The real reason for building clusters is because one of these days it will be transparently obvious to someone as to how to elegantly–simply–craft the software tools needed to make full use of these machines.
Yeah, use them in 64s… so there’s one per bit :-D
You could put them all on stacking headers and write an SPI-ethernet bridge. 20 node selects using GPIO, dual SPI for 10MHz synchronous comms, i2c for issuing management commands. That way you could build 21 pi towers and only one USB connection needed.