cluster

43 Articles

Raspberry Pi Hive Mind

August 29, 2016 by 34 Comments

Setting up a cluster of computers used to be a high-end trick used in big data centers and labs. After all, buying a bunch of, say, VAX computers runs into money pretty quickly (not even counting the operating expense). Today, though, most of us have a slew of Raspberry Pi computers.

Because the Pi runs Linux (or, at least, can run Linux), there are a wealth of tools out there for doing just about anything. The trick is figuring out how to install it. Clustering several Linux boxes isn’t necessarily difficult, but it does take a lot of work unless you use a special tool. One of those tools is Docker, particularly Docker Swarm Mode. [Alex Ellis] has a good video (see below) showing the details of a 28 CPU cluster.

Continue reading “Raspberry Pi Hive Mind” →

Hackaday Prize Entry: An Oven Of Raspberry Pis

August 13, 2016 by 49 Comments

When the Raspberry Pi was introduced, the world was given a very cheap, usable Linux computer. Cheap is good, and it enables one kind of project that was previously fairly expensive. This, of course, is cluster computing, and now we can imagine an Aronofsky-esque Beowulf cluster in our apartment.

This Hackaday Prize entry is for a 100-board cluster of Raspberry Pis running Hadoop. Has something like this been done before? Most certainly. The trick is getting it right, being able to physically scale the cluster, and putting the right software on it.

The Raspberry Pi doesn’t have connectors in all the right places. The Ethernet and USB is on one side, power input is on another, and god help you if you need a direct serial connection to a Pi in the middle of a stack. This is the physical problem of putting a cluster of Pis together. If you’re exceptionally clever and are using Pi Zeros, you’ll come up with something like this, but for normal Pis, you’ll need an enclosure, a beefy, efficient power supply, and a mess of network switches.

For the software, the team behind this box of Raspberries is turning to Hadoop. Yahoo recently built a Hadoop cluster with 32,000 nodes used for deep learning and other very computationally intensive tasks. This much smaller cluster won’t be used for very demanding work. Instead, this cluster will be used for education, training, and training those ever important STEAM students. It’s big data in a small package, and a great project for the Hackaday Prize.

The HackadayPrize2016 is Sponsored by:

Raspberry Pi Cluster Build Shows How And What

May 26, 2016 by 36 Comments

Raspberry Pi clusters are a dime a dozen these days. Well, maybe more like £250 for a five-Pi cluster. Anyway, this project is a bit different. It’s exquisitely documented.

[Nick Smith] built a 5-node Pi 3 cluster from scratch, laser-cutting his own acrylic case and tearing down a small network switch to include in the design. It is, he happily admits, a solution looking for a problem. [Smith] did an excellent job of documenting how he designed the case in CAD, prototyped it in wood, and how he put the final cluster together with eye-catching clear acrylic.

Of interest is that he even built his own clips to hold the sides of the case together and offers all of the files for anyone who wants to build their own. Head over to his page for the complete bill of materials (we didn’t know Pis were something you could order in 5-packs). And please, next time you work on a project follow [Nick’s] example of how to document it well, and how to show what did (and didn’t) work.

If 5 nodes just doesn’t do it for you, we suggest this 120-node screen-equipped monster, and another clear-acrylic masterpiece housing 40 Pis. This stuff really isn’t only for fun and games. Although it wasn’t Pi-based, here’s a talk at Hackaday Belgrade about an ARM-based SBC cluster built to crunch numbers for university researchers.

Designing A High Performance Parallel Personal Cluster

May 9, 2016 by 86 Comments

Kristina Kapanova is a PhD student at the Bulgarian Academy of Sciences. Her research is taking her to simulations of quantum effects in semiconductor devices, but this field of study requires a supercomputer for billions of calculations. The college had a proper supercomputer, and was getting a new one, but for a while, Kristina and her fellow ramen-eating colleagues were without a big box of computing. To solve this problem, Kristina built her own supercomputer from off-the-shelf ARM boards.

Continue reading “Designing A High Performance Parallel Personal Cluster” →

Moore’s Law Of Raspberry Pi Clusters

April 17, 2015 by 52 Comments

[James J. Guthrie] just published a rather formal announcement that his 4-node Raspberry Pi cluster greatly outperforms a 64-node version. Of course the differentiating factor is the version of the hardware. [James] is using the Raspberry Pi 2 while the larger version used the Model B.

We covered that original build almost three years ago. It’s a cluster called the Iridris Pi supercomputer. The difference is a 700 MHz single core versus the 900 Mhz quad-core with double-the ram. This let [James] benchmark his four-node-wonder at 3.048 gigaflops. You’re a bit fuzzy about what a gigaflops is exactly? So were we… it’s a billion floating point operations per second… which doesn’t matter to your human brain. It’s a ruler with which you can take one type of measurement. This is triple the performance at 1/16th the number of nodes. The cost difference is staggering with the Iridris ringing in at around £2500 and the light-weight 4-node built at just £120. That’s more than an order of magnitude.

Look, there’s nothing fancy to see in [James’] project announcement. Yet. But it seems somewhat monumental to stand back and think that a $35 computer aimed at education is being used to build clusters for crunching Ph.D. level research projects.

40-Node Raspi Cluster

February 17, 2014 by 48 Comments

Multi-node RasPi clusters seem to be a rite of passage these days for hackers working with distributed computing. [Dave’s] 40-node cluster is the latest of the super-Pi creations, and while it’s not the biggest we’ve featured here, it may be the sleekest.

The goal of this project—aside from the obvious desire to test distributed software—was to keep the entire package below the size of a full tower desktop. [Dave’s] design packs the Pi’s in groups of 4 across ten individual cards that easily slide out for access. Each is wired (through beautiful cable management, we must say) to one of the 2 24-port switches at the bottom of the case. The build uses an ATX power supply up top that feeds into individual power for the Pi’s and everything else, including his HD array—5 1TB HD’s, expandable to 12—a wireless router, and a hefty fan assembly.

Perhaps the greatest achievement is the custom acrylic case, which [Dave] lasered out at the Dallas Makerspace (we featured it here last month). Each panel slides off with the press of a button, and the front/back panels provide convenient access to the internal network via some jacks. If you’ve ever been remotely curious about a build like this one, you should cruise over to [Dave’s] page immediately: it’s one of the most meticulously well-documented projects we’ve seen in a long time. Videos after the break.

Continue reading “40-Node Raspi Cluster” →

33 Node Beowulf Cluster Built With Raspberry Pi

May 21, 2013 by 81 Comments

Not only did [Josh Kiepert] build a 33 Node Beowulf Cluster, but he made sure it looks impressive even if you don’t know what it is. That’s thanks to the power distribution PCBs he designed and etched. In addition to injecting power through each of the RPi GPIO headers they host an RGB LED which is illuminated in blue in the images above.

Quite some time ago we saw a 64-node RPi cluster. That one used LEGO pieces as a rack system to hold all of the boards. But [Josh] used stand-offs to create the columns of hardware which are suspended between top and bottom plates made out of acrylic. The only thing that’s unique about each board is the SD card and that’s why each has a label on it that identifies the node. These have been flashed with almost identical images; the host name and IP address are the only thing that changes from one to the next. They’ve been put in order physically so that you can quickly find your way through the rack. But functionally this doesn’t matter… put the card in any RPi and it will automatically identify itself on the network no matter where it’s located in the rack.

Don’t miss the demo video where [Josh] explains the entire setup.

Continue reading “33 Node Beowulf Cluster Built With Raspberry Pi” →