If you’ve programmed much in Linux or Unix, you’ve probably run into the fork system call. A call to fork causes your existing process — everything about it — to suddenly split into two complete copies. But they run on the same CPU. [Tristan Hume] had an idea. He wanted to have a call, telefork, that would create the copy on a different machine in a Linux cluster. He couldn’t let the idea go, so he finally wrote the code to do it himself.
If you think about it, parts of the problem are easy while others are very difficult. For example, creating a copy of the process’s code and data isn’t that hard. Since the target is a cluster, the machines are mostly the same — it’s not as though you are trying to move a Linux process to a Windows machine.
Continue reading “Beam Your Program To Another Computer”
Raspberry Pi clusters are a common enough project, but a lot of the builds we see focus on the hardware side of the cluster. Once it’s up and running, though, what comes next? Raspberry Pis aren’t very powerful devices, but they can still be a great project for learning how to interact with a cluster of computers or for experimental test setups. In this project from [Dino], four Pis are networked together and then loaded with a basic set of software for cluster computing.
The first thing to set up, after the hardware and OS, is the network configuration. Each Pi needs a static IP in order to communicate properly. In this case, [Dino] makes extensive use of SSH. From there, he gets to work installing Prometheus and Grafana to use as monitoring software which can track system resources and operating temperature. After that, the final step is to install Ansible which is monitoring software specifically meant for clusters, which allows all of the computers to be administered more as a unit than as four separate devices.
This was only part 1 of [Dino]’s dive into cluster computing, and we hope there’s more to come. There’s a lot to do with a computer cluster, and once you learn the ropes with a Raspberry Pi setup like this it will be a lot easier to move on to a more powerful (and expensive) setup that can power through some serious work.
In it, the user is challenged in their knowledge of how to use the ubiquitous
curl in a variety of different ways. Perhaps this doesn’t sound terribly daunting, especially if your knowledge of curl is limited to knowing it is a command-line tool to fetch something from a web server. But curl has a staggering number of features. The man page is over 4500 lines in length. The software’s main site offers a (free) 250+ page guide on how to use curl and libcurl. Reflecting on this is exactly what led [Ben] to create his challenge.
It’s a wonderful piece of work, but things get really interesting once [Ben] starts talking about the infrastructure behind it all. At its core the game works by giving the user a problem and a virtual machine, and catching outgoing HTTP calls to see whether they look correct. If the outgoing HTTP call is the right solution for the problem, terminate the current VM and start up the next one with the next problem. He’s put a lot of work into getting suitable VMs up and running quickly, securely, and properly isolated. The code can be found on the project’s GitHub repository for those who want a closer look.
But that’s not all. [Ben] says that in the past he’s had a bad habit of presenting interactive features in his blog posts that can’t keep up with sudden demand. So to address that, the system auto-scales as needed with a small Linux cluster; small brick-sized PCs are started and shut down automatically to meet demand. Hey, the only thing cooler than a functioning cluster is a cluster doing an actual job, like this one that detects NSFW images.
Many readers will have had their first taste of experimentation with cluster computing through the medium of the Raspberry Pi. The diminutive Linux capable boards can easily be hooked up as a group via a network hub, and given the right software become a whole that is greater than the sum of its parts. None of us will however have reached the heights of the Raspberry Pi cluster shown by Oracle at their Oracle OpenWorld conference, a mighty rack packing a cluster of no less than 1060 Pi 3 B+ boards. This machine is touted as a supercomputer and it’s worthy of the name, though perhaps it’s not in the same class as the elite in that field.
Getting that number of individual 3Bs into a human-sized rack is no easy feat, and they have gone for custom 3D-printed racks to hold the boards. PoE would have resulted in too much heat dissipation, so instead they use USB power from an array of large multi-way USB power supplies. A set of switches provide the networking, and a conventional server sits in the middle to provide storage and network booting.
It’s certainly a cool way to wow the crowds at a conference, but we’re unsure whether it delivers the best bang for your supercomputing buck or whether it’s more useful as a large room heater. Meanwhile you can take a look at a few more modest Pi clusters, with unusual operating systems, or slightly more adherence to convention.
Thanks [Frisco] for the tip.
It’s that time of year again, with the holidays fast approaching friends and family will be hounding you about what trinkets and shiny baubles they can pretend to surprise you with. Unfortunately there’s no person harder to shop for than the maker or hacker: if we want it, we’ve probably already built the thing. Or at least gotten it out of somebody else’s trash.
But if they absolutely, positively, simply have to buy you something that’s commercially made, then you could do worse than pointing them to this very slick Raspberry Pi cluster backplane from [miniNodes]. With the ability to support up to five of the often overlooked Pi Compute Modules, this little device will let you bring a punchy little ARM cluster online without having to build something from scratch.
The Compute Module is perfectly suited for clustering applications like this due to its much smaller size compared to the full-size Raspberry Pi, but we don’t see it get used that often because it needs to be jacked into an appropriate SODIMM connector. This makes it effectively useless for prototyping and quickly thrown together hacks (I.E. everything most people use the Pi for), and really only suitable for finished products and industrial applications. It’s really the line in the sand between playing around with the Pi and putting it to real work.
[miniNodes] calls their handy little device the Carrier Board, and beyond the obvious five SODIMM slots for the Pis to live in, there’s also an integrated gigabit switch with an uplink port to get them all connected to the network. The board powers all of the nodes through a single barrel connector on the side opposite the Ethernet jack, leaving behind the masses of spider’s web of USB cables we usually see with Pi clusters.
The board doesn’t come cheap at $259 USD, plus the five Pi Compute Modules which will set you back another $150. But for the ticket price you’ll have a 20 core ARM cluster with 5 GB of RAM and 20 GB of flash storage in a 200 x 100 millimeter (8 x 4 inch) footprint, with an energy consumption of under 20 watts when running at wide open throttle. This could be an excellent choice for mobile applications, or if you just want to experiment with parallel processing on a desktop-sized device.
Amazon is ready for the coming ARM server revolution, are you? Between products like this and the many DIY ARM clusters we’ve seen over the years, it looks like we’re going to be dragging the plucky architecture kicking and screaming into the world of high performance computing.
[Thanks to Baldpower for the tip.]
In today’s world of over-the-air firmware upgrades in everything from cars to phones to refrigerators, it’s common for manufacturers of various things to lock out features in software and force you to pay for the upgrades. Even if the hardware is the same across all the models, you can still be on the hook if you want to unlock anything extra. And, it seems as though Suzuki might be following this trend as well, as [Sebastian] found out when he opened up his 2011 Vstrom motorcycle.
The main feature that was lacking on this bike was a gear indicator. Even though all the hardware was available in the gearbox, and the ECU was able to know the current gear in use, there was no indicator on the gauge cluster. By using an Arduino paired with an OBD reading tool (even motorcycles make use of OBD these days), [Sebastian] was able to wire an LED ring into the gauge cluster to show the current gear while he’s riding.
The build is very professionally done and is so well blended into the gauge cluster that even we had a hard time spotting it at first. While this feature might require some additional lighting on the gauge cluster for Suzuki to be able to offer this feature, we have seen other “missing” features in devices that could be unlocked with a laughably small amount of effort.
Continue reading “Adding Upgrades To A Stock Motorcycle”
There seems to be a universal truth on the Internet: if you open up a service to the world, eventually somebody will come in and try to mess it up. If you have a comment section, trolls will come in and fill it with pedantic complaints (so we’ve heard anyway, naturally we have no experience with such matters). If you have a service where people can upload files, then it’s a guarantee that something unsavory is eventually going to take up residence on your server.
Unfortunately, that’s exactly what [Christian Haschek] found while developing his open source image hosting platform, PictShare. He was alerted to some unsavory pictures on PictShare, and after he dealt with them he realized these could be the proverbial tip of the iceberg. But there were far too many pictures on the system to check manually. He decided to build a system that could search for NSFW images using a trained neural network.
The nude-sniffing cluster is made up of a trio of Raspberry Pi computers, each with its own Movidius neural compute stick to perform the heavy lifting. [Christian] explains how he installed the compute stick SDK and Yahoo’s open source learning module for identifying questionable images, the aptly named open_nsfw. The system can be scaled up by adding more Pis to the system, and since it’s all ARM processors and compute sticks, it’s energy efficient enough the whole system can run off a 10 watt solar panel.
After opening up the system with a public web interface where users can scan their own images, he offered his system’s services to a large image hosting provider to see what it would find. Shockingly, the system was able to find over 3,000 images that contained suspected child pornography. The appropriate authorities were notified, and [Christian] encourages anyone else looking to search their servers for this kind of content to drop him a line. Truly hacking for good.
This isn’t the first time we’ve seen Intel’s Movidius compute stick in the wild., and of course we’ve seen our fair share of Raspberry Pi clusters. From 750 node monsters down to builds which are far more show than go.