[Jenny] and I were talking about [Bitluni]’s experiment in scale, where he will take 65,536 cheap microcontrollers, network them all together, and give each one an RGB pixel. From there, antics will surely ensue. Right now, he’s only got 8,192 of them up and running, and already the novel problems and opportunities are rearing their heads.
We all know it from our own hacking. In theory, doing something ten times is ten times doing it once. But then in practice, entirely new phenomena appear as you scale up that were simply not there in the small. Maybe it happens when you repeat it one hundred times, or a thousand.
Viewed positively, this is the property of emergence: how the whole can be more than the sum of its parts, and how biology isn’t just chemistry multiplied by a few million interactions. In our blinky world, a massive wall of LEDs is a display, not just a bunch of pixels.
On the flip side, going from one microcontroller with a 10 mA current draw to 64 Ki controllers, with 655 A, is more than just a difference in scale. You need to learn a new skill set to handle the problem. Making a single prototype is a different problem from making a run of badges for a conference of 5,000 – you’ll need a team, and won’t be able to just hack it alone – not to even mention the parts sourcing woes.
So I loved watching [Bitluni] going through the upscaling. He certainly had an idea of what he was getting himself into, but as with the emerging properties of a big system, there are often emerging problems, and those you can’t always see ahead of time. Have you gotten into a project that scaled itself into something qualitatively different? Tell us about it.
I don’t understand what’s the problem power electronics is a solved thing for 10-15 years already. You can go to local Castorama and buy 250A IGBT welder for 50€. Lost of welding current for cheap price.
It’s not 1990 anymore where you could only go to bazaar and buy massive ex-soviet military welder transformer that needed 400V power and worked only with 4 mm electrodes because smaller ones melted itself before making a joint.
655 Amps at 5V? That would be quite the task. To my understanding, the outputs of welders are quite noisy and the voltage varies significantly.
Possibly using a welder circuit at the front end, then followed by the necessary filtering, then regulation, might be a good start, but it still sounds like a colossal problem.
Maybe someone with a lot of power supply design experience can chime in.
Have a good day.
The moment I read 655 Amps, I thought: step down switch mode power supply.
Example: 256 controllers at 10 mA is 5V 2.56 A, or 50V 256mA. Or even 100V, 128mA.
So 5V 655A becomes 50V 65.5A. Round up to 70A to cover your losses. Or 100V 35A.
Due to the scale you need multiple supplies in parallel, but that 655A figure isn’t as bad as it looks.
It is obvious you don’t.
You need a very clean power supply with very little humm on it and a high resistance for momentanous power surges without dropping any voltage on the whole line. Maybe bitluni need some additional capacitors (Electrolytic) spread out on the planes and faster ceramic ones on each microprocessor.
The power supply isn’t the problem here, you can get 5V/300A at reasonable cost. Distributing that juice is where things get nasty, you’ll probably spend more on copper, connectors and fuses than on the SMPS itself.
When working on HPC clusters, one of our rules of thumb was that any given design could reasonably expected to double in size without problems. Once you tried to quadruple the scale, new problems would arise. Clock distribution will be interesting in this project, as will signal propagation in general. I’m sure the author is already dealing with initialization failures, that is, some number of units will fail to boot properly every time the collection is powered up or reset. While you, should, only rarely see initialization errors of a single unit, even with 8k units, multiple init failures are likely, let alone 65k. BTW, this is even true for highly validated CPUs & GPUs used in large-scale HPC systems.
What Bitluni has done is called “horizontal scaling”. Such scaling is effective only on solving problems that are highly parallel and have very low (or zero) data cross-dependencies. Some problems are exactly like this, others need to be adjusted to fit (like FT -> FFT for example), and some problems don’t fit at all (by chance or by design).
Other than that, such computational cluster will also need to be managed – all nodes need to be optimally loaded in order to exploit the system’s parallelism. Such management needs to track which nodes are busy/idle, to upload code/data to the idle nodes and to download data from completed nodes. It’s statistically expected that multiple nodes can/will need such data transfers at the same time, so either the system’s bus will transfer data to/from multiple nodes if it’s capable to do so, or the transfers will happen sequentially and nodes can happen to be under-utilized.
And then is the question of how power supplies and nodes are serviced/upgraded without downtime.
Reliability, Availability, Serviceability.
I’m not trying to hold Bitluni’s to a higher standard than needed. He’s playing with the idea of a computational cluster, others (like me and you, dear reader) are just enjoying it and simply pointing the obvious hypothetical logical continuation of such a development.
As for the topic of “emergence”, for me it’s the accumulation of negligible factors that becomes a significant factor and crosses the experimenter’s threshold of attention, like this:
Step 1. Connect a single risc-v dev board to the lab power supply. It consumes 10mA, it’s ok/
Step 2. Connect 16 boards with 16 mcus each, and power them up. Damn, my power supply was a 2A model…
Thanks for the term “data cross-dependencies”.
GPUs can access lots of model and texture data. This cluster does not have that. Maybe it can raytrace, but the scene would have to be rather simple.
Note that this machine is a sponsored advertisement piece commissioned by Altium, for their development platform. It’s stated so directly in the video.
All the discussion about the efficiency or organization as a computing cluster are void, because it was never designed to do that. Its sole purpose is to blink LEDs.
…and of course, generate clicks on Youtube. You’d think that people would recognize viral marketing by now.
Why does this matter? Well, Google’s advertising revenue is around $240 billion and increasing year on year, and around 80% of that is coming from North America and the EU, which is roughly 1 billion people.
In other words, the people in these regions are paying Google around $200 a year or $16 a month. That’s like a Netflix subscription you cannot cancel, because the money is charged on you through the prices of everything you buy. It’s almost like a private VAT. This is why, promoting these people and projects whose main purpose is collecting some form of advertising revenue is pointlessly pushing cost on everyone and causing this useless “ad-economy” to balloon further out of control.
It’s also crazy expensive and inefficient to sell stuff through online ads. It’s basically grinding large volumes at tiny margins, wasting a dollar of your customers’ money to earn a penny yourself.
https://verdemedia.com/blog/the-rising-cost-of-online-advertising-and-what-to-do-about-it
It looks like some major reduction in current per device is needed, down to 100 or 200 microAmps. Of course this will reduce speed. Otherwise he will create a large room heater, with ENIAC-like reliability problems.
This is mostly outside my wheelhouse, but my first thought was “Didn’t we go through some of this with vacuum tubes already?” 18,000 for ENIAC, And I imagine “simulating” those tubes with LEDs would be a more reasonable goal, he’d be almost halfway there now. 65.5k thought? Good luck!
In general, yes. But I would actually say that if we keep digging into that, we’ll find out that what’s really important is not the actual power consumption of the chip. It is:
Computational efficiency of the cpu/mcu/whatever, measured in DMIPS/MHz. This makes it easy to compare different technologies via a benchmark with normalized results.
Energy efficiency, measured in mA/DMIPS/MHz. Now you can completely compare apples with apples in a logical, deductive way.
What I want to say is that it doesn’t really matter the actual power consumption, it’s more important how much work is done per unit of energy. 1 MCU can consume 10mA for example and another – 100mA, but if the 2nd one produces more than 10x amount of results for equal time, then, yeah, it’s more efficient. And you can also lower the clock of the 2nd MCU to achieve 10mA of power consumption, it will still be more efficient.
@Dude – Yep, I agree with you.
But still, Bitluni’s work is out there in the public. I really admire and appreciate that he shares all his development, especially including the mistakes and wrong design choices. We need to be aware that out there in the world, there are lots of (young) people without lot/any experience in HPC (High-Performance Computing) and it will be beneficial to say to these guys: look, Bitluni’s toy project looks like the beginning of a cluster, and clusters behave like this and this, and you’ll most probably have such and such problems, and they can usually be solved in these typical ways.
The HPC community has endured a lot for the last 70-80 years and we need to make sure this experience continues to be shared among researchers and hobbyists, in order to allow us to grow new experiences on top of it, instead of just repeating it (due to accidental lack of knowledge).
PS: There’s also the special case where people are intentionally trying to recreate some foundational technologies from scratch, for the sake of better understanding and self-development. I deeply respect that, even if other people can say we’re “reinventing the hot water and the wheel”. There’s nothing wrong with it, as long as it’s intentional and conscious choice.
Kind regards to all.
At least it’s sponsored directly by a company that I don’t necessarily have to deal with. As such, this would be fine.
But they’re also clickfarming it on Youtube, acting as a serf for a big corporation that already has its hands in my pockets regardless of my consent and is constantly trying to sell my privacy to take even more. After a point, such behavior becomes a question of ethics.
I’ve taken my Light Herder Video Feedback Fractal Device from 720p (the smaller HD resolution) to 4k 60fps. The emergent behaviors are even more spectacular now, but the difficulty making this upgrade was enormous. I needed to be able to control 4k screens’ hue/saturation/brightness/contrast parameters in real-time with analog knobs. I accomplished that, and much more than I ever expected to. I’m in the process of putting together the big “reveal” video. Stand by…
On Hackaday here https://hackaday.com/2021/07/28/video-feedback-effects-make-a-glorious-spectacle-in-hd/ and here https://hackaday.com/tag/video-feedback/
Hey Dave, this is a fine piece of modern digital art, thanks for sharing!
Thanks! (but, it’s analog art ;-) ) Video to come…
Imagine the arcing fault 655A could support…