Driven by a desire for privacy, customization, and lower costs, there’s growing interest in AI models which can be run on local hardware. Few of them go as far as [Tim], though, who built an image generation diffusion model which can run on an RP2350 microcontroller.
As might be expected, its capabilities are limited. The resolution is 128×128, it only generates images of human faces, and it takes about twenty seconds per image – still impressive for such limited hardware. It runs on a Waveshare RP2350 development board, and it can output the generated image over USB or display it with the aid of a VGA adapter board.
The generative model doesn’t directly create an image. Rather, it generates a distribution in a latent space, which a variational auto-encoder’s decoder component translates into an image. The auto-encoder was trained in two parts: an encoder which transforms an image into a latent-space distribution, and a decoder to transform that distribution back to an image; once this was trained, only the decoder was used.
The generative portion of the model uses a latent flow diffusion transformer; this takes in noise to start with, then iteratively predicts changes which bring it toward the desired image. It can also take in a output class, which guides the generator’s direction (toward a smiling face, for example). [Tim] trained two models, one larger and one faster, and quantized the weights for both to 8-bit integers. Both models, along with the inference program, then fit into 4 MB of flash memory.
For such a small model, the results are remarkably good; they don’t look quite natural, but they’re quite recognizable. For more on how diffusion image generators work, check out our article on Stable Diffusion.
I totally agree. Ignore what you don’t find valuable. Yet some people think their opinion is holier than thou… best to ignore them too.
Does it work on a 486 with 4 MB of Ram ?
Joke aside, this is amazing that a diffusion model now runs on tiny Hardware.
..
Yes. A 486 with 4 MB of RAM is/was a joke, alright.
A 486 with 16MB though used to run AutoCAD and 3D animation software such as Autodesk 3D Studio. It was a real workstation.
It’s a shame and incompetence that nowadays software is so bloated that it can’t run on high-end systems of yesterday anymore.
run? walk, stumble.
Ok, “brisk walk” then. 😉
Here’s an example of 3D Studio R4 (I think):
https://www.youtube.com/watch?v=SFIcdGRBLV8
Sure, it’s run in emulation merely (I could not find a better one).
– But someone who has used DOSBox before knows that it doesn’t beat a physical Pentium 1.
At max cycles (on semi-current hardware) its performance is roughly the equivalent of a 486DX4 or a slow Pentium.
That’s because single-core performance matters (-emulation can’t be easily multi-threated) and modern CPUs are worse at it than their predecessors were (Pentium 4, Core 2 Duo etc).
(Even PCem v17 and 86Box tops out below Pentium II performance.)
The Autodesk software (-3D or rendering software in general-) also is/was quite FPU heavy.
And the 486DX FPU was very advanced for its day (new x87 interface, cache access). To many it was a game changer.
But that being said, a quality 386 PC with an advanced third-party FPU (Cyrix Fasmath, Weitek) was quite a contender, too.
Especially the late 386DX-33 and 386DX-40, thanks to their higher throughput (bus, memory).
It’s also notable that the example video runs 3D Studio in 800×600 pels resolution (Super VGA).
Performance of simlultanous rendering and displaying at this resolution is quite demanding.
I’m not sure if a native driver for the S3 card was used or a generic one (VBE).
If it’s the latter, then the performance shown isn’t too chabby. 🙂
Especially if linear frame buffer (LFB) isn’t used (VBE 2.0 and up).
On real hardware, the stock ISA bus would have been a bottleneck, also.
VLB, EISA, MCA, Opti local bus or an overclocked ISA bus would have made a difference.
That’s why many ISA-only 486 systems made the 486 CPU look slower than it really was (slower 30pin SIMM RAM and lack of motherboard cache also was a bottleneck).
I think these days the software at the base are eating up a lot of overhead (OS) and the libraries they are calling is bloated. Remember the 1993 demo from Future Crew called 2nd Reality? Used to run smoothly on a 386 DX with 4MB RAM. Ask someone to recreate it today and just watch them heating up their Mac M3 Studio.
What also comes to mind.. So-called “morphing” software was a thing in the 486 days.
On surface level it looked kind of similar to what is done here.
https://en.wikipedia.org/wiki/Morphing
Thats just some basic image interpolation and distortion, much simpler than a diffusion model.
Probably, but it would be much slower. the RP2350 is quite a beast. 2x300MHz core, single cycle SRAM access, and special instructions for parallel MAC. A 486 would not only be clocked much lower but is also slower cycle by cycle.
Yep. I would guess RP2350 is somewhere around 1GHz Pentium 3 performance wise.
Nah, not that much Pentium 3 is ooo superscalaer with IPC>1. CM33 is not quite there.
This sounds like the exact same approach that much larger and more capable image and video generation models use. So it really isn’t that different from Stable Diffusion/Qwen/Flux/Krea/Anima/Wan/etc (and presumably many of the closed models used by ChatGPT/Gemini/Grok/etc).
So the secrets to making it run on a microcontroller instead of a massively parallel GPU seem to boil down to limiting the resolution, training on a very specific goal (faces), and then further quantizing the training weights to reduce memory usage and avoid floating point math.
(Which is still to say, it’s a very impressive achievement!)
Indeed, it is actually quite a modern implementation of a diffusion model with a transformer backbone AdaLN conditionioning, CFG, and flow matching objective.
That was one of the more surprising parts. All the fancy tricks from the large models were also required the get the tiny model to work well!
I applaud this effort, the brute force methodology of LLM design makes them unwieldy and resource hungry. Harnessing and restricting the models is wonderful, despite Nvidias efforts to have us run a GPU on max constantly. I have a $100 10 year old gpu, huang wants me to meter pay for accessing a $20k enterprise gpu
“it only generates images of human faces” Debatable :D
Awesome job btw!
Finally, a computer program that generates british people.
Disappointed they used another board to break out to a monitor instead of a small LCD directly connected.
That would actually have been a lot easier.
I didnt look too deep into it. The driver and whatnot might’ve been a ram limitation too
I love the project, now how can I scale it by using 10 RP2350?
Man I love this!!! Did something similar in May, a 12.6M parameter DCGAN on a CH32H417 RISC-V MCU with no external memory…maybe I’ll incorporate some of his methods for a v2!! Great stuff!! … https://youtu.be/yjWKd108WVo?si=h_wC8kTJ6DGEr5rl