There are lots of reasons to run LLMs locally, from privacy concerns to just wanting to futz about with the technology, but if you want to run the big models, the standard logic is that you need big money for lots and lots of VRAM. [MattMo] is calling that into question with his recent video — embedded below, naturally — in which he gets GLM 5.3 Flash, Qwen 3.8 Flash and Qwen 3.8 27B all running on a 14-year-old server without a single GPU.
The secret, if you can call it that, is that they aren’t running very fast: four tokens per second was about the max. Those four tokens are excreted from the twin Xeon processors of the vintage Dell PowerEdge R720 server, with the models living in its 348 GB of DDR3 system memory. That’s enough even for the largest flagship models, but as you can see by the speed, things are a bit bottlenecked by having only 20 threads available between the two processors. Said processors are also old enough to lack certain instructions that might have helped speed things up. Still, [MattMo] argues in the video that this is more than just a dancing bear: there are workloads where batch-processing at 4tps might make sense, and there are people who already have servers of this class laying around in their homelabs. The intersection of that Venn diagram is probably pretty lonely, but if that’s you — hey! [MattMo] says it’ll work, so give it a shot.
If you had to buy the hardware, well, it’s also quite reasonable on the second-hand market, with [MattMo] estimating about $600 given prevailing prices. He also points out that the newer models may still get some optimization to improve speeds, but don’t expect real-time conversations with Hal 9000. Still, in terms of local LLMs, it certainly beats the pants off toy models running via Llama on the PSP or the C64.
Continue reading “No GPU, No Problem: Flagship LLMs On A GPU-less Teenaged Server”
