LLM

103 Articles

AI On Your Gaming PC

October 4, 2026 by 13 Comments

If you want to experiment with LLMs, you typically have a choice of sending your requests to someone else’s computer or fielding a very large GPU and CPU setup to run models locally. However, a recent crop of projects aims to bring bigger models to much more modest hardware.

One example is Strata, a project from [Niko1221], which lets you run a 125-billion-parameter LLM on hardware you might already have for gaming. It won’t run on your old Pentium laptop, but it doesn’t require a supercomputer-like farm of graphics cards, either.

Strata can use several Qwen3.8 model variants, including different quantizations of the original model as well as coding and other specialized versions. Qwen3.8-Flash-Next is a mixture-of-experts model containing 24,576 small experts, of which only ten are needed for each token. The clever part is that Strata effectively treats VRAM as a cache for the much larger model. Frequently used experts stay on the GPU, while the complete collection normally remains in system RAM. The model also includes a roughly 29 GB lookup table that stays on the SSD and is accessed as needed.

The software also uses the model’s multi-token prediction machinery for speculative decoding, allowing several candidate tokens to be checked in a single pass. According to the project, an RTX 5070 with 12 GB of VRAM can produce roughly 50 to 90 tokens per second, depending on quantization. Tokens, of course, aren’t usually entire words, but it is still a respectable clip, once everything gets set up.

We did have some trouble setting everything up due to some incompatibility with the NVIDIA C compiler, our gcc version, and some headers, but your problems will surely be different. The setup.sh file asks you a few questions on the first run. After that, it just handles your selected startup options, which can take a few minutes while everything loads.

Once running, Strata lets you interact through a web browser. It also exposes OpenAI- and Anthropic-compatible APIs on localhost, so existing chat front ends, coding assistants, and other tools can use the local model without much special handling. Of course, you can’t expect its answers to compete with the big models out there for every task. When asking about Hackaday, for example, it got a lot of it right but also got confused about who founded the site and our authors (unless we forgot that [Tom Nardelli] once wrote some posts). Turning up the “thinking level” and turning down the temperature didn’t help much, although it did move its confusion to different facts. It did better when asked to identify some problem code or outline how to port a particular C compiler to a new target.

You’ll still want at least 32 GB of system RAM, 12 GB of VRAM, and around 80 GB of storage, so “modest” is relative. Still, it’s a neat demonstration of how mixture-of-experts models and some clever memory management can stretch ordinary PC hardware surprisingly far.

These economical LLMs can even run on older hardware, just slower.

FLOSS Weekly Episode 880: The Two Wolves

September 2, 2026 by 2 Comments

This week Jonathan chats with Benjamin Samuels of Trail of Bits! The conversation focuses on Patch the Planet, a new initiative to help Open Source projects deal with the fallout from AI coding and vulnerability research. What’s the unexpected dichotomy driving the polarized response to LLMs? And what does the future look like for Open Source in the age of AI? Watch to find out!

Continue reading “FLOSS Weekly Episode 880: The Two Wolves” →

LLM Moats Quickly Evaporating

August 30, 2026 by 55 Comments

In the business world, a moat is a quality of a business that makes it difficult for competitors to take that company’s profits. With how hard it is to train models for large language models (LLMs) and generative AI, it might seem like Anthropic, Open AI, and other LLM companies would have huge moats given the amount of compute it takes to build models. But open source models are quickly draining that moat, and now the only thing standing in the way of a customer using one of these models on their own hardware instead one from the larger companies is physical computing resources. [TerminalBytes] demonstrates a few of these models on personally owned computers to show the current state of the art.

[TerminalBytes] started off running the 27B version of the Qwen3.8 on a Mac Studio with 256 GB of unified RAM, which is plenty for this task. But it’s also enough to benchmark a few different models. Qwen3.6 is compared to 3.8, and then the different quants of each model are also compared. Quants are compressed versions of models that need fewer bits to store weights, meaning that the same models can run in less memory with smaller losses in fidelity. Many of these quants run on machines with 32 GB of RAM or less, encompassing many average gaming PCs. There’s even a 1-bit quant that [TerminalBytes] tested which can easily run on a machine with 16 GB, although with mixed results.

Keep in mind that this is just the current state of affairs with open LLMs. Future versions of these models are likely to optimize the number of tokens produced per unit time, or otherwise increase quality of responses while requiring less computer resources. We don’t really think that the ease of running local models will be the sole reason that the AI bubble pops, though. The fact that not every computer user is running Linux is proof enough of that.

Same As It Ever Was

August 29, 2026 by 143 Comments

Whether you like it or not, the use of LLMs to write code is kind of a big deal at the moment. We’ve been asking ourselves what, if anything, this means for us here at Hackaday. Should we try to figure out what percentage of a project was done by an actual human and how much was done by a machine? Does it really matter? What is our AI policy anyway?

Clearly, Hackaday is pro-human. We’re in it for the hackers as much as for the hacks. Our community is, like Soylent Green, made of people. It’s your inspirations and innovations that keep us reading and writing every day. And we produce 100% of our content the old-fashioned way, with projects selected through the taste and judgement of our writers, and their own words telling the story.

What about the hacks? We’ve seen a lot of projects recently that were coded with the help of an LLM. Does that diminish the work? In the end, what rings truest to us is what has always been Hackaday’s editorial guiding star: Is there something special in the hack that makes it worth talking about? Then we write about it. Was it written using vim or emacs? Did the author consult friends or a chatbot while working on the project? That’s not really relevant.

But in the past few years, the BS-generation machines have found our hobby, and we’re finding a lot more projects that don’t have any spark to them. We’re seeing circuits that make no sense, and claims that defy physics. Of course, we always have. The LLM-nonsense project is today’s version of the perpetual motion machines of old. Just like we never trust a hardware project that is all renders, seeing only AI-generated images is a huge red flag. It’s our job to separate out the wheat from the chaff for you all, but it’s something that you must be doing everyday as well.

We’ve seen amazing hacks over Hackaday’s 22-year history. Hackaday is older than YouTube and older than Stack Overflow. We’ve seen technology come and go. We’ve seen C-beams glitter in the dark near the Tannhäuser gate. (OK, maybe not.) And in the end, our AI policy is our same-old policy: we write up hacks that inspire us in the hope that they inspire you.

So if you’re using Claude to help you with the UI bits, or if you’re hand-writing it all in assembly, or wiring up the logic in diodes, we just want to see your cool hacks. And we hope that our collective signal will be so loud that we drown out the noise, at least in our own little corner of the hacker universe.

This article is part of the Hackaday.com newsletter, delivered every seven days for each of the last 200+ weeks. It also includes our favorite articles from the last seven days that you can see on the web version of the newsletter. Want this type of article to hit your inbox every Friday morning? You should sign up!

This Week In Security: BugTraq, AI Hacks, And Being Dumb On Planes

August 14, 2026 by 17 Comments

After a multi-year hiatus, the venerable BugTraq mailing list is back!

For decades, BugTraq was the place where vulnerabilities were disclosed, from the early days when nearly all vendors viewed all security research as a hostile force, through to the modern era of working with vendors to coordinate disclosing bugs. With the rise of bug bounty programs and other social changes, the mailing list slowly died: what started in 1993 ended in 2021 is returning. The new maintainer, Jonathan Brossard, says in his announcement “The mission is unchanged: full disclosure, researcher-first, no corporate filter.”

Don’t Be Dumb on Planes

In the unlikely event anyone here needs to be told: Don’t do dumb things on planes.

It seems that someone coming home from from the DEF CON hacker conference in Las Vegas decided to mess with the in-plane WiFi, and is likely now in the “find out” phase of doing something dumb. There hasn’t been any public followup beyond the original reports: a passenger on a Delta flight leaving Las Vegas brought up a fake WiFi hotspot named “Delta WiFi FAST” to trick other passengers into connecting, and attempted to disable the in-flight WiFi using a denial of service attack.

Continue reading “This Week In Security: BugTraq, AI Hacks, And Being Dumb On Planes” →

Wrangling Datacenter GPUs Into A Desktop

July 30, 2026 by 20 Comments

As we’ve seen many times before, there’s usually some way wrangle a bit more life out of what would otherwise be considered old and obsolete technology. Perhaps one thing that has been passed over by the masses a bit to early is older datacenter GPUs, which is understandable in one sense because of the rate NVIDIA is pumping out new ones, but these cards have plenty of useful life left in them for the average person, as [Andrew] demonstrates.

The cards [Andrew] is using are Tesla V100s of 2017 vintage. Despite being older hardware they have high-speed memory which allows them to run modern LLMs locally, competitively with online models. In this test, Gemma 4 26B and Qwen3 35B are run, with Gemma being a bit faster because it fits entirely in GPU memory and Qwen3 being a bit more capable but more hungry for resources. [Andrew] built a PCI card that can host two V100s, allowing these larger models to fit completely in memory.

Even though these don’t perform at the same level as the latest top-tier online models, they’re surprisingly capable and also have the benefit of running completely locally. This might be concerning for those looking at the global economy being propped up by companies that essentially have no moat for motivated users, especially as more and more datacenter hardware becomes available on the secondhand market. While this build by [Andrew] goes into detail on getting the software stack up and running, we recently featured another build using the same GPUs that focuses a bit more on hardware for those looking to get started with local hosting.

Chain-of-Thought Spoofing Targets Reasoning AI Models

July 2, 2026 by 23 Comments

Researchers [Charles Ye], [Jasmine Cui], and [Dylan Hadfield-Menell] have shown that AI Large Language Models (LLMs) can fail to correctly distinguish between different instruction sources because they prioritize writing style over metadata tags, and this role confusion leads to a powerful attack called CoT (Chain of Thought) Forgery. We’ll explain exactly how it works after a bit of background review.

Prompt injection was where “getting an LLM to do something it shouldn’t” started by exploiting the fact that LLMs communicate like people, but are much more obedient. For a while, simply telling an LLM “ignore all previous instructions and <do something funny>” yielded results no matter how transparently dumb the instructions were, and the reason it worked at all was because LLMs do not have separate data and instruction streams; it’s all one big lump of input. It’s up to the model to sort legit instructions from untrusted, user-provided data. One step towards mitigating this was the addition of roles. Continue reading “Chain-of-Thought Spoofing Targets Reasoning AI Models” →