Claude Plays DOOM

August 26, 2026 by Zoe Skyforest 31 Comments

Large language models (LLMs) are generally thought of as machines that accept textual prompts and spit out textual content. However, if you’re creative in the way you interface with them, you can get them to do a wider range of tasks. For example, [Andrea Ricci] figured out how to get one to play DOOM.

For this project, [Andrea] began by porting the game to the SCINTIX P4. It’s a rather interesting device, being a single board designed in the Raspberry Pi CM4/CM5 form factor, but carrying an ESP32-P4 and an ESP32-C6 instead. The game runs on the P4 and is displayed on a 1024×600 MIPI DSI panel, but it’s only stepped through a few frames at a time. These frames are then passed to Claude Sonnet via a WebSockets setup. With only the same information as a human player would get, the LLM has to figure out what it’s looking at, and then respond with movement and fire commands to play the game.

It’s quite interesting to watch the system play—the LLM mostly accurately describes the game world, navigates down corridors, opens doors, and shoots at enemies. There is a bit of work behind the scenes to enable it to see and understand the game world—namely, using a depth fan across the field of view so it can figure out where walls are and how not to bang into them. There’s also an ASCII automap used to allow the system to keep track of where it has already been. But fundamentally, the LLM is playing the game without any other sort of additional assistance.

We’ve seen some other great ways in which AIs have been whipped up to play various games, like Trackmania.

Continue reading “Claude Plays DOOM“ →

New Controller Makes Heavy Machinery Intuitive

August 26, 2026 by Bryan Cockfield 33 Comments

As children, many of us looked wistfully into the cockpits of heavy machinery, wondering just how the series of knobs and levers would do something like operate a bulldozer, crane, or excavator. The nature of these myriad of hydraulic and electronic controls for equipment like this is often inscrutable to adults as well; it takes a considerable amount of training to be able to competently operate most of these machines. But this new controller from MIT may help shorten that training time.

The controller is specifically meant for excavators. In a standard excavator, a pair of joysticks is typically used, with one controlling the swing and the boom and the other controlling the stick and the bucket. Getting used to this combination can take practice, so instead the group of researchers replaced them with a model excavator arm that the operator controls directly with their own arm. The new controller is more intuitive to use as it translates the movements of the model to that of either a real excavator or a training simulation.

The researchers plan to include haptic feedback in future versions, which will hopefully further increase the ease of which new operators can get a feel for using these machines. For those not working towards a new career or an ambitious weekend with rental equipment, there are some other ways of learning how to operate excavators and other pieces of heavy machinery.