The most recent of the Zelda franchise, Breath of the Wild, is known for its many, many puzzles. One of the more frustrating ones involved bowling with a giant snowball at the top of a hillside. [Bertrand] did not like this, so he cheated the system hacked the Nintendo Switch so that he “genuinely earned” a strike every time he played. He achieved this by writing a script for a Teensy module that got him those sweet rupees.
The Teensy houses an Atmel 90USB1286 microcontroller. When paired with LUFA software, it can emulate numerous controllers including keyboards, joysticks, etc. It also handily has a Mini-B USB connector located on its rear, allowing it to communicate to the Switch with ease. After confirming the hardware was compatible, [Bertrand] looked towards the software side noticing the similarity between what already existed and what he was attempting to accomplish. He happened upon this in a Splatoon 2 fork that allows players to draw posts.
In essence, it takes image files as input and emulates the controls and buttons to draw a 1-bit version of the image automatically. This takes care of syncing the hardware as well as how to simulate the button presses. But instead of reading an image file, it needed to take a custom script as the input. This required starting from scratch. The first logical step — of course — was to create a language similar to Logo, a name that surely brings back memories of the time of big hair and shoulder pads. He only needed a handful of simple commands to control Link:
typedef enum {
UP,
DOWN,
LEFT,
RIGHT,
X,
Y,
A,
B,
L,
R,
THROW,
NOTHING,
TRIGGERS
} Buttons_t;
It is notable that the THROW and TRIGGERS are composite commands, meaning that they act as multiple button pushes simultaneously. This takes care of the issue of pressing the R button while nudging the analog stick forward to throw the snowball, as well as unlocking the screen by squeezing both L and R triggers concurrently. The script was then painstakingly, manually input until the perfect conditions for a strike were found. Once those measurements were captured, a script of the entire conversation leading up to the mini game was added to implement a loop that allowed the mini game to be played non-stop on repeat. With a test period of a little under an hour, Link successfully rolled 22 strikes in a row, making bank. [Bertrand] has posted all of the instructions and source code so that everyone has an equal opportunity to swindle rupees out of poor Pondo.
HA! Brilliant. I have done a few similar things with Autohotkey on PC games. :) Much easier than having to code my own language, but just as rewarding.
One 2d space game required a ton of just collecting money. (the grind) So I setup 4 accounts, and put them all in the same location in space, and made a script that jumped between each instance of the game firing at spawning enemies and collecting loot.
For “educational purposes” I attempted to make an aim bot out of analyzing the image under the targeting reticule. CPU/GPU wasn’t up to the task… some day aimbots will be undetectable as they won’t need to hook into the actual memory system of the game… yikes.
Defeating grind systems really wakes you up to how pointless and unsatisfying many games are, but I had a blast coding around their stupid tricks, score!
“some day aimbots will be undetectable as they won’t need to hook into the actual memory system of the game”
This is already how a lot of bots work, they just analyze the frames. That’s why some publishers don’t allow the use of overlay software anymore (such as in Destiny 2).
In before game running in a special version of VirtualBox.
This! If you can’t tell you’re living in the Matrix, you can’t break out.
Or second PC with HDMI capture + HDMI splitter.
(Many Chinese splitters can accept HDCP input and strip HDCP in the process… So trying to protect against recording with HDCP fails here. As I understand it, many games enable HDCP for this reason.)
A robot that presses the buttons!
I wonder if there is a reason for the (relatively) long pauses between button presses. I’d be hammering A as fast as possible to get through those dialogs.
His language only issues one button press per prompt, and he doesn’t want to make it go too early and have the switch miss it. The reality is he could tweak the timing of that script and significantly speed up the iterations.
i just use python with dxgi capture to copy the framebuffer from the gpu into opencv and tensorflow.
for movement and actions i use a teensy in serial+mouse+touchscreen(for absolute coordinates)+keyboard+joystick configuration and control it via serial from python.
have yet to see a game detecting it. :-)