Some hacker events are muddy and dusty affairs in distant fields, others take place in darkened halls, but I went to one that can be experienced as a luxury break in a European city steeped in culture and history. Newline takes place at Hackerspace Gent, in the Belgian city of that name, and I was there last weekend to catch the atmosphere as well as the programme of talks and workshops. And of those a good start was made by [PoroCYon], whose fascinating introduction to the glitching techniques involved in recovering the boot ROMs from a Nintendo DSi taught us plenty of things we hadn’t seen before.
The talk which you’ll find below the break starts by describing the process of glitching — using power supply interference to interrupt the operation of a microprocessor and avoid certain instructions — to bypass security code. It then moves on to some of the protection mechanisms used in the various generations of Nintendo consoles and handhelds, before moving on to the work on the DSi at which point the talk moved onto a field which may be old hat in glitching circles but was new to me; that of EM glitching.
EM glitching involves using a small coil to generate precisely timed electromagnetic pulses which induce the glitch voltages in the chip. The fascinating part is that the EM probe can be made small enough to target individual areas of the chip, so using it involves a brute-force technique trying all combinations of timing and position with the probe held in a computer-controlled X-Y mount.
The DSi has two processors on board, this achieves success with the ARM7 but leaves its companion ARM9 as yet untapped. There are a promising set of attack vectors left to try, of which the ARM7 placing the ARM9 into a state from which it can be glitched seems to be the most promising. It’s fairly obvious that there’s plenty more to come from this quarter.
Reverse engineering or modifying a device often requires you to access the firmware stored on a microcontroller. Since companies are usually not fond of people who try to peek into their proprietary data, most commercial devices are readout protected. [rumpeltux] ran into this problem when he tried to dump the firmware on an HC-12 wireless serial communication module for yet undisclosed reasons. Hacking into the device was a challenge that he gladly accepted and in the end, he succeeded by building a low-cost setup for voltage glitching.
Voltage glitching is a form of fault injection that has, e.g., been successfully used to hack the Playstation Vita. It involves the injection of voltage spikes on the power line in order to force the bootloader to skip security checks. The hard thing is trying to find the right shape of the waveform and the best way to inject the signal.
While there are already open-source boards for fault injection like ChipWhisperer, [rumpeltux] chose to build his own setup around an FPGA. By using a cheap EPM240 board, some MOSFET, and a USB-to-Serial converter, the total costs of the glitching setup were under 20 Euros. [rumpeltux] then recorded a larger number of voltage traces on the VCC pin around the reset phase and analyzed the differences. This helped him to pinpoint the best time for injecting the signal and refine the search space. After some unsuccessful attempts to glitch the VCC and GND pins, he got lucky when using one of the voltage regulator pins instead.
Most readers will be familiar with the ESP32, Espressif’s dual-core processor with integrated WiFi and Bluetooth. Few of us though will have explored all of its features, including its built-in encryption facilities and secure booting capability. With these, a developer can protect and secure their code, and keep their devices secure.
That sense of security may now be illusory though, thanks to [LimitedResults] who has developed a series of attacks on the chip that compromise its crypto core, secure boot, and flash encryption. This enables both the chance of arbitrary code execution and firmware extraction on locked-down ESP32 devices.
To achieve all this he used a glitching technique on the device’s power supply, inserting a carefully timed glitch in the rail to coincide with a particular instruction being executed. For those of us who are not experts in this technique, he provides a basic primer with a description of his home-made glitcher made using a CMOS switch chip.
It appears that there is no solution to this attack short of new silicon, however, it should be borne in mind that it’s something that depends upon a specialist hacker with a well-equipped bench, and is thus only likely to be a significant headache to manufacturers. But it undermines a key feature of a major line of microcontrollers, and as such it remains a significant piece of work.
CDs were a great advancement in audio quality when they were first put on the market. There’s no vinyl-style degradation of the medium if it’s played over and over, and there’s no risk of turning them into a giant pile of ribbon while rewinding like a cassette tape. The one downside was that if you were to take them on the move you needed special hardware and software to prevent the inevitable skipping. If you look at the skipping not as a downside, though, but as a way to produce interesting music, you might end up with a pretty unique piece of hardware.
[Dmitry] is known for his interesting art installations, and the latest one uses parts from three 1988 Sony D2 CD players that have been reassembled in order to take advantage of a skipping and glitching CD. The modified equipment is able to play during pause or rewind thanks to a processor modification, and can also change the rotational speed of the disc. There are other pieces of hardware included for more fine control of glitching and skipping of the audio being read off of the CD.
A factory is a machine. It takes a fixed set of inputs – circuit boards, plastic enclosures, optimism – and produces a fixed set of outputs in the form of assembled products. Sometimes it is comprised of real machines (see any recent video of a Tesla assembly line) but more often it’s a mixture of mechanical machines and meaty humans working together. Regardless of the exact balance the factory machine is conceived of by a production engineer and goes through the same design, iteration, polish cycle that the rest of the product does (in this sense product development is somewhat fractal). Last year [Michael Ossmann] had a surprise production problem which is both a chilling tale of a nasty hardware bug and a great reminder of how fragile manufacturing can be. It’s a natural fit for this year’s theme of going to production.
The saga begins with [Michael] receiving an urgent message from the factory that an existing product which had been in production for years was failing at such a high rate that they had stopped the production line. There are few worse notes to get from a factory! The issue was apparently “failure to program” and Great Scott Gadgets immediately requested samples from their manufacturer to debug. What follows is a carefully described and very educational debug session from hell, involving reverse engineering ROMs, probing errant voltage rails, and large sample sizes. [Michael] doesn’t give us a sense for how long it took to isolate but given how minute the root cause was we’d bet that it was a long, long time.
The post stands alone as an exemplar for debugging nasty hardware glitches, but we’d like to call attention to the second root cause buried near the end of the post. What stopped the manufacturer wasn’t the hardware problem so much as a process issue which had been exposed. It turned out the bug had always been reproducible in about 3% of units but the factory had never mentioned it. Why? We’d suspect that [Michael]’s guess is correct. The operators who happened to perform the failing step had discovered a workaround years ago and transparently smoothed the failure over. Then there was a staff change and the new operator started flagging the failure instead of fixing it. Arguably this is what should have been happening the entire time, but in this one tiny corner of the process the manufacturing process had been slightly deviated from. For a little more color check out episode #440.2 of the Amp Hour to hear [Chris Gammell] talk about it with [Michael]. It’s a good reminder that a product is only as reliable as the process that builds it, and that process isn’t always as reliable as it seems.
[Micah Elizabeth Scott], aka [scanlime], has been playing around with USB drawing tablets, and got to the point that she wanted with the firmware — to reverse engineer, see what’s going on, and who knows what else. Wacom didn’t design the devices to be user-updateable, so there aren’t copies of the ROMs floating around the web, and the tablet’s microcontroller seems to be locked down to boot.
With the easy avenues turning up dead ends, that means building some custom hardware to get it done and making a very detailed video documenting the project (embedded below). If you’re interested in chip power glitching attacks, and if you don’t suffer from short attention span, watch it, it’s a phenomenal introduction.