Supercon: Ruth Grace Wong and Firmware From the Firehose

Firmware and software are both just code, right? How different could the code that runs Internet-scale distributed web stuff be from the code that runs a tiny microcontroller brain inside a personal hydroponics device? Night and day!

Ruth Grace Wong works in the former world, but moonlights as a manufacturing engineer with some friends. Their product had pre-existing firmware that contained (at least) one bug, and Ruth’s job was to find it. The code in question was written by the Chinese PCB engineer, who knew the electronics intimately but who had no software background, providing Ruth an opportunity to jump head-first into the rawest of raw embedded programming. Spoiler alert: she found the bug and learned a lot about firmware along the way. This talk follows her along the adventure.

“The code is very well documented, in Chinese” but the variable names are insanely non-descriptive. Similarly, while the PCB engineer knows full well what a 24C02 is, if you’re a software geek that might as well be Chinese. As you’d expect, web searches came to the rescue on both fronts.

The bug ended up hiding in a logical flaw in the PWM-setting code inside an interrupt service routine, and it kept the fan from ever coming full on. Once found, it was easily fixed. But getting to the point where you understand the codebase deeply enough to know where to look is four-fifths of the battle. Heck, setting up the toolchain alone can take a day or two.

If you’re a fellow software type, Ruth’s talk (embedded below) will give you a quick glimpse into the outer few layers of the onion that is embedded firmware development, from a familiar viewpoint. Give her quick and value-packed talk a watch! Grizzled hardware veterans will nod along, and maybe even gain a little insight into how our code looks to “them”.

13 thoughts on “Supercon: Ruth Grace Wong and Firmware From the Firehose

  1. “Once found, it was easily fixed. But getting to the point where you understand the codebase deeply enough to know where to look is four-fifths of the battle”

    I thought you were trying to make the point that firmware development is different from other forms of software development!

  2. “Similarly, while the PCB engineer knows full well what a 24C02 is, if you’re a software geek that might as well be Chinese.”

    What!? If you are writing firmware for a device with a 24C02, you’re going to be intimately familiar with the part and how it works. I think it is far more likely that a PCB Engineer won’t know what a 24C02 is rather than the Engineer writing the code lives in it.

  3. I write plenty of firmware, I I don’t buy the notion that firmware is wildly different at all. Firmware is just software of a particular kind, and it can be well written or horribly written just like anything else. Maybe bad coding practices are more common with people writing firmware, but I don’t think this is necessarily so.

      1. You’re joking. It’s really hard to get embedded places filled. ‘devops’ is what everyone and their dog seems to want to do. Skilled firmware engineers are worth their weight in gold.

  4. Good presentation. Just one nit:
    Flash memory isn’t persistent? Wrong.
    EEPROM is byte addressable and persistent through power cycles.
    Flash is also persistent, but in NAND flash (most common type), a whole block must be written, instead of just one byte.

    1. Erk.. not the only thing wrong with that presentation! Anyone who gets the basics so utterly wrong is a major red flag, they obviously are not an embedded developer. I really hope that people are not learning their craft from dodgy internet videos like this.

      However, firmware is similar to software in that there are pretty clueless people working in both fields who manage to
      still get paid to do it, partly because there is a shortage of good people but mainly because managers know even less about software/firmware than the developers. I had one who literally did not know the difference between RAM and ROM.

    2. > Flash is also persistent, but in NAND flash (most common type), a whole block must be written, instead of just one byte
      No. A block must be ERASED in one go. It can often be written a byte at a time.

      That may depend a bit on the device that the flash is in. If you have an SD card or EMMC, the interface that YOU get does not give you the ability to write a byte at a time. But for both AVR and STM32 I know for a fact that once erased you get to write the data in very small blocks. (on STM32 at your choice byte, half-word or word, less choice if the power supply is “low”).

  5. Hmm… Kind of a weird mix in this presentation. I’m not really sure who the intended audience was.

    At first it seems like it’s intended for absolute beginners who don’t know what EEPROMs, flash memory, PWM or main loops are, but then whizzes through ISRs, modulo arithmetic, and timers. (I’m still trying to puzzle out why (65536 – 60)/256 was used for TH1 for example).

    And I think the lesson / trope about hardware engineers being bad programmers wasn’t really borne out by the actual bug. Yeah, it was a silly bug, but it wasn’t some bug that was a result of horrendously written code, but probably an overlooked copy-paste.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.