Linux Fu: Debugging Bash Scripts

December 11, 2019 by Al Williams 23 Comments

A recent post about debugging constructs surprised me. There were quite a few comments about how you didn’t need a debugger, as long as you had printf. For that matter, we’ve all debugged systems where you had nothing but an LED to flash or otherwise turn on to communicate with the user. However, it is hard to deny that a debugger can help with complex code.

To say you only need printf would be like saying you only need machine language. Technically accurate — you can do anything in machine language. But it sure makes things easier to have an assembler or some language to help you work out your problem. If you write a simple bash script, you can use the equivalent to printf — maybe that’s the echo command, although there is usually a printf command on a typical system, if you want to use it. However, there are other things you can do with bash including a pretty cool debugger if you know how to find it.

I assume you already know how to use echo and printf, but let’s dig into how to use trace execution line by line without the need for echo statements on every other line. Along the way, you’ll learn how to get started with the bash debugger.

Continue reading “Linux Fu: Debugging Bash Scripts” →

Finding USB Bugs The Hard Way

October 20, 2019 by Sharon Lin 17 Comments

Sometimes debugging just doesn’t go the way you want it to. When USB problems arise, you can usually use a protocol analyzer to find the issue causing trouble. For [Paul Stoffregen], it was only the first step in a long process to find the culprit.

Procotol Analyzer

The complaint that came up was from a customer whose 2 port USB hub wasn’t working on their Teensy 3.6. The hub had been tested on Linux, Mac, and Windows, so it made sense to test what was different about the Teensy. Furthermore, all other USB hubs worked on the Teensy. As it turns out, these weren’t the most helpful assumptions to make when finding the bug.

Any protocol analyzer can be used, for instance the Beagle480. The way it works is by passing through USB communication, making a copy of the communication coming in and out, and sending it to the PC.

Normally, the analyzer has a small buffer memory and must sustain fast data flow. Unfortunately, this can occasionally cause software lockup. From what could be gathered from the verbose printing, USB descriptors were found for the hub. As it turns out, the faulty hub was a Multi-TT type hub, while most others are single TT (transaction translator).

Fixing Software Lockup

Since it was necessary to get the rest of the descriptor data, fixing the software lockup was the next step. Writing in a panic function – a breakpoint of sorts – into the code allowed the USB host’s power to terminate, and stepping through the program revealed that while the 2 port hub was initially being read, some issue arose afterwards.

As it turns out, the issue relied on USB split transactions, used only between USB hosts and hubs. Communication happens by tokens, which begins with a SPLIT-START token.

As it turns out, the issue was that the tokens weren’t being sent in the correct order. The other hubs seemed to be handle this nevertheless. By applying a fix to the C++ code of the bad hub, which had previously not been implementing the data structure for accessing register properly, the hub was able to work again.The hub appeared to be rejecting bad token, which was causing the issue in the first place.

All in all, while I’m sure this had to be a head scratching experience, at least it gives us some insight into the low-level design of USB communication.

Supercon: Ruth Grace Wong And Firmware From The Firehose

February 22, 2019 by Elliot Williams 13 Comments

Firmware and software are both just code, right? How different could the code that runs Internet-scale distributed web stuff be from the code that runs a tiny microcontroller brain inside a personal hydroponics device? Night and day!

Ruth Grace Wong works in the former world, but moonlights as a manufacturing engineer with some friends. Their product had pre-existing firmware that contained (at least) one bug, and Ruth’s job was to find it. The code in question was written by the Chinese PCB engineer, who knew the electronics intimately but who had no software background, providing Ruth an opportunity to jump head-first into the rawest of raw embedded programming. Spoiler alert: she found the bug and learned a lot about firmware along the way. This talk follows her along the adventure.

“The code is very well documented, in Chinese” but the variable names are insanely non-descriptive. Similarly, while the PCB engineer knows full well what a 24C02 is, if you’re a software geek that might as well be Chinese. As you’d expect, web searches came to the rescue on both fronts.

The bug ended up hiding in a logical flaw in the PWM-setting code inside an interrupt service routine, and it kept the fan from ever coming full on. Once found, it was easily fixed. But getting to the point where you understand the codebase deeply enough to know where to look is four-fifths of the battle. Heck, setting up the toolchain alone can take a day or two.

If you’re a fellow software type, Ruth’s talk (embedded below) will give you a quick glimpse into the outer few layers of the onion that is embedded firmware development, from a familiar viewpoint. Give her quick and value-packed talk a watch! Grizzled hardware veterans will nod along, and maybe even gain a little insight into how our code looks to “them”.

Continue reading “Supercon: Ruth Grace Wong And Firmware From The Firehose” →

Crash Your Code – Lessons Learned From Debugging Things That Should Never Happen™

January 22, 2019 by Sven Gregori 58 Comments

Let’s be honest, no one likes to see their program crash. It’s a clear sign that something is wrong with our code, and that’s a truth we don’t like to see. We try our best to avoid such a situation, and we’ve seen how compiler warnings and other static code analysis tools can help us to detect and prevent possible flaws in our code, which could otherwise lead to its demise. But what if I told you that crashing your program is actually a great way to improve its overall quality? Now, this obviously sounds a bit counterintuitive, after all we are talking about preventing our code from misbehaving, so why would we want to purposely break it?

Wandering around in an environment of ones and zeroes makes it easy to forget that reality is usually a lot less black and white. Yes, a program crash is bad — it hurts the ego, makes us look bad, and most of all, it is simply annoying. But is it really the worst that could happen? What if, say, some bad pointer handling doesn’t cause an instant segmentation fault, but instead happily introduces some garbage data to the system, widely opening the gates to virtually any outcome imaginable, from minor glitches to severe security vulnerabilities. Is this really a better option? And it doesn’t have to be pointers, or anything of C’s shortcomings in particular, we can end up with invalid data and unforeseen scenarios in virtually any language.

It doesn’t matter how often we hear that every piece of software is too complex to ever fully understand it, or how everything that can go wrong will go wrong. We are fully aware of all the wisdom and cliches, and completely ignore them or weasel our way out of it every time we put a /* this should never happen */ comment in our code.

So today, we are going to look into our options to deal with such unanticipated situations, how we can utilize a deliberate crash to improve our code in the future, and why the average error message is mostly useless.

Continue reading “Crash Your Code – Lessons Learned From Debugging Things That Should Never Happen™” →

“DB” = Abbreviated Microcontroller Debugging

December 20, 2018 by Drew Risinger 19 Comments

We’ve all been there. When debugging a microcontroller project, we just want to put in a print statement to figure out what’s going on with the microcontroller in real time. However, advanced embedded programmers know that printf statements are verboten: they’re just too SLOW. While not fixing this plight entirely, [Atakan Sarioglu] has come up with a clever way to create readable debug messages with minimal runtime overhead.

[Atakan Sarioglu]’s innovation, called BigBug (Github), is a dynamically-generated codebook. The codebook translates abbreviated messages sent over serial (UART here) to longer-form human-readable messages. To generate the codebook, BigBug automatically parses your comments to create a lookup between an abbreviation and the long-form message. When you are running your program on the microcontroller, BigBug will translate the short codes to long messages in real-time as you send log/debug data over serial. Continue reading ““DB” = Abbreviated Microcontroller Debugging” →

Debugging Arduino Is Painful: This Can Help

November 7, 2018 by Al Williams 27 Comments

If you are used to coding with almost any modern tool except the Arduino IDE, you are probably accustomed to having on-chip debugging. Sometimes having that visibility inside the code makes all the difference for squashing bugs. But for the Arduino, most of us resort to just printing print statements in our code to observe behavior. When the code works, we take the print statements out. [JoaoLopesF] wanted something better. So he created an Arduino library and a desktop application that lets you have a little better window into your program’s execution.

To be honest, it isn’t really a debugger in the way you normally think of it. But it does offer several nice features. The most rudimentary is to provide levels of messaging so you can filter out messages you don’t care about. This is sort of like a server’s log severity system. Some messages are warnings and some are informational, and some are verbose. You can select what messages to see.

In addition, the library timestamps the messages so you can tell how much time elapsed between messages and what function you were in during the message. It can also examine and set global variables that you preconfigure and set watches on variables. It is also possible to call functions from the serial monitor.

Continue reading “Debugging Arduino Is Painful: This Can Help” →

Simulate PIC And Arduino/AVR Designs With No Cloud

August 22, 2018 by Al Williams 17 Comments

I’ve always appreciated simulation tools. Sure, there’s no substitute for actually building a circuit but it sure is handy if you can fix a lot of easy problems before you start soldering and making PCBs. I’ve done quite a few posts on LTSpice and I’m also a big fan of the Falstad simulator in the browser. However, both of those don’t do a lot for you if a microcontroller is a major part of your design. I recently found an open source project called Simulide that has a few issues but does a credible job of mixed simulation. It allows you to simulate analog circuits, LCDs, stepper and servo motors and can include programmable PIC or AVR (including Arduino) processors in your simulation.

The software is available for Windows or Linux and the AVR/Arduino emulation is built in. For the PIC on Linux, you need an external software simulator that you can easily install. This is provided with the Windows version. You can see one of several videos available about an older release of the tool below. There is also a window that can compile your Arduino code and even debug it, although that almost always crashed for me after a few minutes of working. As you can see in the image above, though, it is capable of running some pretty serious Arduino code as long as you aren’t debugging.

Continue reading “Simulate PIC And Arduino/AVR Designs With No Cloud” →