As a debugger, GDB is a veritable Swiss Army knife. And just like exploring all of the non-obvious uses of a those knives, your initial response to the scope of GDB’s feature set is likely to be one of bewilderment, subsequent confusion, and occasional laughter. This is an understandable reaction in the case of the Swiss Army knife as one is unlikely to be in the midst of an army campaign or trapped in the wilderness. Similarly, it takes a tricky debugging session to really learn to appreciate GDB’s feature set.

If you have already used GDB to debug some code, it was likely wrapped in the comfort blanket of an IDE. This is of course one way to use GDB, but limits the available features to what the IDE exposes. Fortunately, the command line interface (CLI) of GDB has no such limitations. Learning the CLI GDB commands also has the advantage that one can perform that critical remote debug session even in the field via an SSH session over the 9600 baud satellite modem inside your Swiss Army knife, Cyber Edition.

Have I carried this analogy too far? Probably. But learning the full potential of GDB is well worth your time so today, let’s dive in to sharpen our digital toolsets.

Godmode for Running Code

The concept behind a debugger is fairly uncomplicated: all too often something is preventing code which you have written from working, or you want to take a closer look at certain states within the application as it is executing. A simple way to do this is by printing out the values of variables to a terminal or serial port, but the much more powerful way is to use a debugger like GDB to interactively work with the code as it executes.

This means pausing the execution, stepping forward and backward through individual lines of code, inspecting stack frames and parts of memory, modifying the contents of memory and specific variables, and so on. Along with the setting of breakpoints and the watching of variables, virtually any part of the application’s execution can be monitored and influenced. This includes when things go south and the execution terminates with some fault condition, allowing a stack trace to be recalled.

While GDB can be used with any application’s binary, it’s infinitely more useful when the debug symbols are also provided to the debugger. These debug symbols are text strings including source code, as well as other information. They are included in the binary by the compiler when instructed to do so. For GCC-based compilers and LLVM this is usually done using the -g flag.

Local Hero

Running through a local debug session is a good way to become acquainted with how to use GDB’s command line interface. Be sure to have a handy command reference within easy reach at any time when using GDB. Doing so will make it easy to become familiar with the more involved commands.

The essential ones to know are break (b), for setting a break point, info for obtaining information locals, threads, etc. Further backtrace (bt) and continue (c), next (n) and step (s) for printing out a backtrace, continuing execution and moving through the code in increments, respectively. After loading the executable with GDB, the program is started with run (r), which can be supplied with any command line arguments to the executable.

Let’s write a very simple program we can use for debugging practice:

/* hello.c - Hello World */

#include<stdio.h>

int main(void) {
        char hello[] = "Hello World";
        printf("%s\n", hello);
        return 0;
}

We’ll use the -g3 flag when compiling to include debug symbols. Now let’s walk through this C-based Hello World example:

gcc -o hello -g3 hello.c
gdb ./hello

[..]
(gdb) b main
Breakpoint 1 at 0x1169: file hello.c, line 5.
(gdb) run
Starting program: /home/hackaday/hello

Breakpoint 1, main() at hello.c:5
5 int main(void) {
(gdb) n
6 char hello[] = "Hello World";
(gdb) n
7 printf("%s\n", hello);
(gdb)

We first set a break point at the main()function, then run to start the program. After the breakpoint, with each execution of next (or just hitting enter on an empty input to repeat the previous command), we’ll proceed to the next line of the application, without stepping into function calls. That’s what step is for. If we use printf() in the code, for example, using step would cause us to examine every line of that function and its implementation as well. Whether this is desirable depends on one’s needs.

Finally, we can examine variables and memory using print (p) for printing variables and x to print bytes at a memory address. E.g.:

(gdb) print hello
$1 = "Hello World"

Most of the commands are quite straight-forward, and safe. Barring the use of GDB’s set command, using which one can not only change GDB’s settings, but also edit memory contents. Use this one with caution.

Jacking Into the Remote

Running a remote GDB session is roughly the same as a local session, with the obvious complication of having to establish the session on a remote system. Said remote system can be anything from a server, desktop or other system running a full-blown OS, down to a microcontroller (MCU) running straight on the bare metal.

The main requirement for GDB to establish a debugging session on a remote system, is for there to be a GDB server (gdbserver) instance which the GDB tool can connect to. This GDB server then acts as a bridge between GDB and the active debug session. This connection can be established via TCP or a serial line. This makes it a highly portable approach that works both for remote servers or desktop systems, as well as industrial boards with an RS-232C link.

Even more interesting is to use the GDB server approach to create a bridge to the in-circuit debugger functionality provided by microcontroller platforms such as ST’s STM32 Cortex-M-based systems. This same approach will work with few modifications for Microchip’s ARM-based SAM and AVR platforms.

OpenOCD as GDB server

Anyone who has done MCU development is likely familiar with OpenOCD. This tool is invaluable in the programming of a wide variety of MCUs, but also comes with a built-in GDB server. As an example, imagine wanting to establish a GDB session on an STM32 MCU, on a common development board like the STM32F4-Discovery one.

The first step is to start OpenOCD’s GDB server:

openocd -f board/stm32f4discovery.cfg

Next, we can connect to this server via the loopback interface, while also providing GDB (from the arm-none-eabi toolchain) with the path to the ELF binary containing the firmware:

arm-none-eabi-gdb --eval-command="target remote localhost:3333" "hello_world.elf"

GDB will now connect to the GDB server, with OpenOCD using the STM32F4-Discovery board’s in-circuit debugger feature of the onboard ST-Link/V2 interface. All protocol translation is now done by OpenOCD, enabling all the usual GDB features, even though the code we are debugging runs on the MCU on the development board.

As the MCU will already have booted the firmware we wish to debug, we will still have to perform one more step, which is to reset the MCU to get a GDB session we can use:

(gdb) mon reset halt

The MCU will now have been reset and in a halted state until we do something. We will now add a new temporary breakpoint and continue:

(gdb) tbreak main
(gdb) c

After continuing execution, this temporary breakpoint puts us right at the beginning of our main function, from which we can set up breakpoints and more as needed. For example, we can check out the value of a specific register of the GPIOA peripheral on this STM32F4-based board. Say we want to see whether the input and output states were set properly in the GPIO_MODER register:

(gdb) x/4tb 0x40020000
$1 = 0100 0000 0000 0000

The special syntax of the x command prints a single 32-bit address, as blocks of single bytes. The GPIOA peripheral location is found in the STM32F407 datasheet, with the Reference Manual (RM) listing the offsets for specific registers within the memory-mapped IO for that peripheral type. In this case the MODER register is at offset 0x00, with GPIOA at address 0x40020000. The byte order is printed left to right, meaning that the first byte is on the left side.

In this case we can see that MODER1 (for pin 1) is set to ’01’, meaning general-purpose output mode.

Time to Quit Guessing

Many are the times when I found myself or others get stuck pouring over lines of code, speculating which one of those lines might be the cause of the weird symptoms. Suffice it to say that doing so is neither fun nor productive. Along with tools like Valgrind, debuggers like GDB are perfect for getting answers to questions, even questions you didn’t know you wanted to ask. It’s especially useful with something like embedded development, where the immediate feedback from newly flashed firmware might be… absent or not quite as expected.

It pays to establish a strict routine of testing for isolating test cases, and to hit the problematic firmware in-situ with a targeted testing plan, using tools like GDB. Create a checklist of which items to check first when something doesn’t work, then work your way up from there.

As non-deterministic as debugging sometimes may seem — and with Heisenbugs certainly endeavoring to make it appear that way — in the end there’s a good, solid reason for every issue. You just need to find out what bit to look at in which manner. Becoming comfortable with a powerful tool like GDB is definitely a major asset there.