A recent post about debugging constructs surprised me. There were quite a few comments about how you didn’t need a debugger, as long as you had printf. For that matter, we’ve all debugged systems where you had nothing but an LED to flash or otherwise turn on to communicate with the user. However, it is hard to deny that a debugger can help with complex code.
To say you only need printf would be like saying you only need machine language. Technically accurate — you can do anything in machine language. But it sure makes things easier to have an assembler or some language to help you work out your problem. If you write a simple bash script, you can use the equivalent to printf — maybe that’s the echo command, although there is usually a printf command on a typical system, if you want to use it. However, there are other things you can do with bash including a pretty cool debugger if you know how to find it.
I assume you already know how to use echo and printf, but let’s dig into how to use trace execution line by line without the need for echo statements on every other line. Along the way, you’ll learn how to get started with the bash debugger.
Tracing
It isn’t always easy to read, but you can get a trace of the entire execution of a running script by adding the -x option to its invocation of bash. There are at least three ways to do this. First, you can do a set -x in your current session. The problem with this is that any fancy things you have happening in your shell prompt will show up, too. That makes the output even harder to read. If you do brave this method, you can turn the tracing back off at a shell prompt with set +x.
#!/bin/bash -x
for I in {1..10}
do
banner $((9-I)) - HAD
echo
echo
echo
done
You can add also add the flag when executing the script from the command line: bash -x ./had.sh or to the shebang on the first line of your script. Usually, you’ll have #!/bin/bash at the top of the file, just add the -x on that line.
For example, the following script has a few errors in it. I put the -x at the top, so if you run it, you’ll see what the trace looks like. Note the lines that start with a plus sign. That’s the line of the script that is executing. The other lines are actually output from the script.
Installing the Bash Debugger
There is a bash debugger available. However, Ubuntu and similar distributions haven’t included it in their software repos for awhile. You can, of course, go get it yourself. You’ll want to know what version of bash you are using (try bash --version). Download the version that matches what you have and follow the instructions.
The debugger itself is written in bash. It uses a special trap that fires at the end of each statement that is there especially for debugging. In theory, you could create your own debugger using that trap, but it is quite a bit of work.
Using the Debugger
There are at least two ways to invoke the debugger. You can run it along with the name of the script which is pretty safe: bashdb had.sh.
The only problem with this is that it can confuse $0 and the script itself shows up in the call tree. Running bash with the –debugging option will also run the program, or at least it should. It works for me, but anecdotally, some people have trouble with this method.
If you ask for help on the bashdb program, there are plenty of options, but none you’ll use often. Inside the program, you have commands similar to gdb commands. You can review them all by typing “help” at the prompt.
Asking for help in the bashdb program will show how to set breakpoints, look at backtraces or even have commands run when certain conditions are met. Quite a few of these commands worth further investigation:
- list – Shows the script with line numbers
- break – Set a breakpoint on a line (note: tbreak sets a one-time breakpoint)
- backtrace – View call stack
- condition – Sets up a conditional breakpoint
- display – Set up to print a variable before each stop
- handle – Set up trap handling
- step, next, skip – Single stepping commands
- trace, untrace – Like the -x argument
- watch, watche – Set up watches
The debugging process is a familiar one. Whether single-stepping through code, or setting break points to pause the program flow, this lets you track down problems with much more flexibility than echoing out information would.
There are two things wrong with my script. Well, at least two things. First, I meant for the banner to count down: 10, 9, 8, etc. I also wanted everything on one line. One of those is easy to find with the debugger.
If you want a video walkthrough, the one below is pretty old, but still good.
The best way to not have to debug shell scripts is to not put bugs in to start with. There are tools to help you strive for that. Then again, you could write your shell scripts in C.
One of my favorite things to do with one-liners is something like
for x in $(do this thing); do echo command -flag /path/path/ $x;done
then if it works the way I wanted, just pipe the output to bash:
for x in $(do this thing); do echo command -flag /path/path/ $x;done | bash
don’t even have to edit it. Just up arrow | bash and Enter. done.
I feel like a lot of the “who needs a debugger if you have (printf|an LED|GPIO & scope|a trace stream) comes from the large number of things which are timing dependent where there is a network protocol or server with a session timeout, or a system with a watchdog timer, or buffers of limited size, or another thread waiting on a lock, or a device which may burn out if you stop the clock with the PWM output held high, or the list goes on…
For such non-trivial systems whose functionality depends on interaction between the thread you’re examining and another (even if the other is remote over the network) or on interaction with real-world peripherals, the perturbation of time caused by even a scripted breakpoint action in a debugger but especially waiting for a slow human-debugger interaction may cause deviations in behavior/outcome that are hard to pick apart from the bug you’re chasing.
On the rare occasion when you have the luxury to stop a thread and play with it a debugger is great to have. When you don’t have that luxury, it is good to know how get the maximum leverage out of that debug log or spare GPIO pin.
The first two lines below the shebang in my shell scripts is always #set -x and #trap read debug Simple debugging then just becomes deleting the #’s.
The first enables debug output of course, the second single steps on enter.
You can also enable/disable these within the same script to skip over parts with set -x/set +x if the output is a bit too verbose.
The shebang in all my scripts reads: #!/usr/bin/perl
Using a language with proper syntax and built-in debugger, high level concepts like structures and data types built in, makes scriptwork much easy.
Script didn’t work? Rerun by hand using the “-d” option (perl -d script.pl), set breakpoints where you want, and print variables when you get there. Done and done.
Command output didn’t parse the way you thought it should? Print it and have a look. Reenter the command and try a new parse method *from the debugger prompt* until you get the right results. Cut and paste the parse back into the code when it’s right.
(Don’t like perl? Python is just as powerful, and I hear it has a debugger also.)
Unless I’m modifying an existing script (ex: rc.local) I never bother with shell syntax. Perl has an operator to execute a command and capture the output to a variable, so making a script that pieces together any system commands is a snap.
You can do this stuff in any just about any language, for example both java and node.js also have excellent support for streaming shell commands. For extra bonus points you can run on Windows, and you can leverage your existing language skills instead of poking around with an obscure old language whose syntax has often been compared to random noise.
+1 for using existing skills and not poking around in other languages.
(I take issue with “just about any language” though. C and C++ are not as effortless as more modern languages – but your point is valid. Use an expressive language that you know and your scripting productivity will skyrocket.)
Another perl guy here (also C/++, asm etc). I find it amusing that someone cuts down perl syntax while implicitly defending bash. As someone who came to bash later – it drives me nuts with the “sometimes you need a $igil, sometimes not” vs perl’s “think of the sigils as hungarian notation” and C’s just plain declarations. Sure, anyone can write horrible code, and certain pathological programmers really push things to the limits in the languages that allow some freedom – but that’s not the language, it’s the sinecure programmer looking for job security by obscurity. My own perl gets kudos for being easy to understand – freedom cuts more than one way. If you’re going to attack a language for being inconsistent, I give you that “fractal of bad design”…you know what I’m talking about.
In Bash, or any current-day POSIX shell, the dollar sign indicates a substitution. So, the rules are actually pretty simple: If something is a substitution, put a dollar sign infront of it, no matter if it’s a simple variable substitution, a substitution involving pattern expansion or (since backticks have, fortunately, been superseded by “$()”) a command substitution.
Also, the default for shell scripts is to ignore errors. -e quits when an error is encountered. Used in conjunction with -x has been enough for me to debug most shell scripts.
set -eu
Are my defaults in every script.
4th comment!!!!
There’s always a trade-off between the utility of a debugger, and the amount of time it takes to remember how to work the debugger. For a simple script, printf/echo might be quicker than the time it takes to set up and re-learn the debugger. Articles like this are great to have for reference, and move the needle on when a debugger makes sense.
> There’s always a trade-off between the utility of a debugger, and the amount of time it takes to remember how to work the debugger.
That is why my debuggers follow `gdb` commands. If you master `gdb`, which is reasonably complete, then to a large extent you use the same knowledge to cebug: bash, zsh, ksh, GNU make, perl, python, ruby, javascript, go (although my debugger their is rather old and in need of attention), in addition to the languages that GDB supports natively: C, C++, Ada, …
$ bash -x myscript.sh
I use it everyday.
Not a debugger, but very useful is shellcheck which does a great job of linting your bash for you with helpful error messages.
last line – “Then again, you could write your shell scripts in C” led me to look at the link from Sept as planned Al :-) –
So .. you can just use csh (know it’s not popular but my first exposure to unix was – Solaris / C shell) it’s available in the universe repo….
Why would anyone program in bash? If misery and foolishness is your game, go with the csh or tcl.
Friends don’t let friends write bash scripts. There are so many better choices.
Such as dash or ash!
Friends don’t let friends write in bash, or any non POSIX shell ;)
Sufficiently short / simple code needs no debugging. I only write really short snippets in Bash. (Although I write _many_!) So I’ve never needed a Bash debugger.
If your code is a two-liner, you’ve got a 50% chance of figuring out which line has the bug with your eyes closed. :)
They could also be both wrong. :P
Every two line bash script I wrote turned into a perl script (nowadays a ruby script), so I don’t bother even starting with bash anymore. Anything that isn’t a linear flow of captured commands belongs in another language. Life is too short to suffer with bash idiosyncracies. Yes there are rare cases (embedded linux things) where bash in the only option.
Why forget `-e`? it’s the flag I always use and add.
Also closing bash execution in functions with local variables helps a lot, splitting big shell scripts in multiple ones and testing… Well treating it as normal programming task really :)
Even BASIC had tron and troff.