Linux Fu: Tracing System Calls

One of the nice things about Linux and similar operating systems is that you can investigate something to any level you wish. If a program has a problem you can decompile it, debug it, trace it, and — if necessary — even dig into the source code for the kernel and most of the libraries the program is probably using. However, the tools to do this aren’t ones you use every day. One very interesting tool is strace. Using it you can see what system calls any program makes and that can sometimes give you important clues about how the program works or, probably more often, why it doesn’t work.

Let’s consider the least complex use of the command. Suppose you want to make symlink from testxmit.grc to the /tmp directory. That command is simple:

ln -sf testxmit.grc /tmp

But if you tell strace to run it, the command becomes:

strace ln -sf testxmit.grc /tmp

You might want to redirect the output to a file using the shell or the -o option, though. Some commands generate a lot and often the first page or two of output isn’t really what you care about anyway.

Let’s Look

Let’s look at the output:

execve("/bin/ln", ["ln", "-sf", "testxmit.grc", "/tmp"], 0x7fff51ddf6f8 /* 91 vars */) = 0
brk(NULL) = 0x562301ce6000
... 
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=3004096, ...}) = 0
mmap(NULL, 3004096, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7c45298000
close(3) = 0
stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=1360, ...}) = 0
lstat("/tmp/testxmit.grc", 0x7fff7ae555d0) = -1 ENOENT (No such file or directory)
symlinkat("testxmit.grc", AT_FDCWD, "/tmp/testxmit.grc") = 0
lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
close(0) = 0
close(1) = 0
close(2) = 0
exit_group(0) = ?
+++ exited with 0 +++

At the top, shortened here, are 25 lines dealing with loading shared libraries into the memory space and four lines dealing with locales. I put the first “real” line in bold, where the program calls stat on /tmp and then makes sure the file doesn’t already exist. Finally, you get to the real system call (symlinkat) followed by a few things to close the program out. A lot of work to get to one system call.

You can probably figure out that the part on the right of the equal sign is the return value of the call. Usually, zero is success and other numbers mean different things. However, the openat call, for example, returns a file descriptor (3) and you can see it sent to fstat in the next line.

In Practice

Of course, the ln command works, but humor me and say we were wanting to understand what arguments passed to symlinkat. You could use the -e option to cut the output down to size:

strace -e symlinkat ln -sf testxmit.grc /tmp

You’ll notice something strange if you did the examples in order. The second time you run the command you get two calls to symlinkat. The first one fails because the file already exists. The second one is to some random file name. Taking off the -e lets you see everything (I’m only showing the interesting part):

symlinkat("testxmit.grc", AT_FDCWD, "/tmp/testxmit.grc") = -1 EEXIST (File exists)
openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 3
read(3, "\337\336\10\324\254\233", 6)   = 6
close(3)                                = 0
getpid()                                = 29340
getppid()                               = 29338
getuid()                                = 1000
getgid()                                = 1000
symlinkat("testxmit.grc", AT_FDCWD, "/tmp/CuzoNWnv") = 0
renameat(AT_FDCWD, "/tmp/CuzoNWnv", AT_FDCWD, "/tmp/testxmit.grc") = 0

Notice that the random part comes from reading some data from /dev/urandom. If you don’t want all that output, try:

strace -e synlinkat,renameat ln -sf testxmit.grc /tmp

Other Options

The -p option lets you supply a PID of a running program. Sending the output to a file and then monitoring the file with a tail -f is a good trick. By default, you only see 32 bytes of the call data and that might not be enough. You can adjust that size with the -s option.

So far we’ve only looked at simple programs. But if you want to trace multiple threads, check out the -f and -ff options.

If you want a survey of what the program is calling, the -c option will give you a summary.

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  0.00    0.000000           0         2           read
  0.00    0.000000           0         7           close
  0.00    0.000000           0         2           stat
  0.00    0.000000           0         3           fstat
  0.00    0.000000           0         1           lstat
  0.00    0.000000           0         1         1 lseek
  0.00    0.000000           0         6           mmap
  ...
  0.00    0.000000           0         4           openat
  0.00    0.000000           0         1           renameat
  0.00    0.000000           0         2         1 symlinkat
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000                    46         5 total

Of Course, There’s More…

This is just one of many tools you can use to examine Linux programs. Debuggers are often useful, especially if you have the source. There are other tools to examine symbol tables and dump executables. But those are topics for another time.

What’s your favorite Linux reverse engineering tool? Let us know in the comments.

11 thoughts on “Linux Fu: Tracing System Calls

  1. If you like strace for syscalls, be sure to check out ltrace for library calls as well. It’s often forgotten, but can be usefull. Unfortunately there is no tool to trace both, so i often have to run the trace using both tools to get usable info.

    1. For me, at least, ltrace sometimes fails in weird ways. The other programs that I’ve found that do very similar things, but fail less often, are sotruss and latrace.

    1. ta – was bout to mention dTrace if no-one else did … not sure how much traction it got under linux, but Apple did port it to OSX (with caveats – it was hobbled to not peek into DRM system calls)

  2. Do we need a refresher on Left / Right Al?
    “You can probably figure out that the part on the left of the equal sign is the return value of the call. …”
    Hold your hands in front of you, palms facing away, which hand’s index finger and thumb make an L shape from your perspective? That’s your left hand.
    The part on the RIGHT of the equals sign is the return. Yes, I know I’m being a pedantic grouch, but precision of language is important.

    1. So it is funny. I’ve spent a lot of time in front of a classroom or an audience. So I always mentally flip my right and left and then I usually catch it. The problem is when I catch it an odd number of times.

  3. Hmmpf, normally, There’s not much new in the linux-fu’s ut this time, i learned two new things!! Strace can trace running programs! (Should read that manual page …) And never had hearned about ltrace. You could have added a bit on ltrace after the break :(

  4. strace is great. BPF is a comparatively new tool that allows a root user incredible insight into the system. Tools like bcc or bpftrace allow you to instrument every aspect of a running Linux kernel, system calls, libraries. For example, this bpftrace one-liner will print out information about readline called through bash

    bpftrace -e ‘uretprobe:/bin/bash:readline { printf(“user input: \”%s\”\n”, str(retval)); }’

    From:
    https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md

  5. Protip: if you’re lazy or can’t recall the parameters to filter calls, or just want to hunt for a certain string, append 2>&1 to the command before piping through e.g. grep. By default, strace outputs to stderr, which is a nuisance when trying to parse the output. This redirect can be a nuisance as well, if you’re running the program via strace (merging program output with strace output), but is no issue if you’re attaching by PID (-p).

    strace -s 1000 (program) 2>&1 | grep something

    I find 1000 to be a reasonable string length, as it is usually sufficient to capture any complete path on a normal filesystem.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.