Linux Fu: Keep an Eye on that File

One of the things that’s nice about Linux or Unix compared to many other operating systems is there’s a good chance a Linux program will spew out informational messages to a log somewhere. Many commands even have a way to turn on more logs. I know that Windows has the event viewer, but many programs don’t have much to say which makes it difficult to know what’s happening when things go wrong.

The problem is, sometimes programs tell you too much information. How do you find what you want to know? It looks cool on a movie where the hacker is in front of a terminal scrolling 500 lines a second of some log file, but in real life, it is hard to read a moving screen, although with some practice you can sometimes — unreliably — pick out a keyword as it whizzes by.

Like most Unix things, there’s a tool for that. In fact, unsurprisingly, there are many tools for that. If you are using the tail command, that’s certainly one of them. But there are others you should consider.

As an example, consider the mother of all logs: /var/log/syslog. Go dump that using cat or less (I always alias more to use less on my systems so if you catch me saying more, I really mean less). It is probably huge and growing. On a normal desktop system it is usually fairly quiet, but on some older systems or a server, you may see a lot of activity on it. But either way, unless you’ve just booted, you probably have pages of information in there.

Finding information that’s already there is easy enough. Use grep or load a copy into your favorite editor. The problem is the new information. Plug (or unplug) a USB device into your system. You’ll see that probably adds a message to your syslog file. If you are getting dozens of those kinds of messages all the time, what do you do?

It isn’t just syslog either. The listing file from a long compile might be interesting to peek in on. The status from having a RAID drive rebuilt. There’s always some big growing file you want to read.

The Tail End

The traditional way to do this is with the tail command. It takes a big file and returns “some” from the end of it. You can add the -f option to make the command wait for more data and output it. Great for a growing file. The -F option is much the same but will keep trying if it can’t open the file. You can control how many lines (-m) to show or bytes (-c). The -s option lets you pick how often it checks to see if the file changed.

Pretty good, right? Try this:

tail -f /var/log/syslog

You’ll see a few lines from the end of your log and now when you plug or unplug your USB device, you’ll see it nearly right away without having to reissue a command.

You probably have done this before. It works fine and is a very common idiom. But there are a few other ways to go that you might not be using.

Less is More

The less command has an option +F that turns it into a good replacement for tail. In fact, if you try it with the command line below you may wonder why it is any different than tail:

less +F /var/log/syslog

Here’s what that looks like on my system:

See how at the end it says “Waiting for data…”? Right now it is acting almost the same as tail. But if you press ^C something interesting happens. Well… maybe something happens. Try pressing ^C. If less goes to command mode, you are fine. Now you can do all the things you normally would do while viewing something with less. But if you exit less, then your Linux distribution is helping you by setting some default options on less using the LESS environment variable. Try this:

set | grep LESS

If you see an option string including --quit-on-intr that’s your problem. It needs to go. Then you can switch back to command mode using ^C. That does mean you need to remember to use the q command to exit less. If you exit this mode and want to get back just press F.

If you are in normal less mode (that is, you didn’t use +F to start with), you can press F to start this “tail mode.” Even more interesting, you can search for something, press ESC-F and the mode will stop with a bell when your search matches something coming in.

You can also add --follow-name to get the same behavior as tail‘s -F option.

Watching

Sometimes a file you want to see isn’t adding new data but is changing every so often. For example, consider /proc/loadavg or many other /proc entries. Using tail or less on these isn’t very satisfying. However, there’s the watch command:

watch -n 5 cat /proc/loadavg

This executes the cat command on the file every 5 seconds and shows the results nicely, as you can see. There are lots of good options such as -d to highlight differences or -p to use a high-resolution timer. The -c option makes things work with colors.

Your Editor

Your editor might have a tail mode. For emacs, there are several ways to do it and instead of telling you about them, I’ll point you to a great write up. I’m not a vim expert, but it looks like you need a plugin to do the same thing there. If there’s a better way, I’m sure we’ll hear about it in the comments.

If you are a hardcore log reader, you might want a tool like ,inavlnav that is specifically made to view logs. KDE and Gnome both have dedicated log viewers, as well.

Wrap Up

As usual in Linux or Unix, there are dozens of ways to do just about anything. Which one is “best?” Everyone will have their own opinion and that’s what makes it an attractive operating system for power users. You get to pick what works best for you.

If you are using Linux on the desktop, on a server, or on a Raspberry Pi, these are a few commands you’ll want to have in your toolbox. Be sure to take a look at all the Linux Fu posts (see below) to find out about other tools we’ve covered in the past.

38 thoughts on “Linux Fu: Keep an Eye on that File

    1. > I don’t really know what that does yet

      Add a “+” to lines that are new, “~” for modified lines, and “_” to indicate that lines below the current one have been removed. Basically a combination of cat (because it shows all lines, including non-modifed ones) and git diff.

    1. In that case, I hope you don’t use microsoft and/or google either. Better yet, unless you have written your own OS, you might want to ditch the PC, the tablet, the cellphone, and maybe even your refrigerator these days…

  1. systemd made getting at your logs a whole lot harder. I don’t understand how it improved Linux on the desktop in any way. I realize that ship has sailed, but most of the advice in this article is of limited use on a modern desktop Linux system.

      1. While I have huge stacks of Slackware CDs at home, I’m going to say that a distro that haven’t sent out an official release in 2 years and stuck at kernel-4.4 is not really in the running for being a modern distro.

    1. How has that ship sailed? So the majority of distros have accepted a craptacular init replacement? It doesn’t have to be permanent! If it did then all distros would still be using sysvinit. Keep pressuring them. Keep switching your own installs to use a better init system. Provide bug reports if you find any and code to those better init projects if you are capable.

  2. Regarding watch, also consider the -d and -c options.

    -d will highlight differences between successive samples
    -c will allow watch to pass ANSI escape characters through e.g. for setting colour, etc.

  3. oh, and all of these *nix commands and way more are also available for the Windows command line for those that didn’t know. Sure, not at appropriate within the Windows command-line environment, but they do all exist.

  4. I’ve used https://vanheusden.com/multitail lately for long running tail -f with filtering.
    inav looks interesting too. Correlation is always painful.

    Once upon a time, I used swatch, a perl based tail -f that can parse the output and hide/show/run other programs. I would play sounds when things happened. Then I didn’t have to watch the logs. I’d listen and know if I had to react immediately or eventually.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.