One of the things that’s nice about Linux or Unix compared to many other operating systems is there’s a good chance a Linux program will spew out informational messages to a log somewhere. Many commands even have a way to turn on more logs. I know that Windows has the event viewer, but many programs don’t have much to say which makes it difficult to know what’s happening when things go wrong.
The problem is, sometimes programs tell you too much information. How do you find what you want to know? It looks cool on a movie where the hacker is in front of a terminal scrolling 500 lines a second of some log file, but in real life, it is hard to read a moving screen, although with some practice you can sometimes — unreliably — pick out a keyword as it whizzes by.
Like most Unix things, there’s a tool for that. In fact, unsurprisingly, there are many tools for that. If you are using the tail command, that’s certainly one of them. But there are others you should consider.
As an example, consider the mother of all logs: /var/log/syslog. Go dump that using cat or less (I always alias more to use less on my systems so if you catch me saying more, I really mean less). It is probably huge and growing. On a normal desktop system it is usually fairly quiet, but on some older systems or a server, you may see a lot of activity on it. But either way, unless you’ve just booted, you probably have pages of information in there.
Finding information that’s already there is easy enough. Use grep or load a copy into your favorite editor. The problem is the new information. Plug (or unplug) a USB device into your system. You’ll see that probably adds a message to your syslog file. If you are getting dozens of those kinds of messages all the time, what do you do?
It isn’t just syslog either. The listing file from a long compile might be interesting to peek in on. The status from having a RAID drive rebuilt. There’s always some big growing file you want to read.
The Tail End
The traditional way to do this is with the tail command. It takes a big file and returns “some” from the end of it. You can add the -f option to make the command wait for more data and output it. Great for a growing file. The -F option is much the same but will keep trying if it can’t open the file. You can control how many lines (-m) to show or bytes (-c). The -s option lets you pick how often it checks to see if the file changed.
Pretty good, right? Try this:
tail -f /var/log/syslog
You’ll see a few lines from the end of your log and now when you plug or unplug your USB device, you’ll see it nearly right away without having to reissue a command.
You probably have done this before. It works fine and is a very common idiom. But there are a few other ways to go that you might not be using.
Less is More
The less command has an option +F that turns it into a good replacement for tail. In fact, if you try it with the command line below you may wonder why it is any different than tail:
less +F /var/log/syslog
Here’s what that looks like on my system:
See how at the end it says “Waiting for data…”? Right now it is acting almost the same as tail. But if you press ^C something interesting happens. Well… maybe something happens. Try pressing ^C. If less goes to command mode, you are fine. Now you can do all the things you normally would do while viewing something with less. But if you exit less, then your Linux distribution is helping you by setting some default options on less using the LESS environment variable. Try this:
set | grep LESS
If you see an option string including --quit-on-intr that’s your problem. It needs to go. Then you can switch back to command mode using ^C. That does mean you need to remember to use the q command to exit less. If you exit this mode and want to get back just press F.
If you are in normal less mode (that is, you didn’t use +F to start with), you can press F to start this “tail mode.” Even more interesting, you can search for something, press ESC-F and the mode will stop with a bell when your search matches something coming in.
You can also add --follow-name to get the same behavior as tail‘s -F option.
Watching
Sometimes a file you want to see isn’t adding new data but is changing every so often. For example, consider /proc/loadavg or many other /proc entries. Using tail or less on these isn’t very satisfying. However, there’s the watch command:
watch -n 5 cat /proc/loadavg
This executes the cat command on the file every 5 seconds and shows the results nicely, as you can see. There are lots of good options such as -d to highlight differences or -p to use a high-resolution timer. The -c option makes things work with colors.
Your Editor
Your editor might have a tail mode. For emacs, there are several ways to do it and instead of telling you about them, I’ll point you to a great write up. I’m not a vim expert, but it looks like you need a plugin to do the same thing there. If there’s a better way, I’m sure we’ll hear about it in the comments.
If you are a hardcore log reader, you might want a tool like ,inavlnav that is specifically made to view logs. KDE and Gnome both have dedicated log viewers, as well.
Wrap Up
As usual in Linux or Unix, there are dozens of ways to do just about anything. Which one is “best?” Everyone will have their own opinion and that’s what makes it an attractive operating system for power users. You get to pick what works best for you.
If you are using Linux on the desktop, on a server, or on a Raspberry Pi, these are a few commands you’ll want to have in your toolbox. Be sure to take a look at all the Linux Fu posts (see below) to find out about other tools we’ve covered in the past.
Thanks, very useful…
User: I’d like to sip from the fountain of status knowledge.
Linux: Sure thing Buddy! *Turns on firehose*
I’m a big fan of ‘lnav’ to keep an eye on my logs in realtime. It has a lot of great functionalities. See: https://lnav.org/
Thanks for this one ! Indeed useful.
Yep it is in there towards the bottom ;-)
Can you fix the spelling, it’s ‘inav’ in the article, not ‘lnav’ ;).
Well it LOOKS like inav ;-) At least the link was right.
and no mention of the most useful command
tail -f | grep “search string” –line-buffered
TIL — that line-buffered really helps ;D Amount of times I’ve sat there waiting for a grep on a long file to complete.
However, i’m not sure you need it when using tail? I never have?
me: *look at that gif*
my subconscious: ctrl+c ctrl+c ctrl+c ctrl+c ctrl+c ctrl+c ctrl+c
Also don’t forget hexdump/hd/xxd for binary files and streams :)
I think there may be a hidden image in there. There are some pixels that don’t change but I can’t make it out.
It is sending hexadecimal data to your cerebral cortex through the optical nerve.
We will add your biological and technological distinctiveness to our own.
That tail -F is pretty slick. Thanks for the tip!
Have you tried bat? It’s cat (more like less) with syntax highlighting, line numbers, etc… It’s even supposed to have git integration although I don’t really know what that does yet.
https://github.com/sharkdp/bat
https://hackaday.com/2018/08/29/linux-fu-modernize-your-command-line/
> I don’t really know what that does yet
Add a “+” to lines that are new, “~” for modified lines, and “_” to indicate that lines below the current one have been removed. Basically a combination of cat (because it shows all lines, including non-modifed ones) and git diff.
I love my computer logging my every move. No, wait, I don’t.
Unless you hate your computer notifying you of errors, like failing hardware or connectivity, your comment doesn’t make very much sense.
In that case, I hope you don’t use microsoft and/or google either. Better yet, unless you have written your own OS, you might want to ditch the PC, the tablet, the cellphone, and maybe even your refrigerator these days…
systemd made getting at your logs a whole lot harder. I don’t understand how it improved Linux on the desktop in any way. I realize that ship has sailed, but most of the advice in this article is of limited use on a modern desktop Linux system.
Not Slackware! It may be the oldest existing distribution, but it’s still modern. I think the lack of systemd doesn’t make it “non-modern”.
Michael
Or Gentoo
Or Devuan
And maybe it works on BSD?
While I have huge stacks of Slackware CDs at home, I’m going to say that a distro that haven’t sent out an official release in 2 years and stuck at kernel-4.4 is not really in the running for being a modern distro.
How has that ship sailed? So the majority of distros have accepted a craptacular init replacement? It doesn’t have to be permanent! If it did then all distros would still be using sysvinit. Keep pressuring them. Keep switching your own installs to use a better init system. Provide bug reports if you find any and code to those better init projects if you are capable.
I use this stuff to search through source code files and dumps of data. Not every task is in a server admin role!
I sometimes need to debug why my USB JTAG or oscope won’t work. picking through system logs is a good start, if you can get to them.
There’s plenty of things that spew logs that aren’t under systemd.
If you squint at that GIF, you can see an ASCII version of “Never Gonna Give You Up”. #Probably
What is that from?
Um.. Rick Astley?
Regarding watch, also consider the -d and -c options.
-d will highlight differences between successive samples
-c will allow watch to pass ANSI escape characters through e.g. for setting colour, etc.
Top picture : Coincidentially, Karate Kid 1, 2 & 3 were shown on Aussie TV last weekend… Just saying.
oh, and all of these *nix commands and way more are also available for the Windows command line for those that didn’t know. Sure, not at appropriate within the Windows command-line environment, but they do all exist.
I don’t think they’re built-in. I think installing Visual Studio adds them.
Been around a hundred years before Visual Studio was invented. http://techgenix.com/windowsntadministrativetoolsetakaaintx/
Your tail section says it’s -m for a certain number of lines, it’s -n
For searching binary files, “strings” is super useful.
I’ve used https://vanheusden.com/multitail lately for long running tail -f with filtering.
inav looks interesting too. Correlation is always painful.
Once upon a time, I used swatch, a perl based tail -f that can parse the output and hide/show/run other programs. I would play sounds when things happened. Then I didn’t have to watch the logs. I’d listen and know if I had to react immediately or eventually.