Linux Fu: Making Progress

The computer world looks different from behind a TeleType or other hardcopy terminal. Things that tend to annoy people about Unix or Linux these days were perfectly great when you were printing everything the computer said to you. Consider the brevity of most basic commands. When you copy a file, for example, it doesn’t really tell you much other than it returns you to the prompt when it is done. If you are on a modern computer working with normal-sized files locally, not a big deal. But if you are over a slow network or with huge files, it would be nice to have a progress bar. Sure, you could write your own version of copy, but wouldn’t it be nice to have some more generic options?

One Way

The pv program can do some of the things you want. It monitors data through a pipe or, at least through its standard output. Think of it as cat with a meter. Suppose you want to write a diskimage to /dev/sdz:

cat diskz.img >/dev/sdz

But you could also do:

pv diskz.img >/dev/sdz

By default, pv will show a progress bar, an elapsed time, an estimated end time, a rate, and a total number of bytes. You can turn any of that off or add things using command line options. You can also specify things like the size of the terminal if it should count lines instead of bytes, and, in the case where the program doesn’t know what it is reading, the expected size of the transfer.

Of course, the output of pv is to stderr so it shows up on the screen. This makes it tricky if you want to use it with something like dialog that wants the stdout to be the progress indication. Luckily, you can redirect stderr to stdout and then redirect stdout to a file. Just be sure to do it in that order. For example:

pv -n /tmp/bigfile.iso 2>&1 >/tmp/bigfile.iso.backup | dialog --gauge "Copy Progress" 10 60

Another Way

There is also progress. It looks around for programs running like cp, mv, dd, and many more, looks at their open files, and shows you progress information for those programs. It only does this once, so you’ll typically marry it with the watch command or use the -M option. You can also monitor arbitrary programs with a little help. For example, suppose you have a big download in firefox:

watch progress -c firefox

Actually, if you are like me and have more than 32 instances of firefox running, that’s going to fail, but you get the idea.

The progress program can also take a pipe, but it still only monitors processes it knows about unless you use -c. You can add to the list with the -a option. For example:

tar czf /tmp/books.tgz ~/library/ebooks | progress -m

Results in:

A long tar command

Note that since gzip is one of the programs that progress knows about, it shows up on the list, too. You could, of course, tell it to only look for tar with -c.

As Usual

As usual with Linux, there are always many ways to do anything. If you are a masochist, you could use ptrace to look inside a program and figure things out from there. You could also do the same trick that progress does and look through a program’s /proc entries.

If you want to add a progress bar to your shell scripts directly, try gum. Or maybe combine progress with widgets to put the result on your desktop?

12 thoughts on “Linux Fu: Making Progress

  1. How about a series on Windows Subsystem for Linux with articles on things like getting USB devices to work or gaining access to a DLNA server the windows host has no problem with, or how to fix crappy streaming performance in VLC when it’s fine in VLC for Windows?

    Since microsoft allowed WSL2 on Windows 10 without kludgy hacks I decided to give it a try. The above are some of the issues with it. “Solutions” I’ve found only cover the Linux part, or are for old builds of WSL and/or Windows and MS has of course *changed stuff*.

    Right now I just want to get my CanoScan LiDE 210 working with GIMP in WSL2 on Windows 10.

    1. Funny, you should mention that I’m actually flashing a 3D printer right now over USB with WSL and waiting for it to finish so I certainly could do a piece on that. I don’t know about the VLC issue though

  2. I’m surprised that pipeview isn’t a standard utility in Linux.
    I’m curious, though, how do you have more than 32 instances of Firefox? Do you mean tabs?
    I currently have 50 tabs open, but ps shows only 13 processes.
    I’ve only been doing this for 13 years so I’m still leaning Linux.

    1. I have multiple virtual desktops and each browser has way too many tabs. But I also have 12/24 processors and 64g of RAM so I am never motivated to do better lol

    2. 50 tabs! Whew. I have two open right now… one to many actually as one is just a search result. Delete that one. Now down to one.

      Never, ever even thought about a progress bar for general utilities like cp. Seems like there is one for utilities that actually need it like the apt suite. Looks like there is a way if someone really wants a progress update :) .

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.