Linux-Fu: Running Commands

One of the things that makes Linux and Unix-like systems both powerful and frustrating is that there are many ways to accomplish any particular goal. Take something simple like running a bunch of commands in sequence as an example. The obvious way is to write a shell script which offers a tremendous amount of flexibility. But what if you just want some set of commands to run? It sounds simple, but there are a lot of ways to issue a sequence of commands ranging from just typing them in, to scheduling them, to monitoring them the way a mainframe computer might monitor batch jobs.

Let’s jump in and take a look at a few ways you can execute sequences from bash (and many other Linux shells). This is cover the cron and at commands along with a batch processing system called task spooler. Like most things in Linux, this isn’t even close to a complete list, but it should give you some ideas on ways to control sequences of execution.

From the Shell

The easiest but perhaps least satisfying way to run a bunch of commands is right from the bash shell. Just put semicolons between the commands:

date ; df ; free

This works for many shells that look more or less like bash. For a simple set of commands like that, this is fine. Each one runs in sequence. However, what if you had this:

Bad example of bash code. We've made this an image to prevent copy/paste.

You want to erase all the files in foo (but not subdirectories; you’d need -r for that). But what if the cd fails? Then you’ll erase all the files in whatever directory you happen to be in. That’s a bad idea and violates the law of least astonishment.

The solution is to use the && operator. Just like in C, this is an AND operator. Nearly all Linux programs return a 0 error status for true and anything else is false. That allows programmers to return an error code for failure and it becomes a false. So consider this:

cd /foo && ls   # not rm so we don't do anything bad by mistake

If /foo exists, the cd command will return 0 which is true. That means the result of the AND could be true and therefore execution continues with the ls. However, if the cd fails, the result will be false. If any input to an AND function is false, the other values don’t matter. Therefore, as soon as anything returns false, the whole thing stops. So if there is no /foo, the ls command won’t execute at all.

You can use more than one set of these operators like:

Bad example of bash code. We've made this an image to prevent copy/paste.

There’s also an OR operator (||) that quits as soon as anything returns true. For example:

grep "alw" /etc/passwd || echo No such user

Try your user ID instead of alw and then try one that isn’t good (surely you don’t have a user named alw). If the grep succeeds, the echo doesn’t happen. You can even mix the operators if you like. If you have a lot of commands that take a long time to run, this may not be the best answer. In that case, look at Spooling, below.

You probably know that you can push a running command into the background by pressing Control+Z. From there you can manipulate it with fg (move to foreground), bg (run in background), and kill commands. All of those techniques work with chained commands, too.

Timing

Sometimes you want to run commands at a predetermined time interval or at a particular time. The classic way to manage timed execution is cron. Many distributions provide predefined directories that run things every hour, every minute, etc. However, the best way is to simply edit your crontab. Usually, you’ll create a script and then use that script in the crontab, although that’s not always necessary.

The crontab is a file that you must edit with the crontab command (execute crontab -e). Each line that isn’t a comment specifies a program to run. The first part of the line tells you when to run and the last part tells you what to run. For example, here’s an entry to run the duckdns update program:

*/5 * * * * ~/duckdns/duck.sh >/dev/null 2>&1

The fields specify the minutes, hour, day of the month, month and day of the week. The */5 means every 5 minutes and the other * mean any. There’s a lot of special syntax you can use, but if you want something easy, try this crontab editor online (see figure).

One problem with cron is that it assumes your computer is up and running 24/7. If you set a job to run overnight and the computer is off overnight, the job won’t run. Anacron is an attempt to fix that. Although it works like chron (with limitations), it will “catch up” if the computer was off when things were supposed to run.

Sometimes, you just want to run things once at a certain time. You can do that with the at command:

at now + 10 minutes

You will wind up at a simple prompt where you can issue commands. Those commands will run in 10 minutes. You can specify absolute times, too, of course. You can also refer to 4PM as “teatime” (seriously). The atq command will show you commands pending execution and the atrm command will kill pending commands, if you change your mind. If you use the batch form of the command, the system will execute your commands when the system has some idle time.

If you read the man page for at, you’ll see that by default it uses the “a” queue for normal jobs and “b” for batch jobs. However, you can assign queues a-z and A-Z, with each queue having a lower priority (technically, a higher nice value).

One important note: On most systems, all these queued up processes will run on the system default shell (like /bin/sh) and not necessarily bash. You may want to explicitly launch bash or test your commands under the default shell to make sure they work. If you simply launch a script that names bash as the interpreter (#!/usr/bin/bash, for example), then you won’t notice this.

Spooling

Although the at command has the batch alias, it isn’t a complete batch system. However, there are several batch systems for Linux with different attributes.  One interesting choice is Task Spooler (the task-spooler in the Ubuntu repositories). On some systems, the command is ts, but since that conflicts on Debian, you’ll use tsp, instead.

The idea is to use tsp followed by a command line. The return value is a task number and you can use that task number to build dependencies between tasks. This is similar in function to at, but with a lot more power.  Consider this transcript:

alw@enterprise:~$ tsp wget http://www.hackaday.com
0
alw@enterprise:~$ tsp
ID State Output E-Level Times(r/u/s) Command [run=0/1]
0 finished /tmp/ts-out.TpAPIV 0 0.22/0.00/0.00 wget http://www.hackaday.com
alw@enterprise:~$ tsp -i 0
Exit status: died with exit code 0
Command: wget http://www.hackaday.com
Slots required: 1
Enqueue time: Fri Jun 9 21:07:53 2017
Start time: Fri Jun 9 21:07:53 2017
End time: Fri Jun 9 21:07:53 2017
Time run: 0.223674s
alw@enterprise:~$ tsp -c 0
--2017-06-09 21:07:53-- http://www.hackaday.com/
Resolving www.hackaday.com (www.hackaday.com)... 192.0.79.32, 192.0.79.33
Connecting to www.hackaday.com (www.hackaday.com)|192.0.79.32|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://hackaday.com/ [following]
--2017-06-09 21:07:53-- https://hackaday.com/
Resolving hackaday.com (hackaday.com)... 192.0.79.33, 192.0.79.32
Reusing existing connection to www.hackaday.com:80.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’

0K .......... .......... .......... .......... .......... 1.12M
 50K .......... .......... .......... ... 6.17M=0.05s

2017-06-09 21:07:53 (1.68 MB/s) - ‘index.html’ saved [85720]

The first command started the wget program as a task (task 0, in fact). Running tsp shows all the jobs in the queue (in this case, only one and it is done). The -i option shows info about a specified tasks and -c dumps the output. Think of -c as a cat option and -t as a tail -f option. You can also get output mailed with the -m option.

Typically, the task spooler runs one task at a time, but you can increase that limit using the -S option. You can make a task wait for the previous task with the -d option. You can also use -w to make a task wait on another arbitrary task.

If you look at the man page, you’ll see there are plenty of other options.  Just remember that on your system the ts program could be tsp (or vice versa). You can also find examples on the program’s home page. Check out the video below for some common uses for task spooler.

Like most things in Linux, you can combine a lot of these techniques. For example, it would be possible to have cron kick off a spooled task. That task could use scripts that employ the && and || operator to control how things work internally. Overkill? Maybe. Like I said earlier, you could just bite the bullet and write a script. However, there’s a lot of functionality already available if you choose to use any or all of it.

26 thoughts on “Linux-Fu: Running Commands

    1. systemd can parse cron files, also, if you set up the right packages. I didn’t mention .timer files, because of that, but looking I see that, yes, you can configure a system that doesn’t know about cron at all. That seems like that would break many things.

  1. There should be a header link or section dedicated to these articles called tutorials… NO SERIOUSLY!!!!!!!!!!!

    This would of been a heaven send a few years back for my noob self of times gone by.
    I had to find these things out bit by bit and trying to find this stuff without knowing what to look for, just knowing to search for what you’re trying to do in the hope someone else has done the same.

    Nowadays regarding this posts’ content: I know all that, Just hope it’ll help those who don’t know yet :)

    Since bricking a generic-x86 tablet and failing to sd-mmc card mod it…
    I have been getting linux-arch (manjaro dist) running on my linx 1010B tablet pc and the learning curve by reverse engineering what hasn’t been tutorialised is monumental.
    I had learnt that no-one wants people to make a custom grub.cfg for their efi systems and instead use a script that has no clue of the hardware it is about to soft-brick! Back in efi shell got somethings working and learnt how to get rotation and other configs working….
    When I’ve sorted some quirks out (Falseaudio) I’ll make a tutorial and probably some scripts to help those who’ll face the same problems: Especially grub.cfg on bootable sticks!

    Oh, the odd RTL5640 config (No_irq, I think) causes pulseaudio(falseaudio) to abort… No sound in modern failfox on modern OS… AGAIN

    1. Oh, I was liking Pulse audio a little better as I started taming it on the cheap snaptop that had the properly wired RTL5640 to include the signal track the Linx 1010b hasn’t got and thus the snaptop* audio worked usably with pulseaudio.

      *the hinges have already gone!, glad I got it for $30/40-ish (can’t remember now) instead of the around $100-ish of the new price.

  2. “#!/usr/bin/bash” isn’t very portable. Some OS put the shells in /bin/ (once upon a time, /usr/ was for extras and / for the basics, “we need it in degraded boot”, but it seems to be a lost cause), other times you get them in /usr/local/bin/ (manual compilation, bsd ports).
    I think “#!/usr/bin/env bash” (or whatever other thing you need, like python, perl, etc) is the best workaround, at least it’s very common.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s