If you ever think about it, computers are exceedingly stupid. Even the most powerful CPU can’t do very much. However, it can do what it does very rapidly and repeatably. Computers are so fast, they can appear to do a lot of things at once, too and modern computers have multiple CPUs to further enhance their multitasking abilities. However, we often don’t write programs or shell scripts to take advantage of this. However, there’s no reason for this, as you’ll see.

Bash Support

It is surprisingly easy to get multiple processes running under Bash. For one thing, processes on either side of a pipe run together, and that’s probably the most common way shell scripts using multiprogramming. In other words, think about what happens if you write ls | more.

Under the old MSDOS system, the first program would run to completion, spooling its output to a temporary file. Then the second program will run, reading input from the same file. With Linux and most other modern operating systems, both programs will run together with the input of the second program connected to the first program’s output.

That’s easy because the programs synchronize themselves over the input and output channel. However, it is just as easy to start multiple programs independently. The key is using “&” to separate programs or end the script line.

It is easy enough to convert a script like:

find /mnt1/usr -name '*.so' >/tmp/libs1
find /mnt2/usr -name '*.so' > /tmp/libs2

# to

find /mnt1/usr -name '*.so' >/tmp/libs1 &
find /mnt2/usr -name '*.so' > /tmp/libs2 &

In the first case, the searches occur one at a time, and only proceeds after the last find has run. In the second case, both commands run at the same time and the script will continue even while both commands are still going.

The Problem

There’s only one problem. While you can spin off multiple programs to run together, it is rare that these programs don’t need to coordinate with each other.

Not that it would be useful to really run these in parallel, but take a look at the output of this command:

alw@Enterprise:~$ banner hello & banner goodbye
[1] 173


# # ###### # # ####
#### #### #### ##### ##### # # ######
# # # # # # #
# # # # # # # # # # # # #
###### ##### # # # #
# # # # # # # ##### # #####
# # # # # # #
# ### # # # # # # # # # #
# # # # # # #
# # # # # # # # # # # #
# # ###### ###### ###### ####
#### #### #### ##### ##### # ######


[1]+ Done banner hello

Not what you expected. Depending on your system, the first program may (or may not) get all of its output done in one go. Interspersing output isn’t really what you want.

Control

There are other problems. You might want to find out the PID of the new process. You could use bash’s job control. However, you don’t really need to go that far.

The jobs command will show you what you have running in the background from the current shell. If you add -l or -p you can also learn the PID.

An easier way to learn the PID of a command is using $!. Of course, if you are running from the shell’s command prompt, it also tells you the PID of the last run command. For example:

ls / & echo PID=$!
[2] 178
PID=178

Of course, your PID number will almost certainly be different.

Then What?

Armed the PID, what can you do with it? There are two things you’ll find most useful. First, you can use wait to understand when a process (or processes) are complete.

The other thing you can do is use kill to send signals to the background program. That’s beyond the scope of this post, but you can use signals to create complex communication between programs or to invoke default behaviors like termination.

An Example

Consider a case where you have a bunch of jpeg files and you want to convert them to png files for a website using ImageMagick. You might try this simple script:

#!/bin/bash
for I in *.jpg
do
    convert "$I" "png/$(basename "$I" .jpg).png"
done
echo Copying files now...
ls png

This will do the job. Presumably, the last line will have some file copy command like an sftp following it, but I used a directory listing just for an example.

Instead, you could launch all the conversions at once, taking advantage of multiple processors, and wait for them all to finish up. Without the wait command, the simulated copy would start before the conversions were complete unless there were very few conversions to do.

#!/bin/bash
for I in *.jpg
do
    convert "$I" "png/$(basename "$I" .jpg).png" &
done
wait
echo Copying files now...
ls png

Still a Problem

There is still one problem. The wait command will wait for any subprocesses active in the shell. That might not be what you want, although in this case, it is probably OK. However, let’s fix it:

#!/bin/bash
PIDs=""
for I in *.jpg
do
    convert "$I" "png/$(basename "$I" .jpg).png" &
PIDs+="$! "
done
wait $PIDs
echo Copying files...
ls png

If you run the timing with a good number of files, you’ll see a big difference. On my laptop with a handful of pictures, the straight versions took about 40 seconds. It took just over 10 seconds with the final version.

Wrap Up

It is easy to forget that you can do more than one thing at a time pretty easily. Of course, this also opens up a whole new realm of problems. If you need to protect your programs from each other, check out our earlier post about critical sections. Not everyone thinks bash is a great programming language, but it is surprisingly capable and while it might not be good for everything, it is great for some tasks.