If you use just about any modern command line, you probably understand the idea of pipes. Pipes are the ability to connect the output from one program to the input of another. For example, you can more easily review contents of a large directory on a Linux machine by connecting two simple commands using a pipe:
ls | less
This command runs ls and sends its output to the input of the less program. In Linux, both commands run at once and output from ls immediately appears as the input of less. From the user’s point of view it’s a single operation. In contrast, under regular old MSDOS, two steps would be necessary to run these commands:
ls > SOME_TEMP_FILE less < SOME_TEMP_FILE
The big difference is that ls will run to completion, saving its output a file. Then the less command runs and reads the file. The result is the same, but the timing isn’t.
You may be wondering why I’m explaining such a simple concept. There’s another type of pipe that isn’t as often used: a named pipe. The normal pipes are attached to a pair of commands. However, a named pipe has a life of its own. Any number of processes can write to it and read from it. Learn the ways of named pipes will certainly up your Linux-Fu, so let’s jump in!
Quick Example: Building a Logging Script
Suppose you want to create a simple logging facility. Of course, making a daemon that runs all the time is an entirely different subject, but I’m just going to create a simple and non-robust script. A named pipe can accept the input lines from other programs and the daemon can timestamp each line and write it to a file. Here’s the daemon:
#!/bin/bash mkfifo /tmp/nplogpipe while true do read LINE </tmp/nplogpipe echo $(date): "$LINE" >>/tmp/nplog.txt done
The mkfifo command creates the named pipe (a first in, first out or FIFO). Older scripts might use mknod for this purpose and that will work, too. If the pipe already exists, the command will fail, but it won’t matter. After that, a read waits for input from the pipe. When it arrives, the script writes out a date and the line to a log file and goes back for more. To test out your quick and dirty logging system, run the script in the background or in one terminal window. Then in the foreground or in another terminal try this:
echo The first log entry >/tmp/nplogpipe echo Read Hackaday every day >/tmp/nplogpipe
You can add a few more lines and then examine the /tmp/nlog.txt file. It should look something like this:
Sat Jun 29 07:37:44 CDT 2019: The first log entry Sat Jun 29 07:39:57 CDT 2019: Read Hackaday every day
The Tricks are in the Details
One small point: in this case, the read command executes repeatedly in the loop, but in general when a sender process exits, it will cause the receiving program to also exit. That isn’t always the behavior you want. The usual way to deal with this is to open the pipe in a way that will hold the pipe open until it is closed.
To observe this effect, try this (with the daemon running):
ls / >/tmp/nplogpipe
The log will only get the first line of the ls. That’s because by the time it finishes processing the first line, the ls command exited and that clears the pipe. Now try this:
exec 3>/tmp/nplogpipe ls / >&3 free >/tmp/nplogpipe exec 3>&- # close pipe
The first exec line holds the pipe open until the last line closes it. Once open you can refer to the pipe as &3 or by its full name. Now all the output will appear in the log file.
Another nuance is that the pipe sort of looks like a file. That means that programs that expect files can usually use pipes. It also means you can control access to a pipe using the same security mechanisms that work with files (e.g., chmod to set permissions for a specific user or group).
Why Use Named Pipes?
You might wonder what advantage these have compared to a regular pipe or a file. Unlike a regular file, the pipe doesn’t fill up. That also means it has more chance of staying in memory although, of course, it could get swapped out just like any other memory. In addition, it is easy to have multiple writers to a named pipe.
Of course, there are other issues to worry about. For example, if multiple programs are writing more than one line of data to the pipe at a time, you’d have to work out your own scheme to sort them all out. Still, for a quick way to push and collect data between possibly unrelated processes, named pipes are an easy way to go.
By the way, this is the twentieth installment of Linux Fu! The links below will take to the earlier postings and stay tuned for even more to come.
You can tell the script to keep the pipe in memory or write it to a log.
Assuming you were wanting to keep that particular pipe in the swapspace.
Funny, I remember using “C:\> DIR *.* > MORE” all the time in MS-DOS without any temp file.
The temp file was there it was just hidden from you. But the directory would put it’s entire output into a file which then more would read. There was no multitasking to support a true pipe.
Also you wouldn’t have used >More as that would write to a file named More, but I know you meant | More.
so the article is still wrong, “two steps would be necessary to run these commands”
Well, I guess I was assuming most people remembered how this worked so I didn’t go into a lot of detail. Yes, you typed DIR | MORE but under the hood it was two completely separate serial invocations. I didn’t mean to imply you can’t say “ls | more” or “dir | more” or whatever. But the fact is it simply does the two steps for you and SOME_TEMP_FILE is made for you without your knowledge. You are not running both things in parallel as one pipeline as you are in Unix/Linux.
The difference is obvious when you have something that takes a long time to run. For example, if you do a recursive ls that takes 5 minutes to do its output, you won’t see the first bit of output for 5 minutes on MSDOS. You will see output appear at once on Unix. Because on Unix it really is one set of things executing at one time. On MSDOS it is one thing and then another thing after that. And for long output that is apparent to the user even though it LOOKS like one thing.
However, that really isn’t the point of the article anyway.
No, it’s not wrong. It just omits the important part. By “regular old MSDOS” they could have meant MS-DOS before version 3 (maybe version 4, so don’t beat me). Back then it was more like CP/M and had nothing like pipes, this was “backported” into MS-DOS from Xenix. Because MIcro$oft wanted to have two partially compatible products (what a time), they added several things from Unix into DOS, like pipes, tree file structure and compatibility layer (look into the open-sourced versions of MS-DOS).
Great article -and series- Al. While we have you on a tangent, here, do you know if DIR /p worked in the same way?
Thanks, and keep educating us on the command line.
I think it did not, but without checking I am not certain.
Thanks for the series Al. Nice reference even if I (think) I already know something…
I learned about the mkfifo command (relatively) recently when trying to set up a Minecraft server. The game has a console that you can enter commands, but obviously you can’t do that while the process is detached / daemonized.
The solution is mkfifo to create a named pipe in /tmp, then use that as std. input to minecraft-server.jar. Now, the server starts up, and you can cat or echo commands into the pipe, and have them take effect on the server!
In my case I rigged up a little php script to accept input and pass it to the pipe, thus making a web console for the server.
That’s literally the easiest way to create a headless Linux setup.
Not the most secure, but if you’re the only person with physical access it works well enough.
You having fun
One of the big reasons to use a named pipe is the same as any pipe:
Block-on-read gives you automatic synchronization between processes.
Make all your pipes for the pipeline of whatever your doing.
Start the processes in the background with & in reverse order feeding them their pipes by filenames or via input redirection.
Watch as virtually none of the inflight data ever touches the disk and you use maximum core count.
I’ve decided the term for this is Reverse Posix Notation whether it’s true or not.
This is why we must read text books on Linux.
I mean, what utility are FIFOs without tee and &, which didn’t even rate a mention?
Maybe they are in the next installment of this pretty cool series?
Rather missed his chance, I’d suggest. Anything’s possible.
M: they are useful for interprocess communication, i.e., not directly from the command line, but by independent programs that read and write the fifo.
Why is the pipe sign never found on a belgian keyboard? The only way to produce it is to copy it from the internet. Been doing that for 20 years. An azerty keyboard with pipe still not exists.
Doesn’t alt+124 work?
Mac:
https://stackoverflow.com/questions/5621173/xcode-mac-keyboard-shortcut-to-type-the-or-sign-double-vertical-bar/45824945
https://infopython.wordpress.com/2007/08/28/mac-keyboard-shortcuts-how-write-a-pipe-or-backslash-on-macbook/
https://discussions.apple.com/thread/943651
PC:
https://retrocomputing.stackexchange.com/questions/3182/why-are-the-and-keys-labelled-the-wrong-way-around mentions that “France AZERTY and Germany QWERTZ layouts have a single split pipe ¦ that types an unsplit pipe | (via Alt Gr + 6 and Alt Gr + < respectively)…”
In the pic at https://d36tnp772eyphs.cloudfront.net/blogs/1/2012/03/Belgium-Keyboard-1.jpg , though, it looks like it could be on the “8” key. Try that with Alt-Gr. (Pic from https://matadornetwork.com/abroad/the-ultimate-guide-to-computer-keyboards-around-the-world/ .) Ah, no… The pic on http://kbdlayout.info/KBDBE/ seems to show that's “!”.
http://kbdlayout.info/0001080C/how/%7C seems to suggest Alt-(or was that Alt-Gr?)-124, i.e. hold down Alt(-Gr?) while typing “1”, “2”, and “4” (in that order), then release Alt(-Gr?)
One user on https://teamtreehouse.com/community/the-pipe-key-in-the-console-doesnt-work suggests Alt-Gr + “1”; another says that doesn't work… Idunno — good luck, I guess?
HTH!
Great article as always Al! I used files to exchange some data between programs, in my ignorance …
Pipes & Daemons have been around for a very long time! See this great illustration drawn by Phil Foglio in the late 1970’s (complete with PDP-11): http://www.mckusick.com/beastie/jpg/foglio.jpg
Indeed. The UNIX implementation of MSG for NSW (National Software Works) used named pipes for interprocess communication ca. 1979
https://apps.dtic.mil/dtic/tr/fulltext/u2/a084088.pdf
Indeed. The UNIX implementation of MSG for NSW (National Software Works) used named pipes for interprocess communication ca. 1979
https://apps.dtic.mil/dtic/tr/fulltext/u2/a084088.pdf
What’s GNU/Linux specific about this?
The linux implementation of pipes and the associated commands have been tested much more extensively on linux than on the other unixes. You will find degenerate situations on AIX or Solaris where processes die and pipes stall because these operating systems simply do not receive the level of QA that linux does. There is nobody left at either Oracle or IBM who has any idea what is going inside AIX and Solaris, any such people left decades ago.
i guarantee the /tmp/nlog.txt file will not be written to if you follow this tutorial. period.
You’re right, but I guarantee that the /tmp/nplog.text file will be.
i fully agree. +1