pipes

5 Articles

If Only The Kids Knew About Pipes

October 18, 2022 by 50 Comments

Being a parent is hard work. You need to prepare your child to operate in the world. Reading, writing, arithmetic, and how to make good choices are just the beginning. They also ought to know regular expressions and Unix pipes. [Jackdoe] can help with that last one, thanks to their Unix pipe card game.

As an example, the task “print the most common line from a file” would require the answer:

cat 03.txt | sort | uniq -c | sort -n | tail -1

You can vary the rules to declare whoever has the smallest pipe or the largest pipe that accomplishes the task as the winner. We’d add a house rule that whoever has the fastest pipe ought to get something. We don’t, however, think this card game will make the Vegas tables, unfortunately.

You can print your own card deck and even the box. Or you can buy a nice set, if you don’t want to spend the time. You should probably know about cat, grep, tail, head, wc, sort, and uniq along with their options.

If you want some other esoteric kids’ learning activities, they also have “Programming Time” which teaches algorithms and Python, and “4917” to teach the basics of machine code.

If you want help with the other key skill — regular expressions — check out regexp golf or crosswords.

Optimizing Linux Pipes

June 3, 2022 by 28 Comments

In CPU design, there is Ahmdal’s law. Simply put, it means that if some process is contributing to 10% of your execution, optimizing it can’t improve things by more than 10%. Common sense, really, but it illustrates the importance of knowing how fast or slow various parts of your system are. So how fast are Linux pipes? That’s a good question and one that [Mazzo] sets out to answer.

The inspiration was a highly-optimized fizzbuzz program that clocked in at over 36GB/s on his laptop. Is that a common speed? Nope. A simple program using pipes on the same machine turned in not quite 4 GB/s. What accounts for the difference?

Continue reading “Optimizing Linux Pipes” →

Linux Fu: Simple Pipes

March 16, 2022 by 15 Comments

In the old days, you had a computer and it did one thing at a time. Literally. You would load your cards or punch tape or whatever and push a button. The computer would read your program, execute it, and spit out the results. Then it would go back to sleep until you fed it some more input.

The problem is computers — especially then — were expensive. And for a typical program, the computer is spending a lot of time waiting for things like the next punched card to show up or the magnetic tape to get to the right position. In those cases, the computer was figuratively tapping its foot waiting for the next event.

Someone smart realized that the computer could be working on something else while it was waiting, so you should feed more than one program in at a time. When program A is waiting for some I/O operation, program B could make some progress. Of course, if program A didn’t do any I/O then program B starved, so we invented preemptive multitasking. In that scheme, program A runs until it can’t run anymore or until a preset time limit occurs, whichever comes first. If time expires, the program is forced to sleep a bit so program B (and other programs) get their turn. This is how virtually all modern computers outside of tiny embedded systems work.

But there is a difference. Most computers now have multiple CPUs and special ways to quickly switch tasks. The desktop I’m writing this on has 12 CPUs and each one can act like two CPUs. So the computer can run up to 12 programs at one time and have 12 more that can replace any of the active 12 very quickly. Of course, the operating system can also flip programs on and off that stack of 24, so you can run a lot more than that, but the switch between the main 12 and the backup 12 is extremely fast.

So the case is stronger than ever for writing your solution using more than one program. There are a lot of benefits. For example, I once took over a program that did a lot of calculations and then spent hours printing out results. I spun off the printing to separate jobs on different printers and cut like 80% of the run time — which was nearly a day when I got started. But even outside of performance, process isolation is like the ultimate encapsulation. Things you do in program A shouldn’t be able to affect program B. Just like we isolate code in modules and objects, we can go further and isolate them in processes.

Doubled-Edged Sword

But that’s also a problem. Presumably, if you want to have two programs cooperate, they need to affect each other in some way. You could just use a file to talk between them but that’s notoriously inefficient. So operating systems like Linux provide IPC — interprocess communications. Just like you make some parts of an object public, you can expose certain things in your program to other programs.

Continue reading “Linux Fu: Simple Pipes” →

Linux Fu: Named Pipe Dreams

July 12, 2019 by 30 Comments

If you use just about any modern command line, you probably understand the idea of pipes. Pipes are the ability to connect the output from one program to the input of another. For example, you can more easily review contents of a large directory on a Linux machine by connecting two simple commands using a pipe:

ls | less

This command runs ls and sends its output to the input of the less program. In Linux, both commands run at once and output from ls immediately appears as the input of less. From the user’s point of view it’s a single operation. In contrast, under regular old MSDOS, two steps would be necessary to run these commands:

ls > SOME_TEMP_FILE
less < SOME_TEMP_FILE

The big difference is that ls will run to completion, saving its output a file. Then the less command runs and reads the file. The result is the same, but the timing isn’t.

You may be wondering why I’m explaining such a simple concept. There’s another type of pipe that isn’t as often used: a named pipe. The normal pipes are attached to a pair of commands. However, a named pipe has a life of its own. Any number of processes can write to it and read from it. Learn the ways of named pipes will certainly up your Linux-Fu, so let’s jump in!

Continue reading “Linux Fu: Named Pipe Dreams” →

Pipe Heating With An ATMega8

February 5, 2010 by 22 Comments

[Viktor] wanted a system to keep his pipes from freezing.  The common method of using heat tape works pretty well, but can be wasteful. Many people just turn it on for the entire winter. [Viktor] wanted to automate the heat tape’s power so that it only activated during times that the pipes could actually freeze. To do this, he rigged an ATMega8 to a temp probe and is using it to control an ATX power supply. Pretty slick [Viktor].

[via HackedGadgets]