[Marek Gibney] poses an interesting puzzle. What does the following bash command line print?
( echo red ; echo green 1>&2 ) | echo blue
You’d like to think it prints three lines: red, green, and blue. But would you be surprised to find out that it can sometimes output “blue green” and sometimes just output blue. The first surprise is that it isn’t deterministic. But the second thing that is surprising is the sometimes the entire left-hand part of the line doesn’t do anything. [Chris Siebenmann] did the analysis and explains what’s going on in a recent blog post.
Before you click the link or read further, you might want to see if you can deduce what’s going on. Give up? Here’s a hint: Part of the solution hinges on the fact that echo is built into the shell.
Breaking the command line down, the left part will spin off a subshell. The “echo blue” part runs in the original shell. Keep in mind that since echo is built-in, there won’t be more processes. The echo red gets piped to the right-hand part and it ignores the input. However, the green part redirects to the stderr stream, so it makes sense you’d get some combination of blue and green, but then why do you sometimes get just blue?
According to [Chris], there are three possible outcomes:
- The left side prints green and exits before the right side tries to write. You get green blue.
- The left side prints red, the right side prints blue, and then the second part of the left side prints green. You get blue green.
- Echo blue runs first and ends. The echo red then tries to write to a broken pipe and dies with a SIGPIPE signal.
This is a good example of how simple tools can combine in surprising ways and sometimes even the simplest commands can require an understanding of underlying details that might seem unimportant.
You could argue, of course, that the line isn’t a great example of shell script. That’s true, but you don’t always pick the code you are analyzing.
There’s plenty of things to consider when writing shell scripts. There’s even a tool for checking scripts similar to the way lint checks C programs.
Hmm, tried that on 3 different systems – Mac 10.13.6, Debian7 & Debian8
The Mac & Deb7 threw up different output every so often, the Deb8 system not at all but it is also the most grunty box if that has something to do with it…
well I kept getting green/blue on my primary old Debian system…
so I decided to script it as follows:
Well, I noted the output would flicker between mostly green/blue with occasional blue/green…
Also I noted that the script started off using the CPU as expected for a “realtime loop”, yet the script became more demanding of the CPU… thus confirming that
does indeed spawn a subshell.
watch -n 1 -d ‘( echo red ; echo green 1>&2 ) | echo blue’
I assume your system was too old to support watch.
BTW, the -d option highlights changes, which makes them easier to notice.
Surprisingly my system has watch…
Well, running that the output is rock steady
green
blue
The hwmon script I use just waits until the state of the /sys/…/BAT0/status changes before doing a refresh routine… otherwise it just runs a loop that overwrites the last display before waiting 0.5s … until the adapter status changes… then it clears the screen…
That’s my current alternative to using watch.
I’m going to put those scripts online at some point…
This is why I don’t like to program in bash. I’ll use it to script lists of commands and sometimes some very basic logic (Loop over files, do something if this exists, etc), but that’s it, unless I don’t have a choice.
That, and the syntax is fussy and the command line flags are cryptic and I like python a lot.
Oddly enough, I’ve experienced the fussy syntax for even the simplest of things in a bash script I’ve wrote.
One of the most odd bugs was getting:
-en Syntax error
((counter– not found
spewing over my output… lol
I found I had to specify “bash” to run a bash script as a bash script in a spawned xterm window…
I thought “#!/bin/bash” was enough to tell terminal emulators it’s a bash script.
The other was figuring out how to do maths in the script and how to compare contents of variables…
Same here, but I use perl, kind of the original glue language – partly because I know it the best compared to other scripting languages, and partly because…CPAN has it done already more often than not. Probably true of some other scripting languages, it’s a case of go with what you know.
Fussy syntax has its proponents to be sure, but I’m not a believer that a picky language always creates better code (and you know which one I’m talking about…I’m holding out for magnetite.).
If I’m going to compile it…C or C++. Most *good* languages have warnings and strictures you can turn on and off…
What drives me nuts with bash, since I do perl day in and day out, is…the sigils – $ in bash vs in perl, holy cow, on one side of an = but not the other? Whitespace sometimes has meaning but not consistently? Maybe it’s just lack of having done tons of bash, but I note an 80+ year old sysadmin and friend struggles with it from time to time, even though he also has sed and awk down cold (and JCL and COBOL and C). Stuff like what is mentioned in the article might be the reason, it’s the first I’ve heard of this quirk. I don’t mind extra threads and processes, but I like to make them a little more obvious/explicit to the reader/maintainer.
>That, and the syntax is fussy and the command line flags are cryptic and I like python a lot.
I totally agree with this, as a Bash newbie I was really struggling to get something to behave the way I expected it to until I realised I could just punt it over to Python and get it working almost immediately. I appreciate that Bash has some powerful features but if Python gets the same thing done faster without having to learn a new and very fussy and complex syntax then it isn’t a hard choice.
The mention of echo being built into the shell brings back an old memory for me. Decades ago when mucking with minimal or broken installs of Minix and Xenix-86, there was not a complete set of commands. With no ‘ls’ I would often make use of ‘echo *’ to list the files.
It is a trick that still has to be used in many embedded linux devices, where most of the commands were also removed. ” echo * ” still helps a lot.
It’s an interesting curiosity, that is all. One does not write such broken code in any programming language and expect a sensible or consistent result.
I got
blue
green
as the output.
Now try this: put a “$” in front of the piped command.
$( echo red ; echo green 1>&2 ) | echo blue
After the same output I had before, I was in a subshell (?) that could (apparently) only be exited by the command `q`.
It’s like ed! man ed! ed man!
~$ $( echo red ; echo green 1>&2 ) | echo blue
blue
green
6
?
8
?
9
?
yr
?
fgherityj
?
Eat flaming death
?
q
~$
Also, note that this article says:
( echo red ; echo green 1>&2 ) | echo blue
but the original article (https://utcc.utoronto.ca/~cks/space/blog/unix/ShellPipelineIndeterminate) says:
(echo red; echo green 1>&2) | echo blue
The addition of spaces turns
green
blue
into
blue
green
on my system.
For anyone looking to write portable shell scripts, this is undefined behavior. The best option here is to never rely on such harebrained code.
Nobody ever claimed that *nix shells be idiot-proof.
But sure behind every corner are lurkers feeding shells with nonsensical input!
Ever noticed that you can hurt yourself even with toddler cutlery?
Good to know, isn’t it?
Thanks HAD, for showing the limits of our everyday tools :-)
Personally I prefer ksh for running scripts. It consistently (maybe not 100%) produces green blue, unless you put a sleep in between red green, then it will print blue green consistently. That said, bash will consistently produce blue if you put the sleep in place. ksh is a bit smarter allowing the left side to finish instead if killing it prematurely when the right side closes first.
( echo red ; sleep 1 ; echo green 1>&2 ) | echo blue
I don’t get what’s weird..
echo does not print stuff from stdin ( when you do echo red | echo green, you only get ‘green’), so what’s inside the parenthesis wouldn’t print anything (as both their outputs are piped to the echo), except the second echo is redirected to stderr, which will be printed asynchronously to the terminal.
To check that both echos are piped to the third one, when you do (echo green; echo red) | sed ‘s/e/a/g’, you indeed get ‘graad rad’, meaning both echo outputs do get into the pipe.
To get green blue instead of blue green from time to time (never happened to me yet) is to be expected because error printing is asynchronous, right?
Or is there something I really don’t understand?