Linux Fu: Failing Pipelines

16 Comments

Bash is great for automating little tasks, but sometimes a little script you think will take a minute to write turns into a half hour or more. This is the story of one of those half-hour scripts.

I have too many 3D printers. In particular, I have three that are almost — but not exactly — the same, so each one has a slightly different build process when I want to update their firmware. In all fairness, one of those printers is heading out the door soon, but I’ll probably still wind up building firmware images for it.

My initial process was painful. I have a special directory with the four files needed to configure Marlin for each machine. I copy all four files and ask PlatformIO to perform the build. Usually, it succeeds and gives me a file that looks like firmware-yyyyddmmhhmm.bin or something like that.

The problem is that the build process doesn’t know which of the three machines is the target: Sulu, Checkov, or Fraiser. (Long story.) So, I manually look at the file name, copy it, and rename it. Of course, this is an error-prone process, and I’m basically lazy, so I decided to write a script to do it. I figured it would take just a minute to bundle up all the steps. I was wrong.

First Attempt

Copying the files to the right place was a piece of cake. I did check to make sure they existed. The problem came from launching PlatformIO, seeing the result on the screen, and being able to parse the filename out of the stream.

I thought it would be easy:

FN=$(pio run | grep '^Renamed to' | cut -d ' ' -f 3 )

That should do the build and leave $FN with the name of the file I need to rename and process. It does, but there are two problems. You can’t see what’s happening, and you can’t tell when the build fails.

Easy Problem First

The pipeline consumes the build’s output. Of course, a tee command can manage that, right? Well, sort of. The problem is that the tee command sends things to a file and standard out, but the standard out, in this case, is the pipe. Sure, I could tee the output to a temporary file and then process that file later, but that’s messy.

So, I resorted to a Bash-specific feature:

FN=$(pio run | tee /dev/fd/2 | grep ...

This puts the output on my screen but still sends it down the pipe, too. Sure, there are cases when this isn’t a good idea, and it isn’t very portable, but for my own use, it works just fine, and I’m OK with that. There are other ways to do this, like using /dev/tty if you know you are only using the script from a terminal.

Harder Problem

The bigger problem is that if the build fails — and it might —  there isn’t a good way to fail the whole pipeline. By default, the pipe’s return value is the last return value, and cut is happy to report success as long as it runs.

There are a number of possible answers. Again, I could have resorted to a temporary file. However, I decided to set a bash option to cause any failing item in a pipe to fail the whole pipe immediately:

set -o pipefail

So now, in part, my script looks like this:

set -o pipefail
FN=$(pio run | tee /dev/fd/2 | grep '^Renamed to' | cut -d ' ' -f 3 )
if [ $? -eq 0 ]
then
   echo Success...
   cp ".pio/build/STM32F103RC_creality/$FN" "configurations/$1"
   echo Result: "configurations/$1/$FN"
else
   echo Build failed
   exit 3
fi

Final Analysis

Is it brain surgery? Nope. But it is one of those bumps in the road in what should have been a five-minute exercise. Maybe next time you run into it, you’ll save yourself at least 25 minutes. This gets the job done, but it isn’t a stellar example of bash programming. I would hate to run it through a lint-like checker.

16 thoughts on “Linux Fu: Failing Pipelines”

  2. Hey, my name’s dave and I run a channel online where I go over bash tips and tricks and I see a lot of ways this code can be cleaned up.

    For starters, you can get the exit code of every command in a pipeline with the builtin array variable in bash PIPESTATUS… for example:

    “`
    $ echo hi | grep hi | grep bye | tail
    $ echo “${PIPESTATUS[*]}”
    0 0 1 0
    “`

    We can quickly identify that 3rd command in this pipeline failed.

    But beyond that, you don’t actually need a complicated pipeline and can break it up into smaller chunks of work. You can rework your whole script (I hope the formatting here works):

    “`
    #!/usr/bin/env bash

    # run command, save stdout and exit code
    output=$(pio run)
    code=$?

    # print all command output
    echo “$output”

    # stop here if the command failed
    if ((code != 0)); then
    echo ‘Build failed’
    exit 3
    fi

    # command was a success, extract filename and copy it
    filename=$(awk ‘/^Renamed to/ { print $3 }’ <<< "$output")

    # stop here if awk failed or the filename variable is empty
    if (($? != 0)) || [[ -z $filename ]]; then
    echo 'Failed to extract build filename'
    exit 3
    fi

    # we are good to go
    echo 'Success…'
    cp ".pio/build/STM32F103RC_creality/$filename" "configurations/$1" || exit 3
    echo "Result: configurations/$1/$filename"
    “`

    This will:

    1. add more error checking
    2. print the `pio run` output to the screen and retains its exit code
    3. parse the filename with a single invocation of `awk` (and error check it)

    Bash can be super fun and I love seeing posts about it on this site!

    dave

    Report comment
    Reply

    1. No doubt there are many ways to do this. While $PIPESTATUS clearly would work, I didn’t like the idea of using output=$(…) for the same reason I didn’t want to use temporary files. There is a ton of output from the build. Granted, this is a quick one-off so it didn’t really matter but that’s kind of the point, too, is it is a one-off. I would have been more fastidious with errors had I wanted to do this in a real environment. As it was, it was just a quick-and-dirty way to be lazy ;-) If you look back in the Linux Fu archives, there are a lot of bash tricks ranging from the silly to the sublime ;)

      Report comment
      Reply

      1. If you put your bash code into a file it is never a one-off but a solution for eternity. =)

        So a little effort to make it serviceable is time well spent. I have such simple helpers that are more than a decade in use.

        And I’m with Dave on this one: catching intermediate output and breaking up a pipe is often the right thing to do. Especially if one needs to react to complex situations.

        Report comment
        Reply

    1. It does make me wonder how many Al’s running, hey. Is it really only 3? Or is it only 3 that are largely identical, with additional non-identical machines rounding out the total?

      Report comment
      Reply

  5. Or, dont care that it failed? Either you found a renamed thingiemabob that is valid, build passed, or its empry, build failed.

    Alternativly spitballing on my phone, something like
    `FN=”$(pio run || exit_fun 1 | grep | cut)”`
    Might work? No clue, never tried it ;p but bshellcheck seems to be ok with it, but it just might not do what you expect too :p

    This should call your exit function (or just `exit 1` if you dont need the result)

    I have a ryle to only write posix compliant scripts and dont use bashisms, so maybe theres a bashy way to do this better.

    Report comment
    Reply

  7. Keptin! I don’t Trrusst Frraiserr!
    … Steady as she goes, Mr Chekov
    [Fraiser adjusts his red shirt awkwardly] I have a bad feeling about this
    … Don’t mix universes, Mr Fraiser. Never quote me the odds. You’ll be fine.
    [10 minutes later]… Hee’s dayed, Jimm

    Report comment
    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.