Software Development in… Bash

Truly good ideas tend to apply in all situations. The phrase is “never run with scissors”, not “don’t run with scissors unless you are just going into the next room.” Software development methodology is a good idea and most of us have our choice of tools. But what if you are developing a significant amount of bash or similar script? Should you just wing it because bash isn’t a “real” programming language? [Oscar] says no, and if you are writing more than two or three lines of script, we agree.

We’ve made the argument before (and many of you have disagreed) that bash is a programming language. Maybe not the greatest and certainly not the sexiest, but bash is near ubiquitous on certain kinds of systems and for many tasks is pretty productive. [Oscar] shows how he uses a source code formatter, a linter, and a unit test framework to bring bash scripting in line with modern software development. We are pretty sure he uses source control, too, but that seems so elementary that it doesn’t come up outside of a link to his repository in GitHub.

The linter is especially important (we have covered it before, by the way). Since bash is interpreted, it is hard to catch all errors until runtime. It also catches portability issues, if you care about that. We were also impressed with the unit test framework since that would enable test-driven development, which is very productive.

Next step? Write a game. Or use bash to scrape website data.

43 thoughts on “Software Development in… Bash

      1. perl… Moderators, the previous post is missing its tags…

        undef $/;open(_,$0);/ \dx([\dA-F]*)/while();@&=split(//,$1);@/=@&;
        $”.=chr(hex(join(“”,splice(@&,0,2))))while(@&); eval$”;

        ($C,$_,@\)=(($a=$/[1]*4)*5+1, q| |x(0x20).q|\||.chr(32)x(0x10).q$*$.
        chr(0x20)x(0x10).(pack(“CC”,124,10)), sub{s/.\|(\s*?)(\S)./\|$1 $2/},
        sub{s/\|(\s*?).(\S)/ \|$1$2 /}, sub{$2.$1.$3},sub{$tt=(3*$tt+7)%$C},
        sub{$1.$3.$2});
        while ($_) {
        select $/, undef, $/, $C/1E3;
        (sysread(STDIN, $k, 1),s/(.)(\*)(.)/(&{$\[(ord($k)-44&2)+2]})/e)
        if (select($a=chr(1),$/,$/,0));

        print 0x75736520504F5349583B2024743D6E657720504F5349583A3A5465726D696F73
        3B24742D3E676574617474722828303D3E2A5F3D5C2423292F32293B2024742D3E
        365746C666C61672824742D3E6765746C666C6167267E284543484F7C4543484F4
        7C4943414E4F4E29293B2024742D3E7365746363285654494D452C31293B24742D
        E7365746174747228302C544353414E4F57293B24643D224352415348215C6E223B0A;

        ($p?(/.{70}\|$/):(/^\|/))||(&{$\[3]}<$/[0])?($p=!$p):&{$\[$p]}||die("$d");
        (&{$\[3]}<$/[1])&&(s/ \|$/\|/);
        (/\|.*\*.*\|$/)||die("$d");
        }

        The above is valid perl… I rest my case.

    1. I’ve been working in big company where somebody thought similar and started rewriting few thousands, not badly written, bash lines into python. To cut it short it was a much worse. Sometimes it’s about right tool to get the job done, while bash isn’t pretty, it excels in doing exactly what’s told.

      BTW. When I left there was a bit more than a 1k bash and quite a few times more python using bash.

      Though I was suprised how many people didn’t know about ‘set – i’ which is essential imo for usable scripts. The same for set x, extglobs etc.

    2. i use stand alone lua for a lot of things. it has some good modules, a fairly easy to use gui, an old but functional version of opengl, serial and networking for talking to embedded projects, and its great for writing text file parsers out of the box. i guess everyone has their language of choice and this one is mine.

  1. Tools to help making good code are always nice, but not having them in one place generally sucks.
    But as for most things, vim and some plugins can take care of most steps:
    The formatting can be done with “=” on selected text. It only corrects indentation, but it is generally the main formatting variation within code (or the most annoying).
    As for the linter, there is the great syntastic addon, that can do regular bash or POSIX (and switch automatically depending on your shabang (#!/bin/xxx), if you use bash or sh), it checks syntax at each write and have pretty clear error explanations.

    As for unit tests, they are generally a pain, but for stuff requiring non pipeable inputs (such as su and interactive scripts), you can use expect (which works with is own scripting language but is the most well known) or empty (package name empty-expect on debian), which creates named pipes for input and outputs so they can be read and written with another script.

  2. The nice thing about shell script is you can manually run the algorithm as you develop it. It’s a capability that would require some sort of insane reflection in other languages but is pretty easy in the whole shell family. That probably makes some traditional tools for debugging nearly irrelevant.

    1. Oh yes. I use the shell to check the correct functioning of recipe lines when writing makefiles. It makes beating make into submission a lot easier without having to use all manner of esoteric debugging constructs.

  3. Probably the trickiest thing I’ve done with bash is video editing. I split the file out into individual frames with FFMPEG, then use a few for loops to copy the files to the correct order I want, and finally combining the files back to a video file. I do this if I have a lot of tight, repetitive edits. It’s easier to script it than to do it all manually in video editing software.

    1. I wrote a fidget for a web site years back that would take a video that was uploaded and look at how long it was, figure out what it would have to be divided by to make it 10 seconds long, grab every X frame and create an animated gif file for the preview. Before that they just took the first frame of the video and that was very unpopular.

    2. ffmpeg and bash ‘for’ loops – possibly the world’s greatest combination for mass conversion of video files. My son’s GoPro has a strange proprietary codec and the video files won’t play on a computer without being put through GoPro’s own software – which is a hideous mess. So I line up the video files in one directory, process them through a ‘for’ loop using ffmpeg to convert them, and output playable MP4 files.

  4. They should have called Bash something else such as Glue, because that is what it is best for, gluing all of the Unix style application together. You can iterate through data and push it through pipelines that each use whatever thread on whatever core the kernel has available so things become as parallel as your CPU/s are capable of, without any extra effort.

    1. Most of Unix/Linux tools are named after either funny noises your body makes or things you want to do to your computer after first attempt at using them. First time I used bash I wanted to do exactly that: bash my computer. Fortunately I don’t have to use Linux too often…

    2. Bash is the Bourne-again shell. The original shell program is designed to do exactly what you say, a safe smooth covering for the wiggly bits. The pipes go all the way back to PDP-11 days when SMP was a futuristic fantasy. Shell scripts make the world go round.

  5. I really wish people would stick to using UNIX shell script because that ensures portability. Bash just adds a few things but that makes it’s wholly incompatible with most shells.

    1. But the shebang takes care of selecting the right interpreter (bash/sh/zsh and so on). And virtually all dists have bash already installed (even if not used as the default shell) so I fail to see the big issue. Of course severely constrained tiny linuxes as those running on embedded systems might not have bash, but they usually lack most of the other external utility softwares the script makes use of

      In the name of compatibility I guess would be more important to not use gnu (and other) extensions for common utilities like grep.

    2. Should we also stick to 80×25 character green screen terminals as well?

      Seriously, if it doesn’t need portability then why waste time and effort ensuring it.

      I’m currently writing a post-install customisation script for Fedora for someone, (don’t ask…), note: for Fedora. It doesn’t need to be portable and I’m making no effort to make it so.

      1. We are very much still stuck with 80×25 text screens, just fire up a terminal window, a faithful software emulation of a VT100, bugs and all. You can even make it green on black if you want.

  6. Is Bash a totally awesome top of the line full featured language? Nope.

    But it is very fast for several situations, both to write and to run.

    Do I personally like writing in Bash or other shell scripts? Not really. I would rather write in Python, honestly. But for something quick? Bash works. I have used it to drive full complication of a Python environment from the ground up (compile GCC through ATLAS for NumPy, all with optimizations I needed for the project). It was nice, because I could start the process, disown it in Bash, and let it keep running until completion (typically ~35 hours later).

    And they I could work on my Python project to my heart’s content.

    Long story short: bash is not bad at what it is suited for. But bash is maybe not the best at things requiring a lot of complexity.

    1. The gnu intention is to use guile (Gnu Ubiquitous Intelligent Language for Extensions) for such things. guile has close POSIX integration so you can also do all of those shell-scrript types of things with ease. LISP syntax? get over it, it’s only syntax, what matters is the power of the language beneath. With inferior-li\sp-mode in emacs, you can develop and debug faster and easier than any other IDE I’ve seen.

    1. +1

      I use bash/python/perl/awk etc. daily at work. It is good to know a variety of tools, because even though each one could do any task; there are times when one shines above the rest. I feel like bash has become an underused/abused tool; and it’s nice to see someone illuminate better use.

      Also on a side note; whatever you have to say about bash it is worlds better than bat files and powershell (although in large part probably due to it’s longevity and documentation)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.