shell scripting

7 Articles

Pnut: A Self-Compiling C Transpiler Targeting Human-Readable POSIX Shell

July 25, 2024 by 16 Comments

Shell scripting is one of those skills that are absolutely invaluable on especially UNIX and BSD-based systems like the BSDs, the two zillion Linux distributions as well as MacOS. Yet not every shell is the same, and not everybody can be bothered to learn the differences between the sh, bash, ksh, zsh, dash, fish and other shells, which can make a project like Pnut seem rather tempting. Rather than dealing with shell scripting directly, the user writes their code in the Lingua Franca of computing, AKA C, which is then transpiled into a shell script that should run in any POSIX-compliant shell.

The transpiler can be used both online via the main Pnut website, as well as locally using the (BSD 2-clause) open source code on GitHub. Here the main limitations are also listed, which mostly concern the C constructs that do not map nicely to a POSIX shell. These are: no support for floating point numbers and unsigned integers, no goto and switch nor taking the address of a variable with &. These and preprocessor-related limitations and issues are largely to be expected, as especially POSIX shells are hardly direct replacements for full-blown C code.

As a self-professed research project, Pnut seems like an interesting project, although if you are writing shell scripts for anything important, you probably just want to buckle down and learn the ins and outs of POSIX shell scripting and beyond. Although it’s a bit of a learning curve, we’d be remiss if we said that it’s not totally worth it, if only because it makes overall shell usage even beyond scripting so much better.

Linux Fu: Bash Strings

January 26, 2022 by 40 Comments

If you are a traditional programmer, using bash for scripting may seem limiting sometimes, but for certain tasks, bash can be very productive. It turns out, some of the limits of bash are really limits of older shells and people code to that to be compatible. Still other perceived issues are because some of the advanced functions in bash are arcane or confusing.

Strings are a good example. You don’t think of bash as a string manipulation language, but it has many powerful ways to handle strings. In fact, it may have too many ways, since the functionality winds up in more than one place. Of course, you can also call out to programs, and sometimes it is just easier to make a call to an awk or Python script to do the heavy lifting.

But let’s stick with bash-isms for handling strings. Obviously, you can put a string in an environment variable and pull it back out. I am going to assume you know how string interpolation and quoting works. In other words, this should make sense:

echo "Your path is $PATH and the current directory is ${PWD}"

The Long and the Short

Suppose you want to know the length of a string. That’s a pretty basic string operation. In bash, you can write ${#var} to find the length of $var:


#/bin/bash
echo -n "Project Name? "
read PNAME
if (( ${#PNAME} > 16 ))
then
   echo Error: Project name longer than 16 characters
else
   echo ${PNAME} it is!
fi

Continue reading “Linux Fu: Bash Strings” →

Linux Fu: Shell Script File Embedding

April 9, 2021 by 25 Comments

You need to package up a bunch of files, send them somewhere, and do something with them at the destination. It isn’t an uncommon scenario. The obvious answer is to create an archive — a zip or tar file, maybe — and include a shell script that you have to tell the user to run after unpacking.

That may be obvious, but it assumes a lot on the part of the remote user. They need to know how to unpack the file and they also need to know to run your magic script of commands after the unpack. However, you can easily create a shell script that contains a file — even an archive of many files — and then retrieve the file and act on it at run time. This is much simpler from the remote user’s point of view. You get one file, you execute it, and you are done.

In theory, this isn’t that hard to do, but there are a lot of details. Shell scripts are not compiled — at least, not typically — so the shell only reads what it needs to do the work. That means if your script is careful to exit, you can add as much “garbage” to the end of it as you like. The shell will never look at it, so it’s possible to store the payload there.

Continue reading “Linux Fu: Shell Script File Embedding” →

Debugging For Sed — No Kidding

April 24, 2020 by 13 Comments

If you do much Linux shell scripting, you’ve probably encountered sed — the stream editor — in an example. Maybe you’ve even used it yourself. If all you want to do is substitute text, it is easy and efficient. But if you try to do really elaborate editing, it is often difficult to get things right. The syntax is cryptic and the documentation is lacking. But thanks to [SoptikHa2] you can now debug sed scripts with a text-based GUI debugger. Seriously.

According to the author, the program has several notable features:

  • Preview variable values, both of them!
  • See how will a substitute command affect pattern space before it runs
  • Step through sed script – both forward and backward!
  • Place breakpoints and examine program state
  • Hot reload and see what changes as you edit source code
  • Its name is a palindrome

There’s only one word for that last feature: wow.

Continue reading “Debugging For Sed — No Kidding” →

Linux Fu: Leaning Down With Exec

December 30, 2019 by 11 Comments

Shell scripting is handy and with a shell like bash it is very capable, too. However, shell scripting isn’t always very efficient. Think about it. If you run grep or tr or sort to do some operation in a shell script, you are spawning a whole new process. That takes time and resources. But there are some answers to reducing — but not eliminating — the problem.

Have you ever written a program like this (in any language, but I’ll use C):

int foo(void)
{
  ...
  bar();

}

You hope the compiler doesn’t write assembly code like this:

_foo: 
....

      call _bar
      ret

Most optimizers should pick up on the fact that you can convert a call like this to a jump and let the ret statement in _bar return to foo’s caller. However, shell scripts are not that smart. If you have a shell script called MungeData and it calls another program or shell script called PostProcess on its last line, then you will have at one time three processes in play: your original shell, the shell running MungeData, and either the PostProcess program or a shell running the script. Not to mention, the processes to do things inside post process. So what do you do?

Continue reading “Linux Fu: Leaning Down With Exec” →

Script Your Way Out Of Video Editing Drudgery

November 26, 2016 by 12 Comments

[Victor Frost] has a deep voice and a fancy top of the line camera. While one would assume this to be a more than generous situation for life to put a person in; it’s got its own set of problems. Mainly that his fantastic fancy camera uses the most modern version of the popular h.264 encoding scheme, h.265. Gasp!

While that too seems like a pro, unfortunately h.265 doesn’t play as nice with his editing software. The solution seems easy, just transcode it and get on your way. However, when you start talking about transcoding 4K video from a top-of-the line source and retaining the quality. Well… It can bring a processor to its knees. Since he’d rather be playing overwatch than transcoding video on his main computer, he decided to offload and automate the drudgery to his spare.

That’s how the Ingest-a-Tron 9000 came into play. It uses a lot of open source software and, yes, windows batch files to take the files off his camera, process it on one computer, and dump it to another. Now he can game (or edit) while he waits. For those of us who are estranged from Linux thanks to our favorite software, it’s good to know that there are still ways to automate away the pain. Video after the break.

Continue reading “Script Your Way Out Of Video Editing Drudgery” →

Shell Game

August 30, 2016 by 56 Comments

A lot of us spend a lot of time switching between Windows and Linux. Now that platforms like the Raspberry Pi are popular, that number is probably increasing every day. While I run Linux on nearly everything I own (with the exception of a laptop), my work computers mostly run Windows. The laptop is on Windows, too, because I got tired of trying to get all the fancy rotation sensors and pen features working properly under Linux.

What I hate most about Windows is how hard is it to see what’s going on under the hood. My HP laptop works with a cheap Dell active stylus. Sort of. It is great except around the screen edges where it goes wild. Calibration never works. On Linux, I could drill down to the lowest levels of the OS if I were so inclined. With Windows, it is just tough.

War is Shell

One place where Linux always used to have an advantage over DOS and Windows was the shell. There are lots of variations available under Linux, but bash seems to be the current pick for most people. If you want more power, you can move to some alternatives, but even bash is pretty powerful if you learn how to use it and have the right external programs (if you don’t believe it, check out this web server).

Continue reading “Shell Game” →