Linux Fu: Scripting for Binary Files

If you ever need to write a binary file from a traditional language like C, it isn’t all that hard to do. About the worst thing you might have to deal with is attempts to fake line endings across Windows and Linux, but there’s usually a way to turn that off if it is on by default. However, if you are using some type of scripting language, binary file support might be a bit more difficult. One answer is to use a tool like xxd or t2b (text-to-binary) to handle the details. You can find the code for t2b on GitHub including prebuilt binaries for many platforms. You should be able to install xxd from your system repository.

These tools take very different approaches. You might be familiar with tools like od or hexdump for producing readable representations of binary files. The xxd tool can actually do the same thing — although it is not as flexible. What xxd can even reverse itself so that it can rebuild a binary file from a hex dump it creates (something other tools can’t do). The t2b tool takes a much different approach. You issue commands to it that causes it to write an original hex file.

Both of these approaches have some merit. If you are editing a binary file in a scripting language, xxd makes perfect sense. You can convert the file to text, process it, and then roll it back to binary using one program. On the other hand, if you are creating a binary file from scratch, the t2b program has some advantages, too.

I decided to write a few test scripts using bash to show how it all works. These aren’t production scripts so they won’t be as hardened as they could be, but there is no reason they couldn’t be made as robust as you were willing to make them.

Continue reading “Linux Fu: Scripting for Binary Files”

Linux Fu: Watch That Filesystem

The UNIX Way™ is to cobble together different, single-purpose programs to get the effect you want, for instance in a Bash script that you run by typing its name into the command line. But sometimes you want the system to react to changes in the system without your intervention. For example, you might like to watch a directory and kick off some program automatically when a file appears from a completed FTP transaction, without having to sit there and refresh the directory yourself.

The simple but ugly way to do this just scans the directory periodically. Here’s a really dumb shell script:

#!/bin/bash
while true
 do
   for I in `ls`
    do cat $I; rm $I
   done
 sleep 10
done

Just for an example, I dump the file to the console and remove it, but in real life, you’d do something more interesting. This is really not a good script because it executes all the time and it just isn’t a very elegant solution. (If you think I should use for I in *, try doing that in an empty directory and you’ll see why I use the ls command instead.)

Continue reading “Linux Fu: Watch That Filesystem”

Linux Fu: Counter Rotate Keys!

If you’ve done anything with a modern Linux system — including most variants for the Raspberry Pi — you probably know about sudo. This typically allows an authorized user to elevate themselves to superuser status to do things.

However, there is a problem. If you have sudo access, you can do anything — at least, anything the sudoers file allows you to do. But what about extremely critical operations? We’ve all seen the movies where launching the nuclear missile requires two keys counter-rotated at the same time and third firing key. Is there an equivalent for Linux systems?

It isn’t exactly a counter-rotating key, but the sudo_pair project — a prelease open-source project from Square — gives you something similar. The project is a plugin for sudo that allows you to have another user authorize a sudo request. Not only do they authorize it, but they get to see what is happening, and even abort it if something bad is happening.

Continue reading “Linux Fu: Counter Rotate Keys!”

Linux Fu: File Aliases, Links, and Mappings

Have you heard it said that everything in Linux is a file? That is largely true, and that’s why the ability to manipulate files is crucial to mastering Linux Fu.

One thing that makes a Linux filesystem so versatile is the ability for a file to be many places at once. It boils down to keeping the file in one place but using it in another. This is handy to keep disk access snappy, to modify a running system, or merely to keep things organized in a way that suits your needs.

There are several key features that lend to this versatility: links, bind mounts, and user space file systems immediately come to mind. Let’s take a look at how these work and how you’ll often see them used.

Continue reading “Linux Fu: File Aliases, Links, and Mappings”

Linux Fu: Regular Expressions

If you consider yourself a good cook, you may or may not know how to make a souffle or baklava. But there are certain things you probably do know how to do that form the basis of many recipes. For example, you can probably boil water, crack an egg, and brown meat. With Linux or Unix systems, you can make the same observation. You might not know how to set up a Wayland server or write a kernel module. But there are certain core skills like file manipulation and editing that will serve you no matter what you do. One of the pervasive skills that often gives people trouble is regular expressions. Many programs use these as a way to specify search patterns, usually in text strings such as files.

If you aren’t comfortable with regular expressions, that’s easy to fix. They aren’t that hard to learn and there are some great tools to help you. Many tools use regular expressions and the core syntax is the same. The source of confusion is that the details beyond core syntax have variations.

Let’s look at the foundation you need to understand regular expression well.

Continue reading “Linux Fu: Regular Expressions”

Linux Fu: A Little Help for Bash

It isn’t uncommon these days for a programmer’s editor to offer you help about what you are typing, ranging from a pop up with choices to a full-blown code template. If you have written a million lines of code in the language, this might even annoy you. However, if you use it only occasionally, these can be very helpful. I’ve used Unix and Linux for many years, but I realize that there are people who don’t use it every day. With the Raspberry Pi, Linux servers, and Windows 10 having a bash shell, there are more people using a shell “every once in a while” than ever before. Could you use a little help? If so, you might try bashelp: a little something I put together while writing about bash completion.

There’s good news and bad news. The good news is that Unix has a built-in help command — man — and has for some time. The bad news is that you need to stop what you are typing and enter a man command to use it. Man, by the way, is short for manual.

There are GUI front-ends to man (like yelp, on the left) and you can even use a web browser locally or remotely. However, none of these are connected to what you are typing. You have to move to another window, enter your search term, then go back to your typing. That got me to thinking about how to get a sort of context-sensitive inline help for bash.

Continue reading “Linux Fu: A Little Help for Bash”

Linux Fu: Custom Bash Command Completion

If you aren’t a Linux user and you watch someone who knows what they are doing use Bash — the popular command line interpreter — you might get the impression they type much faster than they actually do. That’s because experienced Linux users know that pressing the tab key will tend to complete what they are typing, so you can type just a few characters and get a much longer line of text. The feature is very smart so you may not have realized it, but it knows a good bit about what you could type. For example, if you try to unzip a file, it knows the expected file name probably has a .zip extension.

How does that happen? At first, you might think, “who cares how it happens?” The problem is when you write a shell script or a program that runs on Linux, the completion gets dumb. Someone has to make Bash smart about each command line program and if you are the author then that someone is you.

Continue reading “Linux Fu: Custom Bash Command Completion”