Linux Fu: Shell Script File Embedding

You need to package up a bunch of files, send them somewhere, and do something with them at the destination. It isn’t an uncommon scenario. The obvious answer is to create an archive — a zip or tar file, maybe — and include a shell script that you have to tell the user to run after unpacking.

That may be obvious, but it assumes a lot on the part of the remote user. They need to know how to unpack the file and they also need to know to run your magic script of commands after the unpack. However, you can easily create a shell script that contains a file — even an archive of many files — and then retrieve the file and act on it at run time. This is much simpler from the remote user’s point of view. You get one file, you execute it, and you are done.

In theory, this isn’t that hard to do, but there are a lot of details. Shell scripts are not compiled — at least, not typically — so the shell only reads what it needs to do the work. That means if your script is careful to exit, you can add as much “garbage” to the end of it as you like. The shell will never look at it, so it’s possible to store the payload there.

Continue reading “Linux Fu: Shell Script File Embedding”

Linux Fu: Serial Untethered

Serial ports used to be everywhere. In a way, they still are since many things that appear to plug in as a USB device actually look like a serial port. The problem is that today, the world runs on the network. Sure, you can buy a terminal server that converts a serial port to an Ethernet port, but what fun is that? In this article, I’m going to show you how to stream serial ports over the network using some available Linux tools. It isn’t perfect, and it won’t work for every case, but when it works it works well.

Everything is a File, Until it Isn’t

At some point in the past, Unix — the progenitor of Linux — treated virtually everything as a file, and all files were created more or less equal. Programs didn’t care if a file was local, on the network, from a tape drive, or arriving over a named pipe.

But things started to change. Even though a serial port is just a file under Linux, it has some special attributes that let you set, for example, baud rates. Worse, some programs “know” too much about files and insist on certain naming conventions. So, in theory, you should be able to create a network socket, connect one end to a serial port and the other end to a program, and be done with it. In theory.

Continue reading “Linux Fu: Serial Untethered”

Linux Fu: VPN For Free With SSH

If you see a lot of banner ads on certain websites, you know that without a Virtual Private Network (VPN), hackers will quickly ravage your computer and burn down your house. Well, that seems to be what they imply. In reality, though, there are two main reasons you might want a VPN connection. You can pay for a service, of course, but if you have ssh access to a computer somewhere on the public Internet, you can set up your own VPN service for no additional cost.

The basic idea is that you connect to a remote computer on another network and it makes it look like all your network traffic is local to that network. The first case for this is to sidestep or enhance security. For example, you might want to print to a network printer without exposing that printer to the public Internet. While you are at the coffee shop you can VPN to your network and print just like you were a meter away from the printer at your desk. Your traffic on the shop’s WiFi will also be encrypted.

The second reason is to hide your location from snooping. For example, if you like watching the BBC videos but you live in Ecuador, you might want to VPN to a network in the UK so the videos are not blocked. If your local authorities monitor and censor your Internet, you might also want your traffic coming from somewhere else.

Continue reading “Linux Fu: VPN For Free With SSH”

Linux Fu: Send In The (Cloud) Clones

Storing data “in the cloud” — even if it is your own server — is all the rage. But many cloud solutions require you to access your files in a clumsy way using a web browser. One day, operating systems will incorporate generic cloud storage just like any other file system. But by using two tools, rclone and sshfs, you can nearly accomplish this today with a little one-time setup. There are a few limitations, but, generally, it works quite well.

It is a story as old as computing. There’s something new. Using it is exotic and requires special techniques. Then it becomes just another part of the operating system. If you go back far enough, programmers had to pull specific records from mass storage like tapes, drums, or disks and deblock data. Now you just open a file or a database. Cameras, printers, audio, and even networking once were special devices that are now commonplace. If you use Windows, for example, OneDrive is well-supported. But if you use another service, you may or may not have an easy option to just access your files as a first-class file system.

The rclone program is the Swiss Army knife of cloud storage services. Despite its name, it doesn’t have to synchronize a local file store to a remote service, although it can do that. The program works with a dizzying array of cloud storage providers and it can do simple operations like listing and copying files. It can also synchronize, as you’d expect. However, it also has an experimental FUSE filesystem that lets you mount a remote service — with varying degrees of success.

What’s Supported?

If you don’t like using someone like Google or Amazon, you can host your own cloud. In that case, you can probably use sshfs to mount a file using ssh, although rclone can also do that. There are also cloud services you can self-host like OwnCloud and NextCloud. A Raspberry Pi running Docker can easily stand up one of these in a few minutes and rclone can handle these, too.

Continue reading “Linux Fu: Send In The (Cloud) Clones”

Linux Fu: Monitor Disks

If you want a quick view of a Linux system’s process load, you can use top or — slightly nicer — htop. But what if you want a quick snapshot of how the disk system is doing? There are a few tools you can use, some of which are not nearly as common as top.

First, iotop

Most similar to top is iotop. This program shows you the total and current disk read and write numbers for the file system and also shows you who is eating up the most disk I/O.  This screen looks busy:

Continue reading “Linux Fu: Monitor Disks”

Linux Fu: Troubleshooting Incron

You probably know about cron, a program that lets you schedule programs to run at various times. We’ve also talked about incron, which is very similar but instead of time, it reacts to changes in the file system. If you ever wanted to write a program that, say, detects a change in a file and automatically uploads it to a programmer, backs it up, e-mails it somewhere, or anything else, then incron might be for you. Although we’ve talked about it before, incron has some peculiarities that make it very difficult to debug problems, so I thought I’d share some of the tricks I use when working with incron.

I was thinking about this because I wanted to set up a simple system where I have a single document directory under git control. Changing a markdown file in that folder would generate Word document and PDF equivalents. Conversely, changing a Word document would produce a markdown version.

This is easy to do with pandoc — it speaks many different formats. The trick is running it only on changed files and as soon as they change. The task isn’t that hard, but it does take a bit to debug since it’s a bit nontrivial.

Continue reading “Linux Fu: Troubleshooting Incron”

Linux Fu: Global Search And Replace With Ripgrep

If you are even a casual Linux user, you probably know how to use grep. Even if you aren’t a regular expression guru, it is easy to use grep to search for lines in a file that match anything from simple strings to complex patterns. Of course, grep is fine for looking, but what if you want to find things and change them. Maybe you want to change each instance of “HackADay” to “Hackaday,” for example. You might use sed, but it is somewhat hard to use. You could use awk, but as a general-purpose language, it seems a bit of overkill for such a simple and common task. That’s the idea behind ripgrep which actually has the command name rg. Using rg, you can do things that grep can do using more modern regular expressions and also do replacements.

A Note on Installing Ripgrep

Your best bet is to get ripgrep from your repositories. When I tried running KDE Neon, it helpfully told me that I could install a version using apt or take a Snap version that was newer. I usually hate installing a snap, but I did anyway. It informed me that I had to add –classic to the install line because ripgrep could affect files outside the Snap sandbox. Since the whole purpose of the program is to change files, I didn’t think that was too surprising, so I did the install.

Continue reading “Linux Fu: Global Search And Replace With Ripgrep”