Linux Fu: Walk, Chew Gum

August 16, 2021 by Al Williams 7 Comments

If you ever think about it, computers are exceedingly stupid. Even the most powerful CPU can’t do very much. However, it can do what it does very rapidly and repeatably. Computers are so fast, they can appear to do a lot of things at once, too and modern computers have multiple CPUs to further enhance their multitasking abilities. However, we often don’t write programs or shell scripts to take advantage of this. However, there’s no reason for this, as you’ll see.

Bash Support

It is surprisingly easy to get multiple processes running under Bash. For one thing, processes on either side of a pipe run together, and that’s probably the most common way shell scripts using multiprogramming. In other words, think about what happens if you write ls | more.

Under the old MSDOS system, the first program would run to completion, spooling its output to a temporary file. Then the second program will run, reading input from the same file. With Linux and most other modern operating systems, both programs will run together with the input of the second program connected to the first program’s output. Continue reading “Linux Fu: Walk, Chew Gum” →

Linux Fu: Superpowers For Mere Mortals

July 26, 2021 by Al Williams 27 Comments

You can hardly mention the sudo command without recalling the hilarious XKCD strip about making sandwiches. It does seem like sudo is the magic power to make a Linux system do what you want. The only problem is that those superpowers are not something to be taken lightly.

If you are surfing the web, for example, you really don’t want to be root, because if someone naughty takes over your computer they could do a lot more harm with your root password. But still, there are times when you want to run certain commands that are normally root-only and don’t want to bother with a password. Luckily, sudo can handle that use case very easily.

Why?

As a simple example, suppose you like to shut your computer down at the end of the day. You run the shutdown command from the terminal but it doesn’t work because you aren’t root. You then have to do it again with sudo and if you haven’t logged in lately, provide your password. Ugh.

Continue reading “Linux Fu: Superpowers For Mere Mortals” →

Linux Fu: PDF For Penguins

July 6, 2021 by Al Williams 44 Comments

PostScript started out as a programming language for printers. While PostScript printers are still a thing, there are many other ways to send data to a printer. But PostScript also spawned the Portable Document Format or PDF and that has been crazy successful. Hardly a day goes by that you don’t see some kind of PDF document come across your computer screen. Sure, there are other competing formats but they hold a sliver of market share compared to PDF. Viewing PDFs under Linux is no problem. But what about editing them? Turns out, that’s easy, too, if you know how.

GUI Tools

You can use lots of tools to edit PDF files, but the trick is how good the results will look. Anything will work for this: LibreOffice Draw, Inkscape, or even GIMP. If all you want to do is remove something with a white box or make an annotation, these tools are usually great, but for more complicated changes, or pixel-perfect output, they may not be the right tool.

The biggest problem is that most of these tools deal with the PDF as an image or, at least, a collection of objects. For example, columns of text will probably turn into a collection of discrete lines. Changing something that causes a line to wrap will require you to change all the other lines to match. Sometimes text isn’t even text at all, but images. It largely depends on how the creator made the PDF to begin with. Continue reading “Linux Fu: PDF For Penguins” →

Linux Fu: Databases Are Next-Level File Systems

June 8, 2021 by Al Williams 63 Comments

It is funny how exotic computer technology eventually either fails or becomes commonplace. At one time, having more than one user on a computer at once was high tech, for example. Then there are things that didn’t catch on widely like vector display or content-addressable memory. The use of mass storage — especially disk drives — in computers, though has become very widespread. But at one time it was an exotic technique and wasn’t nearly as simple as it is today.

However, I’m surprised that the filesystem as we know it hasn’t changed much over the years. Sure, compared to, say, the 1960s we have a lot better functionality. And we have lots of improvements surrounding speed, encoding, encryption, compression, and so on. But the fundamental nature of how we store and access files in computer programs is stagnant. But it doesn’t have to be. We know of better ways to organize data, but for some reason, most of us don’t use them in our programs. Turns out, though, it is reasonably simple and I’m going to show you how with a toy application that might be the start of a database for the electronic components in my lab.

You could store a database like this in a comma-delimited file or using something like JSON. But I’m going to use a full-featured SQLite database to avoid having a heavy-weight database server and all the pain that entails. Is it going to replace the database behind the airline reservation system? No. But will it work for most of what you are likely to do? You bet. Continue reading “Linux Fu: Databases Are Next-Level File Systems” →

Linux Fu: Mixing Bash And Python

May 4, 2021 by Al Williams 24 Comments

Although bash scripts are regularly maligned, they do have a certain simplicity and ease of creation that makes them hard to resist. But sometimes you really need to do some heavy lifting in another language. I’ll talk about Python, but actually, you can use many different languages with this technique, although you might need a little adaptation, depending on your language of choice.

Of course, you don’t have to do anything special to call another program from a bash script. After all, that’s what it’s mainly used for: calling other programs. However, it isn’t very handy to have your script spread out over multiple files. They can get out of sync and if you want to send it to someone or another machine, you have to remember what to get. It is nicer to have everything in one file.

Continue reading “Linux Fu: Mixing Bash And Python” →

Linux Fu: A Little Bit Of (Network) History Repeating Itself

April 26, 2021 by Al Williams 5 Comments

These days, embedded systems often have networks and that can make them significantly more complex. Networks are usually pretty nondeterministic and there are a variety of oddball conditions. For example, when your public-access pick and place machine gets written up on Hackaday and you suddenly get a 50X surge in traffic, how does your network stack handle it? While there’s no silver bullet for network testing, there are some tricks that can make it easier and one of those is the tcpreplay utilities that allow you to record complex network traffic and then play it back in a variety of ways. This has many benefits, especially if you manage to capture that one thing that triggers bad behavior sporadically. Being able to play it back on demand can speed up diagnostics considerably.

General Idea

You probably know that tcpdump allows you to grab packet captures from a network interface and save them to a file. If you prefer a GUI, you probably use Wireshark, which uses the same underlying library (libpcap) to grab the data. In fact, you can capture data using tcpdump and look at it with Wireshark, although there are other tools like tcptrace or Ngrep that can work with the output, also.

While the output of the command can be a little cryptic without tool support, a program called tcpreplay can take that data and feed it back in a variety of ways. Of course, you can modify the file first — there are tools to make that easier and — if you need to — you can craft your own network traffic by hand or using one of a variety of tools. This process is often called “packet crafting.”

Continue reading “Linux Fu: A Little Bit Of (Network) History Repeating Itself” →

Linux Fu: Shell Script File Embedding

April 9, 2021 by Al Williams 25 Comments

You need to package up a bunch of files, send them somewhere, and do something with them at the destination. It isn’t an uncommon scenario. The obvious answer is to create an archive — a zip or tar file, maybe — and include a shell script that you have to tell the user to run after unpacking.

That may be obvious, but it assumes a lot on the part of the remote user. They need to know how to unpack the file and they also need to know to run your magic script of commands after the unpack. However, you can easily create a shell script that contains a file — even an archive of many files — and then retrieve the file and act on it at run time. This is much simpler from the remote user’s point of view. You get one file, you execute it, and you are done.

In theory, this isn’t that hard to do, but there are a lot of details. Shell scripts are not compiled — at least, not typically — so the shell only reads what it needs to do the work. That means if your script is careful to exit, you can add as much “garbage” to the end of it as you like. The shell will never look at it, so it’s possible to store the payload there.

Continue reading “Linux Fu: Shell Script File Embedding” →