Linux Fu: A Little Bit Of (Network) History Repeating Itself

April 26, 2021 by Al Williams 5 Comments

These days, embedded systems often have networks and that can make them significantly more complex. Networks are usually pretty nondeterministic and there are a variety of oddball conditions. For example, when your public-access pick and place machine gets written up on Hackaday and you suddenly get a 50X surge in traffic, how does your network stack handle it? While there’s no silver bullet for network testing, there are some tricks that can make it easier and one of those is the tcpreplay utilities that allow you to record complex network traffic and then play it back in a variety of ways. This has many benefits, especially if you manage to capture that one thing that triggers bad behavior sporadically. Being able to play it back on demand can speed up diagnostics considerably.

General Idea

You probably know that tcpdump allows you to grab packet captures from a network interface and save them to a file. If you prefer a GUI, you probably use Wireshark, which uses the same underlying library (libpcap) to grab the data. In fact, you can capture data using tcpdump and look at it with Wireshark, although there are other tools like tcptrace or Ngrep that can work with the output, also.

While the output of the command can be a little cryptic without tool support, a program called tcpreplay can take that data and feed it back in a variety of ways. Of course, you can modify the file first — there are tools to make that easier and — if you need to — you can craft your own network traffic by hand or using one of a variety of tools. This process is often called “packet crafting.”

Continue reading “Linux Fu: A Little Bit Of (Network) History Repeating Itself” →

Linux Fu: Shell Script File Embedding

April 9, 2021 by Al Williams 25 Comments

You need to package up a bunch of files, send them somewhere, and do something with them at the destination. It isn’t an uncommon scenario. The obvious answer is to create an archive — a zip or tar file, maybe — and include a shell script that you have to tell the user to run after unpacking.

That may be obvious, but it assumes a lot on the part of the remote user. They need to know how to unpack the file and they also need to know to run your magic script of commands after the unpack. However, you can easily create a shell script that contains a file — even an archive of many files — and then retrieve the file and act on it at run time. This is much simpler from the remote user’s point of view. You get one file, you execute it, and you are done.

In theory, this isn’t that hard to do, but there are a lot of details. Shell scripts are not compiled — at least, not typically — so the shell only reads what it needs to do the work. That means if your script is careful to exit, you can add as much “garbage” to the end of it as you like. The shell will never look at it, so it’s possible to store the payload there.

Continue reading “Linux Fu: Shell Script File Embedding” →

Linux Fu: VPN For Free With SSH

November 23, 2020 by Al Williams 44 Comments

If you see a lot of banner ads on certain websites, you know that without a Virtual Private Network (VPN), hackers will quickly ravage your computer and burn down your house. Well, that seems to be what they imply. In reality, though, there are two main reasons you might want a VPN connection. You can pay for a service, of course, but if you have ssh access to a computer somewhere on the public Internet, you can set up your own VPN service for no additional cost.

The basic idea is that you connect to a remote computer on another network and it makes it look like all your network traffic is local to that network. The first case for this is to sidestep or enhance security. For example, you might want to print to a network printer without exposing that printer to the public Internet. While you are at the coffee shop you can VPN to your network and print just like you were a meter away from the printer at your desk. Your traffic on the shop’s WiFi will also be encrypted.

The second reason is to hide your location from snooping. For example, if you like watching the BBC videos but you live in Ecuador, you might want to VPN to a network in the UK so the videos are not blocked. If your local authorities monitor and censor your Internet, you might also want your traffic coming from somewhere else.

Continue reading “Linux Fu: VPN For Free With SSH” →

Linux Fu: Monitor Disks

November 5, 2020 by Al Williams 7 Comments

If you want a quick view of a Linux system’s process load, you can use top or — slightly nicer — htop. But what if you want a quick snapshot of how the disk system is doing? There are a few tools you can use, some of which are not nearly as common as top.

First, iotop

Most similar to top is iotop. This program shows you the total and current disk read and write numbers for the file system and also shows you who is eating up the most disk I/O. This screen looks busy:

Continue reading “Linux Fu: Monitor Disks” →

Linux Fu: Troubleshooting Incron

October 28, 2020 by Al Williams 18 Comments

You probably know about cron, a program that lets you schedule programs to run at various times. We’ve also talked about incron, which is very similar but instead of time, it reacts to changes in the file system. If you ever wanted to write a program that, say, detects a change in a file and automatically uploads it to a programmer, backs it up, e-mails it somewhere, or anything else, then incron might be for you. Although we’ve talked about it before, incron has some peculiarities that make it very difficult to debug problems, so I thought I’d share some of the tricks I use when working with incron.

I was thinking about this because I wanted to set up a simple system where I have a single document directory under git control. Changing a markdown file in that folder would generate Word document and PDF equivalents. Conversely, changing a Word document would produce a markdown version.

This is easy to do with pandoc — it speaks many different formats. The trick is running it only on changed files and as soon as they change. The task isn’t that hard, but it does take a bit to debug since it’s a bit nontrivial.

Continue reading “Linux Fu: Troubleshooting Incron” →

Linux-Fu: Making AWK A Bit Easier

September 29, 2020 by Al Williams 18 Comments

awk is a kind of Swiss Army knife for text files. However, some of its limitations are often a bit annoying. I’ve used a simple set of functions to make awk a bit better, although I will warn you: it does require GNU extensions to awk. That is, you must use gawk and not other versions. Your system probably maps /usr/bin/awk to something and that something might be gawk. But it could also be mawk or some other flavor. If you use a Debian-based distro, update-alternatives is your friend here. But for the purposes of this post, I’m going to assume you are using gawk.

By the end of the post, you’ll see how to use my awk add-on functions to split up a line into fields even when there is no single character to separate all fields. In addition, you’ll be able to refer to the fields using names you decide. You won’t have to remember that $2 is the time field. You’ll say Fields_fields["time"] instead.

The Problem

awk does a lot of common work for you when you use it to process text files. It reads files a record at a time. Normally, a record is a single line. Then it splits the line on fields using whitespace, or some other choice of field separators. You can write code that manipulates the line or individual fields. This default behavior is great, especially since you can change the end of record character and the field separator. A surprising number of files fit this sort of format.

Until, of course, they don’t. If you have data coming from a data logging instrument or some database, it could be formatted in a variety of ways. Some fields might have structured data with a variety of separators. This isn’t a deal-breaker. Since you can get at the whole line, you can do almost anything you want, but the logic is harder and the whole point to using awk is to make things easier.

For example, suppose you had a file from a data recorder that had an eight-digit serial number, followed by a six-character tag, and then two floating point numbers separated by colons. The pattern might look like

^([0-9]{8})([a-zA-Z0-9]{6})([-+.0-9]+),([-+.0-9]+)$

This would be hard to handle with the conventional field splitting and you’d normally just write code to split everything apart.

Continue reading “Linux-Fu: Making AWK A Bit Easier” →

Linux Fu: Simple SSH File Sharing

September 21, 2020 by Al Williams 28 Comments

If you have more than one Linux computer, you probably use ssh all the time. It is a great tool, but I’ve always found one thing about it strange. Despite having file transfer capabilities in the form of scp and sftp, there is no way to move a file back or forth between the local and remote hosts without starting a new program on the local machine or logging in from the remote machine back to the local machine.

That last bit is a real problem since you often access a server from behind a firewall or a NAT router with an ephemeral IP address, so it can’t reconnect to you anyway. It would be nice to hit the escape character, select a local or remote file, and teleport it across the interface, all from inside a single ssh session.

I didn’t quite get to that goal, but I did get pretty close. I’ll show you a script that can automatically mount a remote directory on the local machine. You’ll need sshfs on the local machine, but no changes on the remote machine where you may not be able to install software. With a little more work, and if your client has an ssh server running, you can mount a local directory on the remote machine, too. You won’t need to worry about your IP address or port blocking. If you can log into the remote machine, you are good.

Combined, this got me me very close to my goal. I can be working in a shell on either side and have access to read or write files on the other side. I just have to set it up carefully. Continue reading “Linux Fu: Simple SSH File Sharing” →