Linux Fu: The Infinite Serial Port

Ok, the title is a bit misleading. Like most things in life, it really isn’t infinite. But I’m going to show you how you can use a very interesting Linux feature to turn one serial port from a microcontroller into a bunch of virtual ports. In theory, you could create over 200 ports, but the reality is you will probably want to stick with fewer.

The feature in question is what’s known as pseudoterminal or sometimes a pty or pts. These special files were made to feed data to programs that expect to accept data from a terminal. The files provide two faces. To the client, it looks like any other terminal device. To the creator, though, it is just another file. What you write to that file goes to the fake terminal and you can read anything that is sent from the program connected to the terminal. You use these all the time, probably, without realizing it since running a shell under X Windows, for example, doesn’t attach to a real terminal, after all.

You could, of course, do the same trick with a composite USB device, assuming you have one. Also assuming you can find a working driver and get it working. However, many microcontrollers have a serial port — even one with a USB converter built-in — but fewer have full-blown USB hardware. Even the ones that do are often at odds with strange drivers on the PC side. Serial ports work and work well even on the simplest microcontrollers.

The Plan

The plan is simple enough. A Linux program listens to a real serial port and watches for special character sequences in the data stream. Those sequences will allow you to switch data so that the data stream will go to a particular terminal. Data coming back from the terminals will go to the real serial port after sending a sequence that identifies its source.

Continue reading “Linux Fu: The Infinite Serial Port”

Linux Fu: An Odd Use For Fork()

If you are a Star Trek fan, you’ll probably remember the phrase “You have to learn why things work on a starship.” The truth is, in most episodes, knowing how to override another ship’s console or make gunpowder didn’t come in very handy, but boy when it did, it really saved the day. Linux is a lot like that. There are a few things you probably don’t need to know very often, but when you do need to know, it makes a huge difference. In this particular post, I want to look at an odd use of the fork system call. For many purposes, you’ll never need to know this particular irregular use. But when you need it, you are really going to need it.

This is actually based on an old client of mine who used Unix to run a massive and very critical report every day.  The report had a lot of math since they were trying to optimize something and then generate a lot of reports. In those days, the output of the report was on old green-bar paper on a line printer. The problem was that the report took something like 14 hours to run including the printouts. If someone discovered something wrong, there was no time to run the report again because the next day’s report would have to start before the second run would finish.

The client had a bunch of Windows programmers and — at that time — there wasn’t anything really analogous to a real fork call in Windows. I looked at the code and realized that probably most of the code was spending time waiting to print the output. The computer had multiple CPUs and there were multiple printers, but that one program was hanging on the one printer. There was a lot of data, so writing it to a database and then running different reports against it wasn’t a great option. The answer was to use the power of fork. With a change in the code that took less than 30 minutes, the report ran in five hours. They were very pleased.

So how did I do it? The answer lies in how fork works. Just about every time you see a fork, you see some sort of exec call to start a new program. So if you think about fork at all, you probably think it is part of how you start a new program and, most of the time, that’s true. Continue reading “Linux Fu: An Odd Use For Fork()”

Linux Fu: Simple Pipes

In the old days, you had a computer and it did one thing at a time. Literally. You would load your cards or punch tape or whatever and push a button. The computer would read your program, execute it, and spit out the results. Then it would go back to sleep until you fed it some more input.

The problem is computers — especially then — were expensive. And for a typical program, the computer is spending a lot of time waiting for things like the next punched card to show up or the magnetic tape to get to the right position. In those cases, the computer was figuratively tapping its foot waiting for the next event.

Someone smart realized that the computer could be working on something else while it was waiting, so you should feed more than one program in at a time. When program A is waiting for some I/O operation, program B could make some progress. Of course, if program A didn’t do any I/O then program B starved, so we invented preemptive multitasking. In that scheme, program A runs until it can’t run anymore or until a preset time limit occurs, whichever comes first. If time expires, the program is forced to sleep a bit so program B (and other programs) get their turn. This is how virtually all modern computers outside of tiny embedded systems work.

But there is a difference. Most computers now have multiple CPUs and special ways to quickly switch tasks. The desktop I’m writing this on has 12 CPUs and each one can act like two CPUs. So the computer can run up to 12 programs at one time and have 12 more that can replace any of the active 12 very quickly. Of course, the operating system can also flip programs on and off that stack of 24, so you can run a lot more than that, but the switch between the main 12 and the backup 12 is extremely fast.

So the case is stronger than ever for writing your solution using more than one program. There are a lot of benefits. For example, I once took over a program that did a lot of calculations and then spent hours printing out results. I spun off the printing to separate jobs on different printers and cut like 80% of the run time — which was nearly a day when I got started. But even outside of performance, process isolation is like the ultimate encapsulation. Things you do in program A shouldn’t be able to affect program B. Just like we isolate code in modules and objects, we can go further and isolate them in processes.

Doubled-Edged Sword

But that’s also a problem. Presumably, if you want to have two programs cooperate, they need to affect each other in some way. You could just use a file to talk between them but that’s notoriously inefficient. So operating systems like Linux provide IPC — interprocess communications. Just like you make some parts of an object public, you can expose certain things in your program to other programs.

Continue reading “Linux Fu: Simple Pipes”

Linux Fu: Fusing Hackaday

Unix and, by extension, Linux, has a mantra to make everything possible look like a file. Files, of course, look like files. But also devices, network sockets, and even system information show up as things that appear to be files. There are plenty of advantages to doing that since you can use all the nice tools like grep and find to work with files. However, making your own programs expose a filesystem can be hard. Filesystem code traditionally works at the kernel module level, where mistakes can wipe out lots of things and debugging is difficult. However, there is FUSE — the file system in user space library — that allows you to write more or less ordinary code and expose anything you want as a file system. You’ve probably seen FUSE used to mount, say, remote drives via ssh or Dropbox. We’ve even looked at FUSE before, even for Windows.

What’s missing, naturally, is the Hackaday RSS feed, mountable as a normal file. And that’s what we’re building today.

Writing a FUSE filesystem isn’t that hard, but there are a lot of tedious jobs. You essentially have to provide callbacks that FUSE uses to do things when the operating system asks for them. Open a file, read a file, list a directory, etc. The problem is that for some simple projects, you don’t care about half of these things, but you still have to provide them.

Luckily, there are libraries that can make it a lot easier. I’m going to show you a simple C++ program that can mount your favorite RSS feed (assuming your favorite one is Hackaday, of course) as a file system. Granted, that’s not amazing, but it is kind of neat to be able to grep through the front page stories from the command line or view the last few articles using Dolphin. Continue reading “Linux Fu: Fusing Hackaday”

Linux Fu: Bash Strings

If you are a traditional programmer, using bash for scripting may seem limiting sometimes, but for certain tasks, bash can be very productive. It turns out, some of the limits of bash are really limits of older shells and people code to that to be compatible. Still other perceived issues are because some of the advanced functions in bash are arcane or confusing.

Strings are a good example. You don’t think of bash as a string manipulation language, but it has many powerful ways to handle strings. In fact, it may have too many ways, since the functionality winds up in more than one place. Of course, you can also call out to programs, and sometimes it is just easier to make a call to an awk or Python script to do the heavy lifting.

But let’s stick with bash-isms for handling strings. Obviously, you can put a string in an environment variable and pull it back out. I am going to assume you know how string interpolation and quoting works. In other words, this should make sense:

echo "Your path is $PATH and the current directory is ${PWD}"

The Long and the Short

Suppose you want to know the length of a string. That’s a pretty basic string operation. In bash, you can write ${#var} to find the length of $var:


#/bin/bash
echo -n "Project Name? "
read PNAME
if (( ${#PNAME} > 16 ))
then
   echo Error: Project name longer than 16 characters
else
   echo ${PNAME} it is!
fi

Continue reading “Linux Fu: Bash Strings”

Linux Fu: Don’t Share Well With Others

In kindergarten, you learn that you should share. But for computer security, sharing is often a bad thing. The Linux kernel introduced the concept of namespaces starting with version 2.6.24. That’s been a few years ago, but namespaces are not used by many even though the tools exist to manipulate them. Granted, you don’t always need namespaces, but it is one of those things that when you do need it, the capability is priceless. In a nutshell, namespaces let you give a process its own private resources and — more importantly — prevents a process from seeing resources in other namespaces.

Turns out, you use namespaces all the time because every process you run lives in some set of namespaces. I say set, because there are a number of namespaces for different resources. For example, you can set a different network namespace to give a process its own set of networking items including routing tables, firewall rules, and everything else network-related.

So let’s have a look at how Linux doesn’t share names.

Continue reading “Linux Fu: Don’t Share Well With Others”

Linux Fu: The Ultimate Dual Boot Laptop?

I must confess, that I try not to run Windows any more than absolutely necessary. But for many reasons, it is occasionally necessary. In particular, I have had several laptops that are finicky with Linux. I still usually dual boot them, but I often leave Windows on them for one reason or another. I recently bought a new Dell Inspiron and the process of dual booting it turned out to be unusually effective but did bring up a few challenges.

If you ever wanted a proper dual-booting laptop, you’ll be interested in how this setup works. Sure, you can always repartition the drive, but the laptop has a relatively small drive and is set up very specifically to work with the BIOS diagnostics and recovery so it is always a pain to redo the drive without upsetting the factory tools.

Since the laptop came with a 512 GB NVMe drive, I wanted to upgrade the drive anyway. So one option would have been to put a bigger drive in and then go the normal route. That was actually my intention, but I wound up going a different way.

Continue reading “Linux Fu: The Ultimate Dual Boot Laptop?”