Archives

Linux Fu: Docking Made Easy

June 21, 2022 by 20 Comments

Most computer operating systems suffer from some version of “DLL hell” — a decidedly Windows term, but the concept applies across the board. Consider doing embedded development which usually takes a few specialized tools. You write your embedded system code, ship it off, and forget about it for a few years. Then, the end-user wants a change. Too bad the compiler you used requires some library that has changed so it no longer works. Oh, and the device programmer needs an older version of the USB library. The Python build tools use Python 2 but your system has moved on. If the tools you need aren’t on the computer anymore, you may have trouble finding the install media and getting it to work. Worse still if you don’t even have the right kind of computer for it anymore.

One way to address this is to encapsulate all of your development projects in a virtual machine. Then you can save the virtual machine and it includes an operating system, all the right libraries, and basically is a snapshot of how the project was that you can reconstitute at any time and on nearly any computer.

In theory, that’s great, but it is a lot of work and a lot of storage. You need to install an operating system and all the tools. Sure, you can get an appliance image, but if you work on many projects, you will have a bunch of copies of the very same thing cluttering things up. You’ll also need to keep all those copies up-to-date if you need to update things which — granted — is sort of what you are probably trying to avoid, but sometimes you must.

Docker is a bit lighter weight than a virtual machine. You still run your system’s normal kernel, but essentially you can have a virtual environment running in an instant on top of that kernel. What’s more, Docker only stores the differences between things. So if you have ten copies of an operating system, you’ll only store it once plus small differences for each instance.

The downside is that it is a bit tough to configure. You need to map storage and set up networking, among other things. I recently ran into a project called Dock that tries to make the common cases easier so you can quickly just spin up a docker instance to do some work without any real configuration. I made a few minor changes to it and forked the project, but, for now, the origin has synced up with my fork so you can stick with the original link.

Continue reading “Linux Fu: Docking Made Easy” →

Linux Fu: Easy Widgets

June 2, 2022 by 5 Comments

Here’s a scenario. You have a microcontroller that reads a number of items — temperatures, pressures, whatever — and you want to have a display for your Linux desktop that sits on the panel and shows you the status. If you click on it, you get expanded status and can even issue some commands. Most desktops support the notion of widgets, but developing them is a real pain, right? And even if you develop one for KDE, what about the people using Gnome?

Turns out there is an easy answer and it was apparently inspired by, of all things, a tool from the Mac world. That tool was called BitBar (now XBar). That program places a widget on your menu bar that can display anything you want. You can write any kind of program you like — shell script, C, whatever. The output printed from the program controls what appears on the widget using a simple markup-like language.

That’s fine for the Mac, but what about Linux? If you use Gnome, there is a very similar project called Argos. It is largely compatible with XBar, although there are a few things that it adds that are specific to it. If you use KDE (like I do) then you’ll want Kargos, which is more or less a port of Argos and adds a few things of its own.

Good News, Bad News

The good news is that, in theory, you could write a script that would run under all three systems. The bad news is that each has its own differences and quirks. Obviously, too, if you use a complied program that could pose a problem on the Mac unless you recompile.

Continue reading “Linux Fu: Easy Widgets” →

Linux Fu: The Infinite Serial Port

May 3, 2022 by 23 Comments

Ok, the title is a bit misleading. Like most things in life, it really isn’t infinite. But I’m going to show you how you can use a very interesting Linux feature to turn one serial port from a microcontroller into a bunch of virtual ports. In theory, you could create over 200 ports, but the reality is you will probably want to stick with fewer.

The feature in question is what’s known as pseudoterminal or sometimes a pty or pts. These special files were made to feed data to programs that expect to accept data from a terminal. The files provide two faces. To the client, it looks like any other terminal device. To the creator, though, it is just another file. What you write to that file goes to the fake terminal and you can read anything that is sent from the program connected to the terminal. You use these all the time, probably, without realizing it since running a shell under X Windows, for example, doesn’t attach to a real terminal, after all.

You could, of course, do the same trick with a composite USB device, assuming you have one. Also assuming you can find a working driver and get it working. However, many microcontrollers have a serial port — even one with a USB converter built-in — but fewer have full-blown USB hardware. Even the ones that do are often at odds with strange drivers on the PC side. Serial ports work and work well even on the simplest microcontrollers.

The Plan

The plan is simple enough. A Linux program listens to a real serial port and watches for special character sequences in the data stream. Those sequences will allow you to switch data so that the data stream will go to a particular terminal. Data coming back from the terminals will go to the real serial port after sending a sequence that identifies its source.

Continue reading “Linux Fu: The Infinite Serial Port” →

Linux Fu: An Odd Use For Fork()

April 21, 2022 by 46 Comments

If you are a Star Trek fan, you’ll probably remember the phrase “You have to learn why things work on a starship.” The truth is, in most episodes, knowing how to override another ship’s console or make gunpowder didn’t come in very handy, but boy when it did, it really saved the day. Linux is a lot like that. There are a few things you probably don’t need to know very often, but when you do need to know, it makes a huge difference. In this particular post, I want to look at an odd use of the fork system call. For many purposes, you’ll never need to know this particular irregular use. But when you need it, you are really going to need it.

This is actually based on an old client of mine who used Unix to run a massive and very critical report every day.  The report had a lot of math since they were trying to optimize something and then generate a lot of reports. In those days, the output of the report was on old green-bar paper on a line printer. The problem was that the report took something like 14 hours to run including the printouts. If someone discovered something wrong, there was no time to run the report again because the next day’s report would have to start before the second run would finish.

The client had a bunch of Windows programmers and — at that time — there wasn’t anything really analogous to a real fork call in Windows. I looked at the code and realized that probably most of the code was spending time waiting to print the output. The computer had multiple CPUs and there were multiple printers, but that one program was hanging on the one printer. There was a lot of data, so writing it to a database and then running different reports against it wasn’t a great option. The answer was to use the power of fork. With a change in the code that took less than 30 minutes, the report ran in five hours. They were very pleased.

So how did I do it? The answer lies in how fork works. Just about every time you see a fork, you see some sort of exec call to start a new program. So if you think about fork at all, you probably think it is part of how you start a new program and, most of the time, that’s true. Continue reading “Linux Fu: An Odd Use For Fork()” →

Linux Fu: Simple Pipes

March 16, 2022 by 15 Comments

In the old days, you had a computer and it did one thing at a time. Literally. You would load your cards or punch tape or whatever and push a button. The computer would read your program, execute it, and spit out the results. Then it would go back to sleep until you fed it some more input.

The problem is computers — especially then — were expensive. And for a typical program, the computer is spending a lot of time waiting for things like the next punched card to show up or the magnetic tape to get to the right position. In those cases, the computer was figuratively tapping its foot waiting for the next event.

Someone smart realized that the computer could be working on something else while it was waiting, so you should feed more than one program in at a time. When program A is waiting for some I/O operation, program B could make some progress. Of course, if program A didn’t do any I/O then program B starved, so we invented preemptive multitasking. In that scheme, program A runs until it can’t run anymore or until a preset time limit occurs, whichever comes first. If time expires, the program is forced to sleep a bit so program B (and other programs) get their turn. This is how virtually all modern computers outside of tiny embedded systems work.

But there is a difference. Most computers now have multiple CPUs and special ways to quickly switch tasks. The desktop I’m writing this on has 12 CPUs and each one can act like two CPUs. So the computer can run up to 12 programs at one time and have 12 more that can replace any of the active 12 very quickly. Of course, the operating system can also flip programs on and off that stack of 24, so you can run a lot more than that, but the switch between the main 12 and the backup 12 is extremely fast.

So the case is stronger than ever for writing your solution using more than one program. There are a lot of benefits. For example, I once took over a program that did a lot of calculations and then spent hours printing out results. I spun off the printing to separate jobs on different printers and cut like 80% of the run time — which was nearly a day when I got started. But even outside of performance, process isolation is like the ultimate encapsulation. Things you do in program A shouldn’t be able to affect program B. Just like we isolate code in modules and objects, we can go further and isolate them in processes.

Doubled-Edged Sword

But that’s also a problem. Presumably, if you want to have two programs cooperate, they need to affect each other in some way. You could just use a file to talk between them but that’s notoriously inefficient. So operating systems like Linux provide IPC — interprocess communications. Just like you make some parts of an object public, you can expose certain things in your program to other programs.

Continue reading “Linux Fu: Simple Pipes” →

Linux Fu: Fusing Hackaday

February 16, 2022 by 9 Comments

Unix and, by extension, Linux, has a mantra to make everything possible look like a file. Files, of course, look like files. But also devices, network sockets, and even system information show up as things that appear to be files. There are plenty of advantages to doing that since you can use all the nice tools like grep and find to work with files. However, making your own programs expose a filesystem can be hard. Filesystem code traditionally works at the kernel module level, where mistakes can wipe out lots of things and debugging is difficult. However, there is FUSE — the file system in user space library — that allows you to write more or less ordinary code and expose anything you want as a file system. You’ve probably seen FUSE used to mount, say, remote drives via ssh or Dropbox. We’ve even looked at FUSE before, even for Windows.

What’s missing, naturally, is the Hackaday RSS feed, mountable as a normal file. And that’s what we’re building today.

Writing a FUSE filesystem isn’t that hard, but there are a lot of tedious jobs. You essentially have to provide callbacks that FUSE uses to do things when the operating system asks for them. Open a file, read a file, list a directory, etc. The problem is that for some simple projects, you don’t care about half of these things, but you still have to provide them.

Luckily, there are libraries that can make it a lot easier. I’m going to show you a simple C++ program that can mount your favorite RSS feed (assuming your favorite one is Hackaday, of course) as a file system. Granted, that’s not amazing, but it is kind of neat to be able to grep through the front page stories from the command line or view the last few articles using Dolphin. Continue reading “Linux Fu: Fusing Hackaday” →

Linux Fu: Bash Strings

January 26, 2022 by 40 Comments

If you are a traditional programmer, using bash for scripting may seem limiting sometimes, but for certain tasks, bash can be very productive. It turns out, some of the limits of bash are really limits of older shells and people code to that to be compatible. Still other perceived issues are because some of the advanced functions in bash are arcane or confusing.

Strings are a good example. You don’t think of bash as a string manipulation language, but it has many powerful ways to handle strings. In fact, it may have too many ways, since the functionality winds up in more than one place. Of course, you can also call out to programs, and sometimes it is just easier to make a call to an awk or Python script to do the heavy lifting.

But let’s stick with bash-isms for handling strings. Obviously, you can put a string in an environment variable and pull it back out. I am going to assume you know how string interpolation and quoting works. In other words, this should make sense:

echo "Your path is $PATH and the current directory is ${PWD}"

The Long and the Short

Suppose you want to know the length of a string. That’s a pretty basic string operation. In bash, you can write ${#var} to find the length of $var:


#/bin/bash
echo -n "Project Name? "
read PNAME
if (( ${#PNAME} > 16 ))
then
   echo Error: Project name longer than 16 characters
else
   echo ${PNAME} it is!
fi

Continue reading “Linux Fu: Bash Strings” →