Linux Fu: The Infinite Serial Port

Ok, the title is a bit misleading. Like most things in life, it really isn’t infinite. But I’m going to show you how you can use a very interesting Linux feature to turn one serial port from a microcontroller into a bunch of virtual ports. In theory, you could create over 200 ports, but the reality is you will probably want to stick with fewer.

The feature in question is what’s known as pseudoterminal or sometimes a pty or pts. These special files were made to feed data to programs that expect to accept data from a terminal. The files provide two faces. To the client, it looks like any other terminal device. To the creator, though, it is just another file. What you write to that file goes to the fake terminal and you can read anything that is sent from the program connected to the terminal. You use these all the time, probably, without realizing it since running a shell under X Windows, for example, doesn’t attach to a real terminal, after all.

You could, of course, do the same trick with a composite USB device, assuming you have one. Also assuming you can find a working driver and get it working. However, many microcontrollers have a serial port — even one with a USB converter built-in — but fewer have full-blown USB hardware. Even the ones that do are often at odds with strange drivers on the PC side. Serial ports work and work well even on the simplest microcontrollers.

The Plan

The plan is simple enough. A Linux program listens to a real serial port and watches for special character sequences in the data stream. Those sequences will allow you to switch data so that the data stream will go to a particular terminal. Data coming back from the terminals will go to the real serial port after sending a sequence that identifies its source.

I built a contrived example because there were certain key features I wanted to test. On the microcontroller, one thread reads an analog quantity and another a digital quantity. The system prints both of these out to separate virtual serial ports. If you press a key in either terminal, the associated data will pause. There’s also a debugging terminal and a command terminal that takes user input to alter things like how often to sample data.

The code, by the way, is on GitHub. If you aren’t using an STM32F411 Blackpill with Mbed, you’ll probably have some work to do. However, keep in mind that the complexity in the code is to fit in with the Mbed libraries. The actual protocol is simple and you can implement it anywhere.

The Protocol

Speaking of the protocol, it is both lightweight and robust. Each device is both a transmitter and a receiver and there is very little connection between them. So while one device is sending to a virtual channel, it might be receiving data from a completely different virtual channel. Each virtual channel has an identification number that can range from 0 to 253.

The transmitter sends most data bytes directly over the port. When there is a desire to change virtual ports, the transmitter sends an FF byte followed by the channel number. There are a few issues to consider.

First, the transmitter has to be aware that if it really wants to send an FF, it can’t. So it sends FF FE instead. If you were sending nothing but FF characters, that would double the amount of data sent, but that’s a relatively rare situation.

Second, there is a robustness problem if the transmitter sends an FF and then dies. The next byte could be considered a channel number. However, since a freshly restarted transmitter should send an initial channel selector, the next byte should be an FF (the initial transmit selector). To combat this, the protocol accepts any number of FF bytes as a legitimate prefix. So all of the following will select channel 4:

FF 04
FF FF 04
FF FF FF FF FF FF 04

The only issue is if the serial connection is intermittent. It is possible to get an FF byte, have the cable go out temporarily and then get a later byte that will trigger an unwanted channel change. There isn’t much you can do about that, although this assumes the next byte is a legitimate channel number since most systems won’t use all possible channels. You could add some robustness by timing out the escape prefix or adding, say, a checksum to the switching sequence but that would only reduce problems not eliminate them. If you are dealing in pure ASCII data, limiting to channels above 0x80 would reduce the issue but, again, not totally eliminate it. In practice, if you have a reliable serial connection, there’s no problem.

Third, there is the case where the receiver dies and starts again midstream. This can cause an issue where some data goes to the wrong virtual terminal. There are two features to help with this. First, if the transmitter sends FF FD to the receiver, the receiver should retransmit its current channel selector. You could also set a transmitter to periodically send the channel selector since it is harmless to send the same selector more than one time.

Another way to combat this partially is to use the -s switch on the Linux server to prevent any data from flowing to the virtual terminals until one is explicitly selected. Note that this only matters during the start-up phase of the server. Once a channel is selected it stays selected until another one is selected.

KISS

You can probably see that all the complexity here is to make the server fit in with Linux and the microcontroller side fit in with the C library calls. If you wanted to send data from, say, an Arduino, it would be easy to just prefix everything with “\xFF\xCC” where CC is the channel number. Receiving data could be almost that easy. You just have to remember when you’ve seen an FF and handle the three escape codes (that is, FE is a real FF, FD is a request to resend the channel selector, and everything else is a channel change).

However, I wanted to be able to just have virtual serial ports on both sides of the cable, so it was a lot more work. Part of it is that working with Mbed I/O streams is quirky at best. First, let’s look at the Linux server.

The Linux Server

The Linux software opens a terminal port (e.g., /dev/ttyUSB0 or /dev/ttyACM0 etc.) and then produces multiple pseudo terminals that most terminal software can use. For USB devices, the baud rate is probably unimportant. However, for a real serial device, you probably need to match up baudrates (untested). The code currently does not change the baudrate so you’d need to set it externally or modify the code if you need to.

The ttymux program takes a few options. The only one that is critical is the -c option which defines a virtual port. Each port has an ID number from 0-253. You can also ask for a symlink. So, for example, look at this command:

ttymux -c 10 -c 33:virtualportA -c 50:/tmp/portB /dev/ttyACM0

Here we are creating three ports. Port #10 has no name. Port 33 and 50 will be in ./virtualportA and /tmp/portB, respectively.

Other options:

  • -d – Autodelete symlinks on exit
  • -n – Do not set attributes on serial port
  • -s – Do not send data to a virtual port until expressly selected

When the program runs you’ll see a list of channels and their associated psuedoterminals (probably /dev/pts/X where X is some number). If you don’t provide a symlink, that’s how you connect to the virtual port. If you provide a symlink, you can use either. Note that the ID number is not the same as the pts number. So channel 10 in the above example probably won’t be /dev/pts/10. If it is, that’s just a coincidence.

To compile, you need pthreads:

g++ -o ttymux ttymux.cpp -lpthread

Looking at the code, the server isn’t really that complex. There are two threads. One reads the serial port and the other writes to it. Nothing else touches the real serial port. Then each virtual channel has its own pty. There’s no need to buffer characters or anything.

The part that creates the pty is very simple:


pty=posix_openpt(O_RDWR|O_NOCTTY|O_NONBLOCK);
if (pty==-1) return -1;
grantpt(pty);
unlockpt(pty);

To create the symbolic links, you need to know the name of the pty which is what the getptyname call is for. Using that file name to open will give you the terminal side of the pty. I usually use picocom which has no problems. But some programs, like cutecom, for example, know too much about what a serial port is supposed to look like so you can’t open a pty with them.

The Microcontroller

The microcontroller code is a bit more convoluted. The SerialMux class does all the work. Like the Linux side, it has two threads that control access to the real port. Unlike the Linux side, each virtual serial port object has its own set of input and output buffers. The threads fill those buffers and — eventually — _read and _write do the work of getting the data in and out of the Mbed stream base class. It is not terribly efficient because the streams eventually call _putch and _getch to do I/O character by character, but there are reasons for that and if you really wanted to tweak it, you could override the underlying methods.

One thing that surprised me was that returning true for the isatty function totally breaks the I/O system. I didn’t figure out why, but I did note that the factory-standard USBSerial class also returns false for this and has a comment about it in the Mbed source code.

Another thing that was odd is that the stream’s built in mutex acted strangely and I never figured out exactly why. It may be that it protects all streams or something but I eventually reverted to having my own mutex to protect each set of buffers and another mutex to protect the physical serial port.

Once you follow the logic, the SerialMux class is not that hard. A lot of the code in the main.cpp is to work around issues when the USB port is disconnected.

I also copied and modified a class I’ve used before to build command processing systems. Probably overkill for this, but I had it hanging around and it was easy to use. It is interesting that the code doesn’t know anything about the multiplex system. It just takes a normal stream and works fine.

In Use

Once you have the microcontroller program loaded you can run the Linux server with this command line:

ttymux -s -c 1:analogport.virtual -c 2:digitalport.virtual -c 100:debugport.virtual -c 10:cmdport.virtual

I then use picocom on each virtual port, although you could probably use any other terminal program. You will want to set some options for the command terminal for your own benefit:

picocom -c --omap crlf cmdport.virtual

In the analog and digital terminals, press any key but a space to pause the loop. A space will resume. In the command window, you can use the help command to get a list of what you can do. Don’t expect backspace to work, although there are ways around that if you wanted to fix that.

Future Directions

The protocol is simple even if the example is not and there are many ports possible. Even simple microcontrollers could emulate many ports.

On the server side, someone will surely ask for Windows. It seems like you could do something similar with com0com or one of its related projects. Other ideas would be to feed network sockets instead of pseudo terminals or to work out a system to duplicate fake ports. The protocol and code are simple enough to make any of that possible.

It is true that USB composite devices are a better answer to this problem if you have the hardware and drivers to support it. However, that’s a lot more work, a lot more resources, and limits the ability of cheap hardware to participate.

If you are interested in the USB route, there are some examples of using a Pi Zero and some user-space code if you can provide your own Windows drivers. If the network angle piques your interest, you might have a go with ser2net.

23 thoughts on “Linux Fu: The Infinite Serial Port

  1. It kinda drives me nuts that so many serial port hacky protocols work by just continually adding escape sequences when it’d be fairly trivial to “alert” the other side that you want to switch modes – just send an illegal sequence to force a framing error (like 10 bits high with the proper timing – which you can do by swapping baud rates and sending 0xFF). You’re already wasting 20% of the bandwidth on encoding, you might as well *do* something with it.

    I mean, it’s not like framing errors really ever happen with UARTs unless the connection’s actually broken (as in not functioning, as in the timing between sender/receiver doesn’t match) – if they’re happening due to noise they’d cause bit errors far more often.

    Unfortunately it’s such a royal pain to recognize framing errors in frameworks like Arduino, etc.

    1. Such an over-long sequence of bits is sometimes called a break sequence. For a real-world example, Linux’ serial console interprets a break sequence as the SysRq key. Minicom can send a break sequence (ctrl+A, F). But yes, it is a bit of a PITA, for example if you have some bytes in the receive buffer, you can’t reliably tell where exactly the break has occured – at least under Linux. But to be honest, I think that limitation is also shared by most UART peripherals, so the OS can’t do much about this.

      1. Well, the Linux side implementation is a ton easier than Arduino, which has absolutely no implementation of serial errors whatsoever and for some inane reason decided *throwing away* parity errors was a good idea. Sigh.

        1. At least you can always use the lower-level libraries with Arduino. For most MCUs, the Arduino environment uses the manufacturer’s SDK anyway so you can just include the relevant headers and use the official libraries alongside Arduino (STM32 HAL, ESP-IDF, or even just the raw register definitions in the case of AVR)

          1. Ha, I wish. For serial ports, the Arduino cores basically act as drivers: you write to a ring buffer, and an interrupt-driven process slowly clears it out. Similarly the receive interrupt feeds a ring buffer that you periodically check.

            The issue is that for most (or all?) cores, the error handling happens in the interrupt function, and they just blindly clear errors as they show up and throw away data. So you have no idea whatsoever that errors are happening, data just goes “poof.” Yes, of course, you can just not bother with the Serial class and use the manufacturer’s UART directly, but that’s really no different than saying “you can just use the manufacturer’s IDE.”

            Plus of course Serial is so common that many other libraries just take it (or a Stream) so in the cases where I’ve actually cared about serial errors (hey, that +/-5% clock agreement requirement is harder to meet than you think when you have a 60 degree C swing in operating temps!) I end up just modifying the damn core (which is a maintenance pain).

    2. Mucking with baudrate settings on the fly like that sounds like a pain. What if your UART is DMA driven? It probably is on the Linux side. It would be a huge performance hit. Keeping the multiplexing protocol in-band is very important for anything high throughout.

      An answer to avoiding excessive escape sequences over a serial line is to use a framing protocol like COBS – consistent overhead byte stuffing. I’ve used it in rockets, self driving cars, and in UAVs.

      1. You’re complaining about the fact that the implementation of serial errors is terrible. That’s totally true – it’s completely hacky because of the implementation on Linux (which isn’t good, as it shoves the errors out of band) and on Arduino (which is god-awful worse).

        That’s why I said it’s frustrating. Framing and other errors should absolutely be preserved in-band so that the receiver knows exactly when it happened, and blindly ignoring errors is like, the *worst* thing you can do.

        Of course it’s not entirely Linux’s fault, as the UARTs themselves are internally buffered and *they* don’t store stuff in-band. You could work around it – we are talking about a damn serial port, after all, the performance is a joke. But the Arduino stuff is absolutely their fault.

        It’s just frustrating as hell that serial ports weren’t designed better. Like I said, you can still make it work, especially if you don’t really care about difficulties regarding when a port is switched, and you don’t switch often (think of it like a KVM switch – you switch very rarely).

        “An answer to avoiding excessive escape sequences over a serial line is to use a framing protocol like COBS”

        It’s not just overhead that matters: it’s also latency. An advantage to having a sideband is that the data can be forwarded with only one element (e.g. 8 bits) latency and that latency’s deterministic. Packetizing data dramatically increases the latency (since you need to depacketize before forwarding). And escaping data adds jitter to the latency.

        1. Some systems actually allow nine-bit serial, wherein you could send 9-bits to an 8+parity receiver… adding a bit of “data” for e.g. escaping via intentional parity-error (which i think is stored with each byte in the linux-side buffer)

        1. Uh, not… really?

          There are COBS to AXI4-Stream encode/decode modules up on Alex Forencich’s verilog-axis GitHub repository. They work straight out of the box (and they’re not particularly complicated, actually).

          https://github.com/alexforencich/verilog-axis/tree/master/rtl

          The only addition I usually make is to add a reset after a string of zeros of some length. If you’re worried about reliability (ensuring you only receive valid packets) you just add a buffer afterwards that waits until a complete error-free frame’s received (tlast with no preceding tuser) and then pushes the frame out. Also relatively easy to do with modules that are located there.

  2. As I was reading the protocol that you were developing, it reminded me of the KISS protocol and I got excited when I saw the heading KISS, but then realized you weren’t referring to the protocol.

    I’ve been using KISS at work and it is a handy protocol for packetizing data over a serial connection and even directing it to different ports. I would definitely recommend taking a look at it:

    https://en.wikipedia.org/wiki/KISS_(TNC)

  3. It’s when you start sending data in chunks and listen for a checksum, channel hopping and retransmitting if it fails or just proceeding to the next block of data if it’s good. Lot’s of bad packets, you start decreasing the amount of data in each chunk and visa versa.

    1. Have used a sort of “BitBus” on 8051’s with multi master halve duplex at 375 kBps for industral use (automotive quality) running from 1992. Started in 1984 and waited for Intel to complete the BitBus protocol and specs but ran out of time so we created our own protocol and where (and are) happy as the Intel version is a master / slave.

      In 1999 started with NT3.51 as a successor for DOS, it took till 2004 before the 1e machine with XP was delivered to a customer, the last DOS machine was delivered in 2017 (8 months after my retirement and end of the DOS era).

      We shoose NT on RMS (NT runs as lowest prio) over Linux cos we didn’t know what to expect from a multi user, multi tasking system in terms of realtime constraints and how to make drivers.

      From what i read here a virtual terminal is a much better way than the drivers we did make for XP and Win7.
      Till now, i had no idea what the use of pty and pts :< and now it's to late ;(

  4. The NVIDIA Tegra/Jetson system software uses this exact same multiplexing technique to allow the many different CPUs and SW environments (e.g. bootloader, secure OS, secure monitor, Linux) to share a single physical UART for debug logs and consoles. The demultiplexer is included somewhere in the public software releases.

  5. I was going to say “data transfer is easy until you consider the possibility of errors; then it becomes enormously complex.” But I think I can say the same thing for computing in general.

  6. Nice article and examples on using serial ports. Didn’t know that the RS232 Parity error bit is ignored in the software or hardware built into the UART. Though it does make more sense to service errors in the software using the serial data rather than trying to figure out how to handle it in the UART.

  7. How do you select virtual channel 253? 253 => FD, but FF,FD just reports back the current channel number. So unless I misread something, virtual channel numbers should be in the range 0-252.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.