In the old days, if you wanted to snoop on a piece of serial gear, you probably had a serial monitor or, perhaps, an attachment for your scope or logic analyzer. Today, you can get cheap logic analyzers that can do the job, but what if you want a software-only solution? Recently, I needed to do a little debugging on a USB serial port and, of course, there isn’t really anywhere to easily tie in a monitor or a logic analyzer. So I started looking for an alternate solution.
If you recall, in a previous Linux Fu we talked about pseudoterminals which look like serial ports but actually talk to a piece of software. That might make you think: why not put a piece of monitor software between the serial port and a pty? Why not, indeed? That’s such a good idea that it has already been done. When it works, it works well. The only issue is, of course, that it doesn’t always work.
The software in question is interceptty. You may have to build it from source, but there aren’t any oddball dependencies. Just:
git clone https://github.com/geoffmeyers/interceptty.git cd interceptty ./configure make sudo make install # or however you like to install things like checkinstall, etc.
Back End, Front End
The software uses the concept of a backend device and a frontend device. The back device is, usually, your normal serial port. The frontend device is something that interceptty creates. So the idea is that you connect the program to the backend, it creates the front end, and then you connect some other program to the front end. The program will log all the traffic between the program connected to the front end and the port on the back end.
You can also use file descriptors, unix sockets, or TCP sockets as a front or back device. The backend can also be a running program. There is also a provision for connecting between two actual ports. You can find all the options on the program’s man page.
The output is a bit awkward, but it is easy to parse by other programs including an example Perl program included with it. A character shows you the direction of the data then you see a character in both hex and ASCII.
An Example
It is probably easier to see an example. I have a small controller on /dev/ttyACM0. Here’s a sample command line.
sudo interceptty /dev/ttyACM0 /dev/ttyCAP
This creates the /dev/ttyCAP fake port. Once the program connects you can see the data it sends to the left and the responses (including the echo) to the right.
< 0x3a (:) > 0x3a (:) < 0x78 (x) > 0x78 (x) < 0x38 (8) > 0x38 (8) < 0x79 (y) > 0x79 (y) < 0x38 (8) > 0x38 (8) < 0x69 (i) > 0x69 (i) < 0x31 (1) > 0x31 (1) < 0x30 (0) > 0x30 (0) < 0x21 (!) > 0x21 (!) < 0x0d ([CR]) > 0x06 ([ACK]) > 0x0d ([CR]) > 0x0a ([LF])
Problematic Software
Unfortunately, not all software likes to work with ptys. In particular, the main program I wanted to use takes advantage of the Sigrok serial port library. It is a known issue that this library makes calls that don’t work well with ptys. However, if you use just about any normal terminal program like picocom or tio, it works fine. Other serial libraries seem to be able to handle it, also. I thought about trying slsnif, but it works the same way, so I doubt it would be any better in that regard.
One other note about the Sigrok library. Recent changes in the kernel have removed an ioctl that the library uses so if you are using software that uses libserialport, you won’t be able to find any ports. Until they fix it, you’ll have to build the library yourself and patch the configure.ac file:
git clone git://sigrok.org/libserialport cd libserialport ./autogen.sh ./configure make sudo make install
Be sure you get rid of old versions of the library or, at least, make sure your program is using your custom library.
Other Options
You can, if you like, try converting a serial port to a network socket and then you have lots of monitoring options. Of course, this has similar problems insomuch as not everything understands the fake serial port you create. I can report, though, that Tio seems well-behaved enough to work with these pty fake port.
It would be nice if all serial software understood that they might be called on to talk to a pty or a named pipe. On the other hand, you may have access to the source code for naughty programs, so if you really had to, you could fix them.
In the first “get” listed, intercepttty is spelled 2 different ways.
Sorry!
“git”
Sorry. Only 2 Ts.
Thanks
Does Wireshark work as a software attachment to sniff serial traffic? I’ve used it for USB before, and it’s nice because it has tons of analysis material already written for it.
For me this was impractical, since serial transmission via USB involves a certain protocol overhead. However, the possibility to display baud rate changes etc. is good.
For plain serial ports, strace is an option.
I have used Wireshark to do this before, by writing a dissector for the USB protocol for my particular adapter (it’s usually pretty obvious which bulk endpoint carries RX and TX data). The immediate drawback is that your data will potentially be chunked across multiple USB packets.
It is possible to write a dissector that does conversation tracking and reassembles them into lines or delimited serial packets, which then lets you do all sorts of analysis on top of that, but it is a pretty high setup cost. I’ve done it a few times now, and it is very neat to be able to see, for example, the expanded CBOR payload of each packet in real-time using Wireshark non-invasively while the application-under-test is talking to the serial port, but the RoI is questionable if you don’t already have a lot invested into Wireshark.
Seems like a good reason to go for a native serial port then.
I find socat to be the solution for many “relay” or “proxy” -type problems.
I recently needed to capture traffic from a windows application, being sent to a serial-attached device. Using two usb-serial adapters in a null-modem configuration (one for Windows, one for Linux, and the original target device), and socat to open the two serial ports, configure baud rates, etc, and the -x option to dump the traffic passing through, I had what I needed.
It can also do the interceptty task of creating a pty, and forwarding between that and the real device.
In fact, a whole series of Linux-fu articles could be dedicated to socat, and not run out of content for a good while!
https://hackaday.com/2021/02/11/linux-fu-serial-untethered/
Yup, nice article, I commented on that one too. But again, it barely scratches the surface of what socat is capable of. As another poster below mentions, socat’s man page can be a bit opaque, purely due to the sheer number of possibilities. Some nice worked examples, showing how it can be tweaked to ones will, would probably be much appreciated.
Why the pain ? socat can do all that and much much more, works with ttys ptys, sockets (tcp, udp, file) is stable as a rock, very small, can just be installed on all known Linux Mac Windows of the galaxy, so, why reinvent the wheel ? why ?
Give Al Williams a break; He is only a red belt as depicted in the artwork. He is still working on his black belt.
B^)
Socat has ~30 command line flags, hundreds of keywords, complex semantics, and more. It’s probably Turing complete, but at some point a program goes down in usefulness as more functionality is added.
I’ve been working with *NIX for over 40 years… personally I liked both the article and programs that do one or a small set of things really well. I’m sure socat, SSH, openssl, and other ridiculously complex programs can also do this, as can any number of programming languages. But at the end of the day I don’t really care, as the documentation is opaque and the only way for most to figure such things out is to copy an example that does exactly what you want.
It’s really easy to complain, but it’s rare and actually pretty difficult to teach and create over a long period of time. I’ll take Al’s depth of knowledge, well meaning sharing of information and mere red belt any day over your relatively anonymous griping.
Point to your writeup about the joys of socat (hey, join Hackaday, I understand they’re hiring) and something that compares to the nearly 3000 other topics that he’s written about on Hackaday and perhaps then you can talk.
I used to complain about poor quality books until I wrote one myself (poor or no… not mine to say)… but now I’m just impressed anyone can survive the process.
TLDR; rock on, Al.
Never heard of interceptty, thanks for the tip. In the past i used “strace -f -e read,write” to eavesdrop on my serial to network process ;)
I used to use ttysnoop for my Linux BBS some 25 years ago. It let me display what users were doing on my BBS on one of my virtual terminals.
I have had success with socat, but also issues like Al mentions with some software not accepting the pseudoterminal end as a serial port. In particular, VirtualBox didn’t like it last time I tried, when I wanted to snoop on what some Windows XP software was doing over serial.
There’s probably more that could be done to make the ptty appear more like a native serial device, but I’ve not figured that out yet.
“There’s probably more that could be done to make the ptty appear more like a native serial device, but I’ve not figured that out yet.”
That’s a pity!
B^)
“Recent changes in the kernel have removed an ioctl”… could anybody shed a bit more light on this?
Apologies for generating noise, https://www.spinics.net/lists/linux-serial/msg41926.html and https://lore.kernel.org/lkml/20210512144827.956315973@linuxfoundation.org/ appear to apply. In summary: in the kernel’s tty_ioctl.c, unimplemented ioctls were changed to return -EINVAL (possibly in some kernel versions -ENOTTY) . This related in particular to TCGETX, which the above says was never implemented at the driver level.