Modern operating systems insulate us — as programmers, especially — from so much work. Depending on how far back you go, programmers had to manage their own fonts, their own allocation space on mass storage, or even their own memory allotments. Every year, though, it seems like things get easier and easier. So why is it so annoying to open a simple serial port? It isn’t hard, of course, but on every operating system it seems to be painful — probably in an attempt to be flexible. And it is even worse if you want portability. I needed to write some C code that read data from an FPGA’s embedded logic analyzer, and I was annoyed at having to write yet more serial port code. I have my own shim library, but it isn’t well tested and isn’t all that flexible — it does what I need, but I wanted something better. What I wound up with the serial library from Sigrok. You know Sigrok? The logic analyzer software.
You might counter that the serial port is old hat, so no one wants to support it with modern systems. While the physical serial port might be on life support, there’s no shortage of equipment that connects via USB that appears to be a serial port. So while I was talking to an FTDI chip on an FPGA board, you could just as well be talking to an Arduino or a USB voltmeter or anything.
I guess the Sigrok developers had the same problem I did and they took the time to write a nice API and port it to major platforms. Although Sigrok uses it, they maintain it as a separate project and it was just what I needed. Sort of. I say sort of because the version installed with Ubuntu was old and I needed some features on the newest release, but — as usual — the Internet came to the rescue. A quick Git command, and four lines of build instructions and we were ready to go.
Getting Started
Since I use Linux, the build instructions on the page worked fine. The install goes into /usr/local so I removed the libserialport-dev package just to be sure I didn’t get the wrong one by mistake.
The library provides a header file, libserialport.h. That file defines an sp_port type that is opaque. Since you don’t know the size of it, you can only create a pointer to it. You can get the pointer filled in by calling sp_get_port_by_name. You can also have the library give you a list of ports available. Once you have the port you can open it. There are simple calls for setting most things, but you usually only need to call sp_set_baudrate.
There’s also simple calls for doing blocking and non-blocking reads and writes and figuring out if there are characters waiting to be read.
My Example
I wound up writing about 50 lines of code to ping the analyzer with two commands and pull the data out to the terminal. You can find it below. It was just quick and dirty — I didn’t try to optimize the reads or anything.
I doubt you’ll implement my code since you don’t have the logic analyzer. I will share that eventually in another post, though. But if you do want to have a go, you could easily code an Arduino to pick up two bytes on the serial port and then dump out 4,096 bytes.
Of course, this wasn’t my final code, but it shows how easy it was to write some portable code to work with the serial port.
// WordPress loves to eat angle brackets, so if the 3 includes below are blank
// they are: stdio.h stdlib.h and libserialport.h
#include <stdio.h>
#include <stdlib.h>
#include <libserialport.h>
#define BAUD 9600
// Commands to LA
#define USERCMD_RESET 0
#define USERCMD_RUN 1
int main(int argc, char *argv[])
{
struct sp_port *port;
int err;
int i,cmd;
int count=4096;
// We need a serial port name
if (argc<2)
{
fprintf(stderr,"Usage la port\n");
exit(1);
}
// Open serial port
err=sp_get_port_by_name(argv[1],&port);
if (err==SP_OK)
err=sp_open(port,SP_MODE_READ_WRITE);
if (err!=SP_OK)
{
fprintf(stderr,"Can't open port %s\n",argv[1]);
exit(2);
}
// set Baud rate
sp_set_baudrate(port,BAUD);
// write reset
cmd=USERCMD_RESET;
sp_blocking_write(port,&cmd,1,100);
// write run
cmd=USERCMD_RUN;
sp_blocking_write(port,&cmd,1,100);
// read data
for (i=0;i<count;i++)
{
int waiting;
int c=0;
do
{
waiting=sp_input_waiting(port);
} while (waiting<=0);
// sort of inefficient -- could read a bunch of bytes at once
// could even block for all of them at once
sp_nonblocking_read(port,(void *)&c,1);
if (i%16==0) putchar('\n');
printf("%02X ",c);
}
putchar('\n');
sp_close(port);
return 0;
}
Portability and Support
The library works on Linux, Mac, FreeBSD, Windows and Android. No Commodore 64 or VAX support, but we can let that slide. It is well documented, too. You might wonder what the big deal is. Well, let’s take a little detour. Keep in mind that you may have a favorite library that hides a lot of this from you and that’s great. Maybe it is even cross-platform, at which point that’s great, too. But I’m talking using the OS’s native API.
Linux borrows heavily from Unix so it thinks serial ports might just be modems or teletypes. That means there are dozens and dozens of obscure options for serial ports, mostly in the wacky termios structures. Everything is a file, so opening the port isn’t that stressful. You do have to decide if you want to set strange bits like O_NOCTTY and O_NDELAY. Oh, and you probably want to tell the port not to hang if no data is available by setting the FNDELAY bit which is via another API call. Plus set the baud rate.
I won’t bore you with all the exact code, but if you really want an idea of what’s involved, here’s a pretty good reference. Just remember the benign code to open the port is just the tip of the iceberg. Keep reading at least until Chapter 3.
Windows isn’t much better. The actual API interface is similar — but not the same — as the Linux code. You create a file almost in the usual way. But then you get a device control block (DCB) and fill it all in to configure the port. Want timeouts? That’s another call. There’s a whole different API for non-blocking I/O although you have a few choices there. There’s also EscapeCommFunction to do other odd things.
None of this is that hard and, in fact, Windows is a bit easier only because it is less flexible. But it is very easy to just use one portable open source library.
Going Forward
Next time you want to talk serial with a PC using any language that can call a library, you ought to consider this one. Even if you don’t need the portability, it is a pleasant enough API and you don’t have to switch gears later if you do port or if you just start a different project.
If you don’t like it, there are other choices. A quick search turned up a C++ library that supports Windows, Linux, and Mac. Some popular libraries like Boost for C++ have serial port handlers, too. Here’s a simple C-language library that is pretty minimal. There are doubtless others and I’d imagine the comments will turn up a few gems, as well.
Photo Credit: Fun with VMWare by [Bill Bradford] CC-By-2.0.
Oh…
I was hoping for a story on easily adding hardware serial ports to any/every thing.
I often use the serial API from boost asio. Works well…
This is the bomb. Is there anything similar for I2C or SPI?
I2C needs pull up resistors.
But both are bitbangable, so only a matter of software.
Or if you’re brave and on Linux, use your GPU’s i2c line that’s in the external monitor port.
Bitbanging doesn’t help with pull-up resistors. If any device is driving the line high and another drives it low, there’s essentially a short between the power rails spread across their driving transistors. That’s a recipe for releasing the magic smoke of the weaker chip.
Pullups are specced at 10k(ish) b/c about any transistor should be able to pull that down without burning out.
The article referred to a portable software library for UART communications. I was referring to a similarly portable software library for i2c that would work for Linux i2c-dev and whatever Windows uses for i2c.
For I2C there is a high-level API called SMBUS, it defines higher level transactions over the I2C bus and an API for them. The API is not universal but it is supported on Linux and some embedded systems. It is a breeze to use compared to the low-level Arduino API. Alas, there are some I2C devices that do not use SMBUS transactions.
Look for RS-232 to SPI converter uses serial port software. On Hackday.IO under my name
“Linux borrows heavily from Unix so it things serial ports might just be ” I don’t ‘thing’ so.
“Linux borrows heavily from Unix” is just plain poor writing and it’s not even correct, remember the SCO lawsuit? The correct way to say this is “Linux adheres to the POSIX standard”.
I wish hackaday would crack down on the comments from the people in their mom’s basement. The linux design philoshiphy borrows from Unix. I don’t know why you think it is poor writing and it isn’t incorrect although it may be a little imprecise. If they had said linux borrows code from Unix that would probably be wrong although I wouldn’t want to bet my last euro on it being wrong.
But not just this comment but there are always a few that are like: hackaday invents cold fusion and the comments will be well it technically is room temperature, not cold. That ruins the whole thing. Hackaday sux. I don’t know if they are 15 year olds or just people with very low self esteem trying to feel better. Either way PLEASE hackaday I know if you delete comments out people squeal about censoring but give us an upvote downvote button and then maybe a default filter for negative scores. So tired of the stupid comments.
No. The tighter a site filter’s it’s comments the more it’s authors’ own biases will come into play. The last thing anybody needs is more echo chambers!
I very much wish that HaD would stop filtering comments altogether. I would however love to see a rating system where the really bad stuff can be rated down until it no longer shows up by default but individual users could still opt to lower their threshold until everything shows.
POSIX only defines the userland environment. GNU is the most common Linux userland implementation and it deviates a great deal from the POSIX standard. Linux does in fact borrow heavily from various UNIX implementations.
Well.. the original intent of Linux was to re-implement Minix. https://groups.google.com/forum/#!topic/comp.os.minix/4995SivOl9o
The Minix website talks a lot about Posix today and v2 of Minix is Posix compliant but v2 came out years after Linux and the original Minix which Linux was imitating began a year before the very first Posix specification was even released!
Also, Posix itself.. what is it based on? Unix!
So yes, Linux definitely borrows heavily from Unix and it is not a secret, it was intended from the beginning.
“remember the SCO lawsuit?”
Yup. So? “Borrowing heavily from Unix” is a vague statement that does not necessarily imply any copying of code and even if it did the long overdue outcome of that lawsuit when it finally came only reinforced what we already knew. Most of the code that makes up the many branches of the bushy tree known as Unix has entered the public domain for various reasons so everything really is ok.
Good programmers do avoid re-inventing wheels you know! It’s sad but I think was forgotten when the current generation of programmers were getting their educations…
Or just use .Net (Mono) or Java and you’ll write even less Code with the same portability.
yea I am sitting here thinking its like 1 line in .net. Seems the case of doing it the hardest way possible for a simple end result and the entire essence of this long (for HAD) article is use a library … no shit, thanks
Yeah right. I write C# for a living (because I’m paid to, not because I chose .NET). Anyway, there is no such thing as 1-line. It’s more like 3 lines of checking for null values, 3 lines of converting data types, 1-line of code and you still feel guilty because you know you took some shortcuts that will break in certain edge cases for those value-checking and type conversion sections.
Java’s serial ports are a mess. Rxtx does what it can but there are a myriad of problems as anyone who has done wide deployment of Java with serial ports can tell you. .Net is ok if you can restrict yourself to the stuff that runs well on Mono. That’s how I deploy the GP3EZ software on Windows and Linux, for example. Even then, I have a little custom code for each platform to enumerate serial ports, etc.
i usually just use lua for writing interfaces to serial devices. granted the rs232 library kind of sucks and doesnt give higher speed options. but it does mean i can write a gui in iup in about 15 minutes.
Or just use .Net (Mono) or Java and you’ll write bloatware with the same portability.
FTFY
I care about resources with a high value – development time.
You obviously care about resources that are plentyful and cheap – like RAM.
So yeah – call it bloatware and have fun developing boiler plate code nobody is willing to pay for all day long.
>”I say sort of because the version installed with Ubuntu was old and I needed some features on the newest release, but — as usual — the Internet came to the rescue. A quick Git command, and four lines of build instructions and we were ready to go.”
This is what baffles me about the Linux ecosystem. Why does anyone tolerate this sort of software distribution failure? You can’t get the software because you’re relying on some Appstore-by-another-name that’s just the same a gilded cage, only it’s managed by an uninterested and unpaid third party that often actively hates the software you want to use because of personal/political reasons.
But sure, you can do the software distributor’s work for them, in some cases, if you know how to, or you’re willing to spend hours googling how to do it, so it’s all “fine” in the end.
The difference to an appstore is that you can choose your appstore (sources.list) and that you have the power to bypass it (manual install). Try this on Windows 10 S (S?)…
Yes, that would be fine if all the repositories weren’t equally ill-maintained or simpy incompatible with your distro.
Getting software from a Ubuntu repository feels exactly like the concept of distributing software through Windows Update. You know that even if Microsoft spent half their budget on packaging and maintaining software for all the developers out there, they still couldn’t do a good job of it, nor can anyone else.
Hence the humble Setup.exe (or any other standard installer on an OS that is binary compatible with itself)
If you care enough about software – you maintain your own package. After all – building a package and having your own repository is easy after you do it the first time.
If you don’t – distro version is just fine.
And yes, I am speaking from experience.
You sound like a FreeBSD user that just doesn’t know it yet.
Having many of the same thoughts you are writing but also being thoroughly fed up with Microsoft lead me to Gentoo. Gentoo has been great, the least of all Linux distro evils. It’s not completely without issues though and waiting for EVERYTHING to compile is kind of a pain. (Doing updates in screen with a nice level set helps).
I’ve longed for a distro that gives me the flexibility of Gentoo’s portage for the packages I want it on while still having a good selection of binary packages for quick install of those where I don’t need that kind of customization.
I briefly tried Arch. AUR feels a little bit too disconnected from the main packaging system though. I don’t want to manually download the source into a folder somewhere. Also I ran into too many things that were only available in AUR that I would have been happy to have in binary form. It kind of seems like a crappier version of Gentoo but where the system packages come as binaries.
I forget the name, I did try a tool that automates the AUR process but it tries to do everything in /tmp which meant it was prone to failure do to running out of space.
I’m playing with FreeBSD now. It feels like a binary Linux distribution but if you want to compile something you can do so using ports which automagically build a package that installs keeping everything consistently handled by the package manager. Unlike Arch they seem to take the strategy of keeping EVERYTHING available as both binary AND source so YOU decide which way to go. I am liking this so far. I’m still struggling a little getting the display manager to work properly and I haven’t tried installing a couple things I want that only seem to officially support Linux nor have I tried Wine yet so I’m not yet a FreeBSD convert though.
Ummm basically any other ecosystem, you just don’t get out of the gilded cage period.
I don’t need a repository/appstore when my OS is standardized enough to run the same binaries across different installations. Just download from anywhere and run.
“Use the source, Luke!”
B^)
ROFL I needed a good laugh
seconded.
I had to track down several packages related to automake before it would compile on my Mac (which already has XCode and the XCode CLI tools.) It wasn’t hard, but it was annoying…
“But sure, you can do the software distributor’s work for them,”
What work? Are you paying them?
pyserial worked well for me on Linux, Windows and OS X.
Oddly I just got finished adding a lot of Python to an open source project (probably increased the line count by 75% or so). I am sorry but I really dislike Python. Yes I can use it and yes I understand most if not all of the features. I just don’t care for it. I don’t know if it is the indentation (and yes, I know there is a version with braces) or if it is the bad taste from every Python program I install insisting on its own set of libraries and breaking all other programs unless you give it its own little container to play in. I don’t know.
They used to call that “DLL Hell” in Windows, 25 years ago.
I also found pyserial much easier than C. Even on Linux, it took me hours to figure out custom baud rates. Termios2 is not well documented.
“screen /Dev/[port name here] 115200” has worked for me for ages. But I guess that is not available on that Microsoft thing they call an operating system.
Having to specify USB, ACM or RFcomm gets a bit tedious at times.
Exactly. No need for additional libraries. Just open the port like a file. Accessible from any software which can read and write files.
How does that work with high speed data? Of course, you also can’t control the control lines and it isn’t very portable off *nix no?
Also IIRC it makes the serial port behave like a TTY, which is fine if you’re sending text over it but is prone to failure if you’re sending binary data back and forward since it will try and interpret control characters. Of course it’s not rocket science to get things properly configured, but it’s a bit more than just ‘snark, everything is a file in linux, fnah fnah’.
This is very useful, thank-you. I just copy all that termios stuff from the last project I used. It came from the Internet originally and I don’t understand it. I’ll definitely be using libserialport next time.
As this runs on FreeBSD, I guesstimate it will also do on NetBSD; then it will also do on VAX, AXP, and many more. It claims to be portable, does it?
If you want to change the install location you simple use “configure” with a different prefix option. /usr/local is simply the default…
Compatible with Windows? That’s a complete load of crap. The Windows environment has gotten steadily worse with each new release and Windows 10 is downright hostile to developers that won’t let Microsoft ram them in the ass.
I used an Atmega32u4 to play around with some ideas. Windows 8 and forward did not like me setting up a Serial-USB bridge and absolutely refused to accept the device without signed drivers. Seriously? It had no problems with me fudging a parallel port, a super tiny fake drive, a keyboard, mouse, anything at all except a virtual RS232 port without forcing Windows into a dev mode. It’s astounding that Microsoft can’t secure their USB ports but locks small developers out of a virtual serial port.
I ended up dropping Windows for my own work. Not just for that, but it was definitely a part of it.
What kind of virtual serial port were you using? A CDC-ACM one? I had no problems with that, if the USB descriptors are set up correctly, Windows 10 will automatically load the correct driver (usbser.sys, which is signed by MS) and everything works automatically. On Win8 you can (IIRC) select USB-Serial manually in the device manager. On older Versions, you can write your own self-signed .inf file to load the usbser.sys. In no case do you need any “dev” mode.
BTW, (as you probably know) libusb is a portable library for accessing USB devices… So, one could just drop the serial stuff altogether and use USB directly. By declaring the device as a WinUSB device (special magic descriptor needed), Win10 will automatically load the WinUSB driver, which libusb can access directly, so there is zero installation/configuration needed. That’s (for the user) even easier than serial (no selecting ports, configuring baud rate).
I don’t get it.
All these old ports are supposedly dead, no longer common on new machines. Serial, PS2, Parallel, PCI, etc… That’s what I’m lead to believe every time hardware serial ports have been mentioned on HaD for the past 10 years or more.
But… at least as an anecdote it just doesn’t match up for me. So long as I don’t count those ridiculous super-thin made to be replaced in less than a year toy laptops that are so popular today… Most desktops and even ‘made to do real work’ laptops that I look at still have one serial port on them. If I shop for motherboards.. they almost all have a serial port or two. Even parallel ports aren’t THAT rare when building from parts.
I just checked the computer that I am typing on. It is a brand new Dell, marketed for office work, purchased less than a month ago by my boss who has no reason to look for a serial port when shopping. Sure enough, there’s that male 9-pin connector that PCs have had since almost the beginning. What really surprised me though.. this thing even has PS2!
The best I can guess is that readers here must be trying to be overly trendy. Sure.. if you are going to buy Apple products and/or Microsoft Surface devices they aren’t going to come with a lot of useful, easy-to-hack with ports on them. That kind of thing isn’t hipster enough for that market. The sight of connectors clashes with the designs of their target market’s latte cups or something.
If you are the kind of person that wants to build or hack on things then why not do yourself a favor and consider that when you make your big purchases? Computers with ports on them are still a thing!
Oh.. and one other thing. USB serial adapters are not really good enough. First, most don’t break out the RTS or DTR pins which can be oh so useful to open/close relays or to toggle the reset pin on a microcontroler. Also since they are removable their character file names or com port numbers can change. Unless you are running Linux and care to edit your udev rules this is why you plug your Arduino in, everything is working then you unplug/replug it, power cycle or something and now there is no communication until you go in and re-select the now renamed serial port from the list. Built in serial ports don’t do that!
At work, we still make heavy use of serial ports. When you’re bringing up a new board, you need to be able to talk to it and later provision it, beyond jtag (assuming jtag is present and working).
Every “managed” router out there today still has a console port that is “serial”.
In case anyone is digging this up. Recent changes in the kernel has removed a key ioctl the libserialport library uses so now it can’t open any ports on Linux! The fix is very simple. You have to remove a single line in the configure.ac file and rebuild. If you get the latest branch that is already there. So:
git clone git://sigrok.org/libserialportcd libserialport
./autogen.sh
./configure
make
sudo make install
However, be sure you remove libserialport, libserialport0, and libserialport-dev or equivalent so you are not still using the distro-provided version. Eventually, of course, the distros will include the new version…. sigh.