While searching for a connector recently, I revisited an old project of mine called the Serial Squid. This was to have been my first open-source hardware design. After completing the entire design, PCB, BOM, and preparing for a crowd-funded campaign, I eventually gave up for reasons discussed below, I’ve always thought of this as a failure, but on further reflection I see it in a new light. There were some good lessons learned along the path to abandonment.
When do you let go? When should you push through?
An Idea Forms
The Serial Squid was to be essentially an Ethernet-connected Bus Pirate. The motivations for this project were many. Too many, in fact, and perhaps that should have been an early warning sign. At first, I just wanted a remote serial UART, and this out of laziness. It annoyed me to carry my laptop from my desk to the lab bench for debugging, or carrying the project to my desk. Surely there must be some way to physically extend my office computer’s I/O across the room or building to the lab bench over the network. So I focused only on serial UART solutions, and in fact had some modest success sharing a second compter’s serial port using socat. But I gradually expanded the scope of my notional widget to include other serial interfaces of interest to the embedded developer.
Around this time, I had stumbled upon a small handheld flash programmer from China called Eagle Comm. It was designed for a variety of MCUs such as PIC, Cortex, AVR, etc., and was super easy to operate. It was perfect for small production runs at the factory or to send to customers to upgrade prototype boards. The more I used it, the more I liked the form-factor and wondered whether I could repurpose it. But the manufacturer wasn’t willing to share the code, and the hardware wasn’t that good a fit for my application to warrant reverse engineering it. But maybe my still-unnamed widget could learn a thing or two from the packaging and layout of the Eagle Comm unit.
Speaking of sending something to a client for flashing a chip, it would also be helpful to have something I could send some of my clients to use for data logging, especially when a system was installed on a remote location with no network access. Could my widget be battery powered and easily used as a logger?
And finally, I had been intrigued for years about packaging an electronics in a flat “enclosure”, made entirely of layers of PCBs. A suitably thin SMT board could be made into a flat package by adding a few fiberglass PCBs, some with cutouts for components. Connections and help could be silkscreened directly on the outer layers, and assembly would be as simple as screwing the layers together.
Specifications Solidify
Gradually this product began to gel in my mind. Ideally, I wanted a small handheld device with a keyboard and display, and the provision for several I/O modules corresponding to each kind of interface. If you’ve ever held an old HP-41C calculator, imagine the plug-in modules being I/O blocks, for example. Alas, this approach eluded me for some time. There wasn’t a perfect intermediate bus for these modules. I wanted something that was not only technically suitable, but also low-cost, license-free, and that wouldn’t disappear next year. I was circling around Ethernet or SPI, when I hit upon the solution: defer a modular design until version 2, and go with a fixed set of ports for version 1.
I was having a tough time with the form factor. Making a half-decent keypad wasn’t going go be easy. And I was considering crowd sourcing, and if I was going to get any traction, a simple calculator-like layout was just plain boring. I wanted my widget to have a catchy look. I pursued several candidate designs with mockups and CAD, but none of them were quite right.
Then one day it hit me, something suggestive of a common food here in South Korea — my widget was going to be a squid. Each arm of the squid would be an interface port, and the head would naturally be the brain. Ten legs would be plenty. Now I could offer almost every serial protocol known. My newly-named Serial Squid would be really versatile.
After several explorations of squid-like shapes, I settled on the design below. I went through an exhaustive exercise to find the best connectors and pinouts for each interface.
Do Your Research
This part of the design took a really long time. Many serial protocols don’t have standard connectors. Sometimes we take for granted those everyday things we use like the TTL-level serial connector pinouts or what connector and pinout should an RS-485 interface have. (There’s no standard, but a fairly well adopted de-facto one.) Knowing the related specifications, history, and common usage can be quite helpful.
I researched as many different implementations as possible from other projects and products. Eventually I selected the most appropriate connector and pinout for each of the interfaces.
All the time I was trying my best to keep the “flat PCB” construction technique in mind. Because of the physical constraints of many of the connectors, some compromise was needed. This would not be a sleek, slim PCB sandwich, but I still feel the result would have been acceptable for the intended application. I was also leaning towards a 3D printed case instead of a PCB sandwich, because the stackup of PCBs needed were getting quite complicated, and I suspect more expensive than 3D printing.
The Death of a Squid
I abandoned the project after almost a year. I was so close. I had the Gerbers ready, the PCBs were quoted from China, parts were in my shopping cart ready to order. There were lots of reasons why, but three main ones come to mind.
Killer #1: Software
While I was cheerfully doing hardware design, the dark clouds of firmware development were looming. For this board to perform as I envisioned, I needed firmware that would present its various serial ports to a remote computer over Ethernet and possibly USB. While this didn’t seem too daunting when I thought about a UART, when you toss in a dozen other interfaces you have a problem. The biggest one being no standard xxxx-over-LAN protocol existed for these ports, nor did computers even have most of these ports natively. The enormity of the firmware task slowly sank in, not to mention possible desktop device driver development, and my enthusiasm began to fizzle out.
Killer #2: No Clear Purpose
Many people I discussed the project with admitted it was catchy and novel, but it was difficult to explain what it did. “It can do anything” was an unrealistic answer, and an honest evaluation led me to conclude it was trying to do too much.
Killer #3: Distracted by Rev 2
I was never quite happy with the the ten legs having fixed I/O ports. Ideally, I wanted the user to be able to mix and match the legs. This way I could make the body smaller, perhaps moving away from form factors based on 10-legged creatures. Four or five legs would be plenty. I kept debating with myself whether I should wrap up the squid first and then move on to Rev 2, or drop the squid and leapfrog directly to a new design. I did neither.
Lessons Learned: Feature Creep
Over the years as an engineer, I’ve experienced the mental letdown when a project ends or gets cancelled. But interestingly, I didn’t feel that way upon giving up this project. I had such a good time doing the design, learned or re-learned a lot of lessons, and gained some new skills along the way.
Beware of broad scopes and feature creep. This is well known by everyone in the community. But despite being a gray-beard, I was tripped up. Try to recognize the warning signs of a bloated specification. And if you see it happening, try to fix it or quit before you get in too deep. The fact that nobody could understand the purpose was a big red flag — I saw it, but ignored it anyway. Also, be flexible, know when to give up or abandon on a feature. Don’t be too rigid in the specifications, especially if it’s a hobby project.
I quickly abandoned my idea of the Squid appearing as port extensions on your desktop. Had I pressed on, I was going to either replace the micro with a Linux SBC, or run a scripting lanuage like Python or Forth on the Squid. If you find yourself having to re-invent or build new protocols from scratch, take a close look. There might be valid reasons to do so, but more often than not, you will be better off adopting existing interfaces and standards.
Don’t underestimate the value of mockups, either physical or simulated in CAD. I used this project as an opportunity to learn OpenSCAD, a skill that I’ve found quite useful. I also made a decision to get up-to-speed on KiCad for this project. Having used several professional PCB design tools over the years probably helped, but I could see that KiCad was going places and I needed to become proficient with it.
Be honest with yourself — brutally honest. As eye-catching as the Serial Squid is, I have to admit that I’m not sure how comfortable it would be to use in real life. Also really look closely at the problem you’re trying to solve. In hindsight, 99% of the use cases I was looking at solving could have been done with a Raspberry Pi and a custom shield or two.
Although strictly speaking the project was a failure, I had a lot of fun doing it. I met new people and companies, and learned new skills that I’m still using five years on. If you have a new project in mind, use some common sense. Bounce your ideas off of colleagues and fellow hackers. Then jump in and start. And don’t be afraid of a flop — even a failure can be valuable.
” At first, I just wanted a remote serial UART, and this out of laziness.”
A lot of great ideas developed from laziness. Someone got tired of doing something, and thought up an easier way to do it.
I have been immersed in feature creep for about the last week. I wanted a way to access a bluetooth (BLE) device without having to log in to the vendor’s website or use a phone app(*).
I’ve previously done this a bunch times (for other devices), and got tired of typing gatttool commands and pasting hex chars into text files and conversion apps so I decided to make a program that would just dump all the BLE info.
Then, a program that would scan all devices and dump the one selected by the user…
And allow for multiple BLE adapters on the system…
Gee, it would be really nice if the system could automatically parse the known generic services and characteristics…
Gee, controlling this thing is ‘kinda complex, should have a GUI…
Gee, a RasPi is ideal for all this, and can be used to control a suite of BLE devices…
Except, you want to be able to access the devices when the RasPi is in a remote location…
The RasPi can put up a website with a GUI allowing the user to scan/explore all adapters, devices, services, and characteristics…
The interface can keep track of old values, so you can do something on your device and see what changed…
But you also want to send individual commands (by local program) to the RasPi to control things…
I’m currently in the middle of that project, A project that turns a RasPi into a remote bluetooth explorer with all the bells and whistles, and allow for remote (ie – operator not physically there) control of said devices.
I expect to be done in about a week. (Double that for a more accurate estimate…)
I’ll throw it up open source on guthub and .IO when it’s done. Maybe it’ll save some other people some time, because it’s sure as hell taking up a lot of mine right now…
(The app, for a smart thermal mattress insert, wants to know my location. Yes, the app for a device that goes between my mattress and the sheet needs to know my location to work. Also, cannot be used without logging into the vendor’s website, or without using the phone app.)
weeel…
a lot of people have turned off their location on their phone because of privacy concerns or battery management and grandma google does not like that. making it an requirement to use wifi or ble solves grandma google’s problem.
am I being paranoid? maybe, maybe not, but google desperately needs to have my routers global coordinates.
“Just because you are paranoid, doesn’t mean someone isn’t out to get you.”
-saying I read long ago
i believe this is from kurt cobain….
A note Google is not your friend, nor is the local police. You might think “I’ve done nothing wrong”, however THEY DON’T CARE.
See this article (https://hackaday.com/2020/03/25/geofence-warrant-sends-mans-privacy-over-the-handlebars/) from last year of what does go wrong with geo location data Google “harvests”, there is something called TMI and I suspect Google has leapt over that a long time ago.
All apps that use BLE on Android need the Location permission. The app in questipn isn’t necessarily nefarious.
See: https://www.polidea.com/blog/a-curious-relationship-android-ble-and-location/
I have assumed location permission is need because Bluetooth Beacons can be used in learning your location, so Android is covering their bases.
And to think that now, so many of those protocols are available on one chip…
https://hackaday.com/2021/01/08/bare-metal-stm32-universal-asynchronous-communication-with-uarts/
Indeed. In my design, too. It was based on the NXP LPC1768, another Cortex licensee who packs a bunch of peripherals on their chips. I was using every single peripheral on the chip. In fact, at the end I was using so many pins for communications that I didn’t have any left for setup control (terminations, any MUXed ports, etc). I ended up using a chain of shift registers controlled by an second auxiliary 10-pin microprocessor for configuration.
If I did this today I would probably use something from the STM32F4 family. For reasons I’m not clear, it seems to have overtaken the NXP family in popularity.
” For reasons I’m not clear, it seems to have overtaken the NXP family in popularity.”
I think their cheap development boards have a lot to do with that.
hah this seems to be the opposite of how i abandoned the one project i imagined mass-producing.
rather than leave some part undeveloped, i developed the whole thing in prototype scale. so there was no vague cloud hanging over my head. i feel like enough software decisions need to be made at the one-weekend breadboard mock-up phase to get a sense for the project. but then!
but then i had a working prototype. i had the 1x that i needed to fulfill my own hypothetical kickstarter pledge and i found that suddenly all interest in designing it as a mass-produced product had evaporated! lol
Discernment is a valued quality in engineering. I just like working on stuff and in much of my early career “couldn’t see the forest for the trees”. Still I’m glad you shared your process. There’s a lot of solid engineering there; the kind that gets projects past the idea phase and into tested shippable phase. I’ll bet a lot of developers will appreciate the experience.
What a delightfully comprehensive insight into the mind of an innovator. Thank you for the honesty of your self analysis.
It’s interesting to see how there are still parallel paths in electronics, repeating themselves over the ages. Many, many years ago, my company produced a 6502 based unit which had two pairs of serial, parallel and IEEE ports, one set down each side of the box. The application was, of course, to connect data arriving on any one of the interfaces to one of the others with the least possible hassle. We never made a fortune from it but it sold reliably for nearly 20 years until boredom set in with the manufacturing industry – and I stopped being able to keep up with the frenetic advances. What I did learn from the product was that users found far more applications for it than we had ever considered because it enabled them to get on with solving an interface problem rather than reinventing low level interface wheels. I can see why existential anxiety caused this project to be abandoned but I suspect that it would still have sold as a hardware platform for others to use to enable their own desperate interfacing fantasies, especially if the price was right. As many people have said about hardware design, ” …. the best is the enemy of the good enough”.
And, on a completely different matter, could someone please explain what the difference is between a “use case” and an application. I’ve obviously missed something somewhere but I can’t quite see what.
For an example, your six port product. Its application is to connect any port on one side to any port on the other side.
The use cases are what people do with it. For example someone has a data logger with an IEEE port but they don’t have a computer with an IEEE port so they wedge your box in between the data logger and their computer’s serial port.
Wedge? Yup. That comes from early barcode scanning wands that ‘wedged’ between an AT or PS/2 keyboard and the keyboard port. Like driving a physical wedge between two objects.
Another use case for one of those port boxes might be to connect an RS232-C port multiplexer to an IEEE to IEEE cable then to another port box and another RS232-C port multiplexer. People did things like that to connect two groups of computers in the days when ARCnet and Ethernet and TokenRing were expensive.
Another example. The application of a box truck is to carry a large amount of stuff. The use cases are what goes in the truck and where it’s picked up from and delivered to, or perhaps the truck gets loaded with a couple tons of electronics and sensing equipment to be used by NEST for finding nuclear stuff.
The application is still the same, carrying a large amount of stuff.
A “Use case” is a small, simple story: a case of using the product. For a pickup truck example, some might be:
– User drives at rated maximum freeway speed for max. rated duration.
– User accelerates from a stop.
– User changes lanes
– User decelerates to a stop from maximum rated speed, N times in a row (brake heating / regenerative braking)
– User loads sheet goods (e.g. 4′ x 8′ sheets of plywood) into bed
– User refuels the vehicle
Etc. etc. leading to all sorts of design requirements (engine power, brake design, fuel efficiency / tank capacity, visibility, steering, …)
An application would be a (more or less) specific usage:
– Commuting (paved roads)
– Tow a trailer (vacation trips)
– Off-road use, up to X% slopes, up to Y MPH
They’re related, but from different perspectives. An application is likely to include multiple use cases, which are also referenced by other applications.
So, in a way, all issue reproduction steps in issue reports are use cases, but, for non-bricked devices, there are also use cases which will reproduce no issues.
When to pull the plug on a project is an important skill, and a challenging one!
My favorite characterization of how to recognize a doomed project is this one, which I will share below:
Every project has some tasks that end up harder than expected, but also some that end up easier than expected.
If EVERYTHING in a project is turning out harder than expected (and nothing is turning out easier), the project is very likely doomed.
“If EVERYTHING in a project is turning out harder than expected (and nothing is turning out easier), the project is very likely doomed.”
Oh great! I should just abandon getting my 1942 Ford Truck running again!
B^)
One can likely add the line:
“unless one thinks the project in itself is fun to do.”
Python has Asynchronous and Multi-threaded comm libraries. Maybe you missed out …
I would say to write All the software first before designing any hardware. Write stubs or simulate the hardware you don’t have, or make it modular so it will work with your prototype and your product. You will probably discover many hardware changes during your software writing. Heck, new stuff will come along as you write and debug. If you hold off on hardware design until the end, then you can take advantage of the new stuff.
I went through the same sort of issue. Best thing I came up with was a raspberry pi running linux that I could plug into console ports, ethernet, and USB. Used VNC to get to it.
Advantages: could handle tftp, ssh, scp connections and could run wireshark and other utilities.
Disadvantage: best to have local power to avoid underpowering over USB and a lot of utility programs from various manufacturer tended not to have great linux support.
I did like that I could leave them connected to problematic remote systems and have kvm and various other connectivity options. Nice that systems are moving to USB management but no so nice that they often need proprietary utilities and the USB does not become availalbe until a system boots.
Manufacturers should start moving toward embedded ethernet interfaces with their own operating systems like mpst server systems have now. I realize the low priced equipment cant affort to do this but it is inexcusable in network equipment that costs many thousands of dollars and have out of band managment ports that do not come up until booted. It is mind boggling how many systems I still need to talk to initially on a 9600bps serial before I can bring them online. Systems that do not dhcp on their out of band ethernets are the bane of my existance.
Hardcoded TCP stacks are a really bad idea, bugs are found all the time, We need to be able to update the software immediately when an update is available. Of course this requires enormous resources and unless you have one of those fabulously expensive service contracts, you are not going to be able to get software updates and your device with all your hard work invested is just another infected mess.
And of course you can look back at historical perspective to see quite readily that even the “professional” companies don’t update their fancy expensive gear, so your expectations are not realistic.
Really it is good that you are forced to deal with the serial port because you can set up something much more secure.
That’s pretty much what everybody hates about the Intel Management Engine though, out of band accessibility with no capability of audit for bugs, exploits and backdoors. “Just trust us” doesn’t fly.
Current loop interface with baud rate negotiation at the start and after ‘x’ number of comms failures, and an eBay Arduino Pro Mini (the bad-fax-machine-copy of the one with the USB-to-TTL circuitry hacked off the side) programmed to be a knockoff Bus Pirate.
Job done.
Current loop interfaces have stupidly insane levels of long-distance coverage potential if you’re willing to crank the speed down to somewhere around the rate of someone yelling “ONE! ZERO! ONE! ONE! TWO — OH WAIT THAT’S AN ERROR! ONE!” through a megaphone lol.
Or turn the voltage up until the deep sea critter trying to chew through your cable gives it up.
What would do the job now is an Ethernet connected device with a bunch of programmable pins and an interface to various interface modules that would each have their own firmware to handle translation between what they’re connected to and the Ethernet half.
The Ethernet to/from half would just take data from the adapter half and packet it up to send off to the software running on the other end of the cable. Data coming back would be un-packeted by it and pushed to the adapter half as commanded by firmware in the adapter half.
Yet another hardware application that would benefit from adopting the method Texas Instruments used for their 99-4 and 99-4A computers. The OS firmware in the console only ‘knew’ about the hardware in the console. All other peripherals contained firmware called a Device Service Routine. The console had a set of ‘ports’ in memory, into which the DSRs ‘plugged in’. Entering OLD DSK1 without a floppy drive controller generated an error, but with the floppy controller the OLD SAVE and other commands and DSK1, 2, and 3 device names were available. The floppy controller DSR also contained all the software required for basic disk and file operations, but a DSR could be strictly hardware support while software had to be run from disk or cartridge to use the peripheral.
I think it’s crazy that this design didn’t take off. Connect new hardware and it works. Disconnect it and it’s like it was never there. No drivers to install and potentially muck up the operating system. If there’s a problem, disconnect the peripheral and the problem goes away.
With an Ethernet connected port adapter there wouldn’t be conflicts because it would only have one ‘slot’ so every adapter would plug its code into the same port in the memory space of the Ethernet half.
Wouldn’t want to have it able to plug in more than one adapter because then that’s an expansion bus and it’d need more than one memory port, adapters would need to be assigned ports and the adapters would need to be able to work with more than one port… it would become a little headless microcomputer instead of a simple Ethernet to whatever the hell anyone can think up to plug into it device.
Creeping Features
Dilbert had the answer first: https://dilbert.com/strip/2001-4-14
Reminds me of this. https://xkcd.com/1406/
You basically want a Glasgow (https://github.com/GlasgowEmbedded/glasgow) over the network.
Which could admittedly be done quite easily by hooking it up to a raspberry pi, though I fear the applet synthesis types which is already pretty long on a beefy desktop…
Feature creep kills a lot of my projects or keeps them at Rev. 1 for a very very long time.
Sounds like a good project to develop “as-needed”.
I need UART over ethernet today?
Sure. Pick an oversized micro with UART but also lot of extra memory, extra pins and the ability to communicate via whatever other interfaces you think you might want later. Also, pick an ethernet solution. Throw it on your favorite prototyping board along with the apropriate level shifter if needed. Write ONLY the software for the UART part. Try to structure it so you can add commands to enable other sorts of ports later.
Tomorrow you need SPI? Pick different pins that don’t conflict with the existing UART solution. Extend the existing code.
Don’t design a case or a PCB until much later when time and chance has lead you to build support for several different kinds of ports.
Share your documentation online every step of the way. Maybe somebody else will have a need for port X in the future and do it for you.
Or just throw a bus-pirate, pi-zero and battery in a box and call it a day. That’s probably the path I would take.
Plus low-battery indication.
After 3 days of hunting intermittent errors, re-checking the software you’re developing, maybe it’s the hardware?, a loose cable?, “why is this *(#$@¿?! thing being so flakey!!!” An hour of troubleshooting later: “Oh.”
Instead of putting everything into the main chip, maybe make the “legs” have their own micros which handle various formats. For example, for CAN bus you would anyway need a transceiver. That way, everyone can design their own extension and you can just start with the basic ones that you need.
I was contemplating USB-over-IP for a while but did not find any good and free solution. The other option, less flexible, would be to use OpenOCD or sigrok over the network.
Still waiting for the ‘swiss army knife’ solution to this, I never found out if anything better than the Bus Pirate was developed (with good support!)
it seems SocketCAN might be useful in this situation. It turns a USB CAN Adapter into a Network interface. It is already supported in most GNU/Linux Distributions and there is already an implementation for the STM32:
https://github.com/candle-usb/candleLight_fw
The SocketCAN interface sends a single CAN frame at a time and while its not a serial interface, you could send 1 byte CAN Frame at a time and have the microcontroller decode that into the serial bus.
This would be a good idea to use as a basis for some SocketSerial interface that supports the most common low level serial interfaces. If the protocol is aligned with SocketCAN then it would be possible to extend the interfaces to multiple types: UART, RS-485, SPI, I2C, (multiple network channels can be created for each implemented interface with one device).
Its even possible to use SocketCAN drivers & libraries to get that up and running.
Doesn’t MOXA literally make a product that does this for most serial interfaces?
I know the one I use has 232, 485, etc.
They make ethernet versions and usb versions
I just realized the final selected shape is missing from the article. For the curious:
https://hackaday.com/wp-content/uploads/2021/01/failure.png
In that orientation, it looks less like a squid and more like the outline of the head of a daft punker!
B^)
Maybe i not understand the result you want, but you want UART over ethernet/wifi? You hear from esp link: https://github.com/jeelabs/esp-link
I have now the esp as bridge for my UART device that tell to the com ports with a Virtual Port tool, whats “perfect” for me now i can use the software from the manufactur right out of the box^^
Not the OP here, but I’ve tried esplink and had trouble talking to a 3D printer. I suspect there are some timing/signaling issues, but at that time (3 years ago) I could not debug it. Just my 2c, trying to say that some years ago esplink wasn’t actually transparent.
ok about transparency, i dont know^^ but for me i want a sensor thats talk over spi or uart to the controller but its an light sensor so i wanted to be remote, but the original tool needed an com port to work and for me the esp link got me a well solution so if the OP mean something like this maybe he could get something from his solution^^
(my programming skills are not the best :D)
Esplink cuts your transmission into packets and then sends them. On the output side you see bursts of data instead of continuous stream. If your device expects continuous transmission or very long packet it is likely to throw an error/time out when it sees the spaces. That was the problem I had with Esplink. Not to mention it was loosing data at anything above 115200.