Although the Internet of Things (IoT) is a reasonably new term, the idea isn’t really all that new. Many engineers and hackers have created networked embedded systems for many years. So what’s different? Two things: the Internet is everywhere and the use of connected embedded systems in a consumer setting.
Like anything else, there’s a spectrum of usefulness to IoT. Watching The Expanse, the other day (which is not a bad show, by the way), I noticed that if you had the right IoT lights, you could run an app that would change your lighting to suit the show in real-time. I don’t have those lights, but I suppose when the action moves to a dark sub-basement, your lights dim and when you are in a space ship’s reactor room, they turn red, and so on. Fun, but hardly useful or life-changing.
On the other hand, there are some very practical IoT items like the Nest thermostat. It might seem lazy to want to monitor and control your thermostat from your tablet, but if you are frequently away from home, or you have multiple houses, it can be a real positive to be able to control things remotely. With the recent blizzard on the U.S. east coast, for example, it would be great to turn on the heat in your weekend cottage 150 miles away while you were still at work or home. However, the Nest recently had a hiccup during an upgrade and it has made many of their customers mad (and cold). I’ll get back to that, in a minute. First, I want to talk about the problems with deploying something that will be in many varied environments (like people’s homes) that controls something real.
Problems arise, though, when you consider that programmers (and sometimes hardware guys) are relatively optimistic. How many times has a Windows update broken something on your computer? Linux used to be better, but lately, I dread updates, especially major ones because they sometimes will stop my machine from even booting, triggering a big debugging session. The Mac, I’m told, has had similar upgrade horror stories.
In the old days, a bad update to a piece of software meant, perhaps, that payroll checks wouldn’t go out on time. That’s a disaster for some people, of course, but it is a survivable one. Maybe you couldn’t get e-mail for a few hours. You’ll live. Once you start connecting to the real world, though, things get more complicated and riskier.
Chip on a Bus
We all have stories where our assumptions during development don’t match up with reality. Years ago, I developed a system that had three boxes, one at a central station and two remote boxes. Periodically (or on command) the remote boxes would call the central one using some modems (this was definitely a few years ago) and send their data.
The system worked great on my bench, and I sent a team 500 miles away to install them. One of the remote boxes was very remote, indeed, and sat on a newly set phone pole in the middle of a swamp. The installer called me from that very phone line to tell me the system didn’t work. I was surprised, to say the least. I knew the phone line had two circuits, and he was using a field phone to call me from the secondary line, so I asked him to trigger a call and hold the phone up to the box.
I heard the dial tone I expected. Then I heard a series of TouchTones. Then I still heard the dial tone. It hit me: the phone line in the middle of the swamp didn’t have TouchTone dialing (and couldn’t have it)! The EPROM that controlled the dialing had the string ATDT in it, where the last T was for tone dialing. It needed to have ATDP (for pulse dialing). A quick fix except, in those days, the only way to get that to them quickly was to put it on a bus.
My point is, when you deploy to strange places in the real world, your assumptions get tested. Sometimes tested hard.
If you work on systems that are known to be dangerous (like weapons or airplanes), there is a lot of effort to make absolutely sure that things work the way you expect. This goes for updates, too. You can’t just make a change, do a quick test and send it out into the field. Even so, sometimes bad things get out.
Even if your system works great in the lab (like mine did), you can still get unexpected problems during installation or just the environment (like my phone line). Airbus found that out the hard way last year.
Getting development right isn’t the only thing when you are rolling out IoT systems. You have to test, of course. But you also need to test over a broad range of environments and circumstances. Even then, you won’t get them all. You need to think about how to do updates in the way that is least likely to break. It is acceptable to roll back to the original version of things, but it is not acceptable to break during an update.
Then there’s security. If you can update something in the field–especially over the network–how can you be sure an update is legitimate and not an attack. Digital signatures, encryption, and other techniques can do that, but how many of us worry about things like that.
Updating your copy of a new video game isn’t that big of a deal. However, depending on the operating system, a rogue update could, in theory, go off and steal your credit card numbers or passwords. Or it could install a Trojan horse that puts your computer on a botnet that sends spam and attacks other computers.
That’s bad enough. What happens when that software update controls your washing machine which now spews water out in the middle of the night so your house floods? Or your dryer overheats and catches on fire?
As end users, we have a vested interest in knowing our IoT devices are safe, even after an update. We also should worry that the update is legitimate.
If you think this is overblown, think again. I mentioned the Nest earlier (that we saw rooted a few years ago, by the way). A recent software update caused thermostats to run their batteries down in some cases, requiring a restart or even a recharge to get them back in service. This isn’t a theoretical problem. It affected many users who woke up to cold houses. People who could not easily get to a remote thermostat were especially peeved.
If you are feeling smug because you have a custom non-IoT thermostat built around an Arduino, don’t. If you have a car with a modern keyless entry system, there are vulnerabilities there, too. If you see someone getting their car keys out of the freezer, you’ll know they read about that. Granted, that’s not a bug, it is a criminal attack, but it points out that IoT is opening the door to many unintended consequences.
The Nest case isn’t the first (nor will it be the last) of an IoT update breaking something. As developers, we have to think beyond our workbench when deploying systems or software. The wider the deployment, the harder you have to think about all the cases, all the chances for exploits, and how to recover when it happens. The alternative is going to be government regulation like certain industries already have. The cost of getting things right will pale in comparison to complying with strict regulation from a government agency.