Although the Internet of Things (IoT) is a reasonably new term, the idea isn’t really all that new. Many engineers and hackers have created networked embedded systems for many years. So what’s different? Two things: the Internet is everywhere and the use of connected embedded systems in a consumer setting.
Like anything else, there’s a spectrum of usefulness to IoT. Watching The Expanse, the other day (which is not a bad show, by the way), I noticed that if you had the right IoT lights, you could run an app that would change your lighting to suit the show in real-time. I don’t have those lights, but I suppose when the action moves to a dark sub-basement, your lights dim and when you are in a space ship’s reactor room, they turn red, and so on. Fun, but hardly useful or life-changing.
On the other hand, there are some very practical IoT items like the Nest thermostat. It might seem lazy to want to monitor and control your thermostat from your tablet, but if you are frequently away from home, or you have multiple houses, it can be a real positive to be able to control things remotely. With the recent blizzard on the U.S. east coast, for example, it would be great to turn on the heat in your weekend cottage 150 miles away while you were still at work or home. However, the Nest recently had a hiccup during an upgrade and it has made many of their customers mad (and cold). I’ll get back to that, in a minute. First, I want to talk about the problems with deploying something that will be in many varied environments (like people’s homes) that controls something real.
Good Intentions
Problems arise, though, when you consider that programmers (and sometimes hardware guys) are relatively optimistic. How many times has a Windows update broken something on your computer? Linux used to be better, but lately, I dread updates, especially major ones because they sometimes will stop my machine from even booting, triggering a big debugging session. The Mac, I’m told, has had similar upgrade horror stories.
In the old days, a bad update to a piece of software meant, perhaps, that payroll checks wouldn’t go out on time. That’s a disaster for some people, of course, but it is a survivable one. Maybe you couldn’t get e-mail for a few hours. You’ll live. Once you start connecting to the real world, though, things get more complicated and riskier.
Chip on a Bus
We all have stories where our assumptions during development don’t match up with reality. Years ago, I developed a system that had three boxes, one at a central station and two remote boxes. Periodically (or on command) the remote boxes would call the central one using some modems (this was definitely a few years ago) and send their data.
The system worked great on my bench, and I sent a team 500 miles away to install them. One of the remote boxes was very remote, indeed, and sat on a newly set phone pole in the middle of a swamp. The installer called me from that very phone line to tell me the system didn’t work. I was surprised, to say the least. I knew the phone line had two circuits, and he was using a field phone to call me from the secondary line, so I asked him to trigger a call and hold the phone up to the box.
I heard the dial tone I expected. Then I heard a series of TouchTones. Then I still heard the dial tone. It hit me: the phone line in the middle of the swamp didn’t have TouchTone dialing (and couldn’t have it)! The EPROM that controlled the dialing had the string ATDT in it, where the last T was for tone dialing. It needed to have ATDP (for pulse dialing). A quick fix except, in those days, the only way to get that to them quickly was to put it on a bus.
My point is, when you deploy to strange places in the real world, your assumptions get tested. Sometimes tested hard.
Safety Dance
If you work on systems that are known to be dangerous (like weapons or airplanes), there is a lot of effort to make absolutely sure that things work the way you expect. This goes for updates, too. You can’t just make a change, do a quick test and send it out into the field. Even so, sometimes bad things get out.
Even if your system works great in the lab (like mine did), you can still get unexpected problems during installation or just the environment (like my phone line). Airbus found that out the hard way last year.
Getting development right isn’t the only thing when you are rolling out IoT systems. You have to test, of course. But you also need to test over a broad range of environments and circumstances. Even then, you won’t get them all. You need to think about how to do updates in the way that is least likely to break. It is acceptable to roll back to the original version of things, but it is not acceptable to break during an update.
Security Dance
Then there’s security. If you can update something in the field–especially over the network–how can you be sure an update is legitimate and not an attack. Digital signatures, encryption, and other techniques can do that, but how many of us worry about things like that.
Updating your copy of a new video game isn’t that big of a deal. However, depending on the operating system, a rogue update could, in theory, go off and steal your credit card numbers or passwords. Or it could install a Trojan horse that puts your computer on a botnet that sends spam and attacks other computers.
That’s bad enough. What happens when that software update controls your washing machine which now spews water out in the middle of the night so your house floods? Or your dryer overheats and catches on fire?
As end users, we have a vested interest in knowing our IoT devices are safe, even after an update. We also should worry that the update is legitimate.
Empty Nest
If you think this is overblown, think again. I mentioned the Nest earlier (that we saw rooted a few years ago, by the way). A recent software update caused thermostats to run their batteries down in some cases, requiring a restart or even a recharge to get them back in service. This isn’t a theoretical problem. It affected many users who woke up to cold houses. People who could not easily get to a remote thermostat were especially peeved.
If you are feeling smug because you have a custom non-IoT thermostat built around an Arduino, don’t. If you have a car with a modern keyless entry system, there are vulnerabilities there, too. If you see someone getting their car keys out of the freezer, you’ll know they read about that. Granted, that’s not a bug, it is a criminal attack, but it points out that IoT is opening the door to many unintended consequences.
The Nest case isn’t the first (nor will it be the last) of an IoT update breaking something. As developers, we have to think beyond our workbench when deploying systems or software. The wider the deployment, the harder you have to think about all the cases, all the chances for exploits, and how to recover when it happens. The alternative is going to be government regulation like certain industries already have. The cost of getting things right will pale in comparison to complying with strict regulation from a government agency.
Sparkfun had a great post about a year ago about fancy devices failing for whatever reason and thereby being unable to provide basic function (https://www.sparkfun.com/news/1674). So, in the spirit of these posts, an open letter to engineers / product designers everywhere:
Dear Maker;
Please, PROTECT THE CORE FUNCTION!
Thanks.
i want to make a sticker that says that and stick one on every project box as a reminder
Years ago Science Fiction author Arthur C. Clarke write a story called “Superiority” that pitted a high tech civilization against a relatively primitive one. The high tech one kept shipping advanced and even more advanced technology to their troops and never getting the bugs out of them. Spoiler – They lost to the low but proven tech civilization. For years this story was required reading at MIT and other such colleges to show the consequences of shipping tech that was not tested and debugged before distribution.
For some reason, “Airbus found that out the hard way last year.” links to a Nest support page.
Oops. Fixed. Not sure what went wrong there. Um.. .hackers… yeah, that’s it.. .hackers!
Now it links to a registration-required news article. Why not just link to Wikipedia?
https://en.wikipedia.org/wiki/2015_Seville_Airbus_A400M_Atlas_crash
This just so perfectly shows how easy it is to make mistakes when trying to set up an internet of things.
Looks like the Airbus link in the article actually leads to the NEST reset page.
why does a thermostat need a battery?
it’s fixed to the wall isn’t it, that’s the point of controlling it from your phone?
why isn’t it plugged in?
pathetic.
In North America thermostats normally have a 120v hot line coming in, a neutral and an output going to the heater but in other markets, Nests come with a battery because of how the houses are wired according to local code. I don’t remember where, but in those places, the thermostat doesn’t have a 120v line coming in, but only a signal line that tells the heater somewhere else to start/stop.
That’s 24VAC (the “C” wire), but your point is valid. It goes way back to when a thermostat was just a dry contact switch on a bimetallic strip. Temperature changes, strip bends, makes contact, furnace runs its routine.
Wrong wrong wrong. Its generally Low voltage AC coming from furnace to power Tstat.
Not nesecarily
My ancient central heating thermostat (Lennox brand) is battery powered and just closes a contact to turn on the heater.
Sorry, this is incorrect. Historically, thermostats did not need a constant source of power. They were simply a pair of wires connected to the coil of a relay in the furnace, fed 24VAC from a bell transformer (so called because the same transformer also powers the doorbell), and this pair of wires is intended to be shorted by a temperature-operated switch when heat is needed. Closing the switch pulls the relay, which turns on the fuel source, ignites it, and makes you warm and toasty.
When the switch is open, there’s a 24VAC differential between the wires. But you can’t simply draw current from it, because the current runs through the relay – if you pull too much, the relay would close and your furnace would turn on! And when you do close it, there’s no power differential to draw from.
Nest thermostats are designed to work with these old two wire systems. What they do is constantly leech a small amount power from the 24VAC differential in the open wires, and use it to charge a battery. They are careful to not draw so much that the relay would close. The battery is there so the CPU can continue to operate on a smooth stream of power, regardless of current availability on the wire, and to provide additional power for when the WiFi radio is transmitting or when the display is lit up.
Of course, that was the olden days. Not every thermostat is still a two wire job. More wires have been introduced over time. A third wire is usually present to operate a fan, pump, or blower, separate from the heating element. Another wire may control the air conditioning pump; another may be present to control the humidifier; a two stage heat pump/furnace (common in colder regions) will operate the heat pump on the traditional heat wire, where a second wire controls the gas furnace; and so on.
More modern installations include a C wire, which carries a constant 24VAC from the furnace transformer, intended to power smarter thermostats. The 1970s saw the introduction of clocks in thermostats to control setback times, but most of these required batteries.
But the C wires are not ubiquitous, nor are they even required. The only constant seems to be that heating contractors pulled exactly the number of conductors they needed because of wire costs. They would never pull a 5 conductor wire when a cheaper 4 conductor wire was enough. So unless the installer was hanging a smart thermostat the day he installed the wire, you will never find a spare wire for C when you need one.
I installed 12 smart thermostats today. A/C split systems with gas furnace (1980s era). Remarkably, every one of them had an unused C wire twisted around the bundle behind the stat, and again at the furnace. Even more remarkable was the original stats were digitals running on unnecessary batteries all these years.
I’ve never seen more ignorance and laziness in any of the industries I’ve worked in more than HVAC.
I must assume there is less concern over copper costs in recent years, though, as I’m seeing lots of newer homes with 8 strand installed on systems only utilizing 4 or 5 strands, all on original equipment.
Couldn’t agree more! And don’t start with the back up memory / time setting nonsense. I had a great thermostat 15+ years ago that did all kinds of magic (7 day / multiple programmable, heat pump / multistage / ac / etc, +/- 1/2 C, amazing PID algorithm that got the house up to temp on time and didn’t overshoot – old house, large windows, massive iron radiator heated water system).
It did NOT have a battery. Not even a soldered in NiCad. It had a decent cap, non-volatile memory and a very good power management strategy to keep it going through multi-hour power failures.
I swear device manufacturers are in cahoots with battery makers. ‘Cause not only do things like thermostats require batteries, but AAA ones at that! (come on – 1/2 the power for double the price of a AA, and no perceivable space savings…).
You are lucky your house had the wiring for that. I grew up in a house that had two wires going to the thermostat, closing them turned it on. Rewire it? The interior walls, even the ceilings, were plaster. Couldn’t put in an A/C because of that, be I remember helping put a giant exhaust fan into the ceiling upstairs to vent the whole house out through the attic; there was plaster dust in that house all winter til we turned that fan on the next year.
Nah. It’s 2 simple pressures from management:
1) HURRY UP
2) Make it thinner
– a product designer
You can get all that functionality in a thermostat that runs for a year on a battery or by sucking the control AC because it’s really low power, but when you put WIFI in “really low power” goes out the window. An ESP8266 draws 100 to 200 milliamps at 3.3V and that’s the bottom end of wifi functionality.
The Nest thermostat has a battery because it’s powered by parasitic power from the furnace in many or most cases. If you are extra diligent, though, you can add a “common” wire to your setup. This wire supplies the opposite pole of the 24 vac transformer secondary and is intended for exactly this situation – where your thermostat needs line power. I did this a while ago because the parasitic power arrangement – for whatever reason – was flaky for me. With this configuration, the battery inside the Nest is entirely moot.
Well not really parasitically powered just strictly powered by those batteries and adding the common wire is exactly right. I was talking to a hvac (heating and air) guy the other day when I asked exactly this question about line powered devices. He told me that most homes don’t have the common wire and sometimes even if they do it isn’t hooked up to the transformer.
No. In the absence of the common wire, the batteries are charged with parasitic power drawn through the switched circuit lines. As you can imagine, the amount of power you can draw this way without actually turning on the furnace is rather small. Ordinarily, this is ok because the Nest sleeps a lot. But, as I said, adding the common wire moots the whole issue.
Right you are just read the nest product info kinda neat Thanks :)
My thermostat is wireless. It’s not IoT, but it’s wireless. Replaced a wired one when my boiler was upgraded as part of a rebuild. I’d have been happy with a wired one, and the manual suggests they expect you to leave it permanently fixed to the wall, so I’d guess it’s just to make it quicker to install – and the price of batteries is pretty low compared to the time for a workman to run a wire through the wall all the way across the house.
However, it’s great to be able to move the thermostat to another room sometimes; it lived in the baby’s room for a few weeks, as that was the most important room to keep the temperature stable.
Really, houses should have a zone and thermostat per room, but I guess it’d be pricy and unreliable.
Here central heating is the most common heating method, so every every building connected to heating system has radiator in every room, and mechanical or electronic thermostat attached to the radiator inlet. This method can be used with any heating configuration…
Programmers are lazy, security is mostly not paid for – so either isolate them Sensors/Actors/Gadgets into an Intranet or expect them to do their own (manufacturers or hackers) thing.
Recommondation: Put all those nice things into an intranet, place a trustworthy gateway on the edge between intra and internet like a OpenVPN box or a trustworthy homeautomation like FHEM or OpenHAB. ONLY allow sensors/actors to contact your intranet ressources (this network edge point), do not allow direct external connections. If remote access is needed, use OpenVPN and set the routing tables properly and then you can still control everything of those gizmos remotly. Never be so stupid to fall for the comfort like IFTTT or other proprietary servcies you can only trust but not control – there is no reason to trust them lazy coders…
Me two cents, Cheerio!
Sadly, some industrial IoT devices will fail after some time if you do that. In many more the support contract will no longer be valid if the company cannot access it and something went wrong (they will say they were planning to solve it before the problem happened, but couldn’t due to your ‘configuration error’)
Support contracts could be invalidated because the supporting companies couldn’t proactively connect to your equipment and update it? I’ve never heard of a situation like that before in my entire life.
Then you have a pretty narrow view of the world. I’ve not only encountered contracts like that — I’ve written them. You want me to be responsible without access? Go pound sand…
Yes jonfrei – be responsible without access. Such contract situations have in tandem with an utter lack of security responsibility negate your point.
Not hardly. Some of my clients provided keys, some an on-call “gatekeeper” to let us in, and others gave remote access. Some provided all of the above.
jonfrei – Appears we are comparing apples to oranges. Your clarification hints at a more commercial application vs. a general topic of consumer grade “IoT”. In my line of business we have managed services/warranty support similar to what you mention. Most consumers do not have an “on call” contact, etc. And most of these IoT consumer grade focused companies do not have the processes in place to actively notify customers their webcam needs security updates, etc. Even home routers are severely lacking Day 2 support in comparison.
I will agree with that.
Tell you what, why don’t you just build a reliable product *in the first place* that doesn’t require you to have access to it once it’s my property (because you’re not going to get that access). Good lord, what is wrong with companies and their hardware designers these days?
I don’t want a manufacturer to be responsible for poking it and prodding it to keep working forever (just so you can conveniently “oops, I guess it’s time for someone to purchase an upgrade” at some later date). What I do want is for manufacturers to *fricking build it right the first time*, test it thoroughly, and take enough goldang pride in their work that it’s going to keep working (correctly) on its own, no matter what! If something should fail *unexpectedly* within a certain number of years of my purchase of your product, that’s why there’s a warranty. But if that product is built right and properly QA’d, very few of those products will ever have to come back in for warranty service in the first place.
Too hard/impossible to do that, you say? Bullcrap, I say. Plenty of companies have been able to do that for a very long time, and somehow they’ve still survived. It’s absolutely possible. Companies that say it isn’t possible are unwilling to pay for good employees who are able to produce good products while keeping costs to their users at a realistic level, usually because those companies have bought into the greed-fueled line of crap that is “shareholder value” (or sometimes because of pure incompetence, but that’s another matter). Those companies all deserve to go die in a fire.
If that’s too dang hard for a company to wrap their heads around, then they sure as heck ought not to be building that hardware in the first place!
Now get off my lawn!
Programmers are usually not lazy, they just won’t work for free. It’s the management over the programmers who bark the features they want, but don’t like to spend a dollar more to optimize the code. That’s what updates are for :\ its a broken system usually, the programmers are usually on the bottom end of the problem.
Thank you, I’m glad someone said it, as I was just about to write the same thing.
That said, HaD does sometimes feature some products with appalling software security vulnerabilities, that any halfway decent developer’s mind shouldn’t even be able to conceive producing
QED, there’s some crappy and/or poorly paid devs churning out insecure tat, which now and then a “reputable” company gets caught selling
There’s a certain logic to getting it to market quickly (making profit, beating competitors), and then using the profit to supply updates.
But yes, security is usually something no-one wants to pay for, but will always wish they did after it mattered.
Having constant Internet access has meant computer games, even on consoles, are now shipped out well before they’re ready. Doesn’t matter if they’re bug-infested pieces of shit, we’ll fix it in an update!
The result is software full of bugs, not even properly beta-tested. I don’t fancy the same ideology applying to my appliances. If something’s fixed in ROM, a manufacturer either makes a proper effort to get it right, or ends up with a terrible reputation and panics before going out of business. This is the correct way!
Can you imagine having to ring the tech-support robots every time you try to make a cup of tea, or use the washing machine? Because companies were competing to get their shit out the door, make a huge pile of money, and decided to issue actually working software later? That’s WORSE than hackers!
There’s a certain godawful logic to it, and now the IOT fad, if it takes off, will bring it into everything we own. Nice!
I agree with a previous poster, all this nonsense really needs a proper gateway, having appliances directly on the Internet is a terrible idea. But I suppose an average consumer doesn’t want to pay for a gateway that they won’t be able to figure out how to work, and don’t understand what it’s for anyway.
You wait til some genius figures out a way to send adverts to your vacuum cleaner. Just you wait!
Ideas similar to Internet of Things originated during the World’s Fairs of the 1930s. Westinghouse Electric started home automation projects in the mid 60s. and the term “smart house” was first coined by the American Association of Housebuilders in 1984. We have been lurching toward this for some time.
This is why when I consider building my own thermostat.. There will be some old school mercury based thermostats in parallel at the max limits (60 min 85 max).
So a worse problem I have seen in the wild.. My furnace failed to read that my igniter worked. Ok fine, don’t blow up the house,that is good… But all it did was blink a red LED. Which is fine unless its -35F outside (true story). Thank goodness we were in a town house and leached off of unit to unit heating. Our place stayed above 65 while I went and fixed the ignitor.
Audible alarms would have been really really nice!!!
I think you’d be better with bi-metal strips than mercury. For reliability as well as reduced deadliness.
As far as heating, when mine went I took the edge off the chill by leaving the oven on, doors open. For an electric oven that’s fine. Didn’t do the same with the gas rings, combination of condensation and CO2 wasn’t something I fancied.
a little mercury never hurt anyone…
Not to necessarily re-beat the dead horse, but the article that talked about the nest being “rooted” doesn’t imply that it was remotely hacked or anything. The firmware was replaced over a diagnostic interface on the back of it expressly designed for that purpose.
It doesn’t really count as a vulnerability if you have to have physical access and partly disassemble it to exploit it (unless the device is intended for unattended operation in a hostile environment, like a vending machine or a land mine).
That escalated quickly
Can you trust the guy who installed it?
Since that guy is me, the answer is “yes.” At least, I think so. Ask me again tomorrow.
Al, you should know it isn’t just turning on the heat up north. Down here it’s turning on the A/C for Galveston beach house. I had a long conversation with a guy in the late 80s about doing that. I proposed he put a lamp under the thermostat that could be turned on by a phone call. That would kick on the A/C while they drove down to the house from Houston. The reverse wold work up north. Leave a lamp on and turn it off with a phone call.
Echelon. 1980’s. Founded by one of the three guys at Apple. Still in business I think. I recall the development systems was expensive and they would not talk to you unless you had a really big deal going, like controling everything in a new high-rise.
They are still around mainly as a provider for chips that talk their protocol “LONWorks”. It’s rather annoying to deal with as there are fees for the hardware, then fees for the software, and then more fees if you forget to de-register a device on your end before you ship it to the customer (who pays fees to register it again). You can however now get service from them on just about any scale, it just doesn’t make much sense for a home user.
At least something like modbus, or bacnet is accessible. Lots of folks on this site could probably hand code modbus packets as it’s a pretty simple protocol. It has no security other than keeping everything else off the network, but it’s normally used on a serial line, so it’s somewhat hard to get into.
And these are ex-Apple guys, you say? Never would have guessed.
Somehow in science fiction films they always have a switch to manual (override).
Always always always always make it work without the interwebs.
.
IOT being new….that’s equivalent to a few years ago everyone saying the cloud storage was the latest, greatest upcoming thing that would change the world, without realizing online storage had been around for many, many years prior….still makes me laugh.
Yup. Home automation has been around since the 1940s at least. Nobody really WANTS to turn their lights on and off from the Internet. We have light switches. It’s a solution for a problem that nobody has.
Hopefully that’ll save us from the hell that’s modern software, invading our homes.
Grenaum, you are right in a way. No one wants to manually turn off lights from the internet. What people may like is for lights to be automatically switched on, dimmed, brightened or shut off, thru intelligent sensors
Amazingly designed, wonderfully featured, intricate & expensive crap is still crap when it breaks. Just like cheap crap is still crap when it breaks. What’s your Failure Mode?
lmao the shortest distance between two points is 8 servers and 3 routers…got it. The IoT is a source of much of my amusement and will continue to be so apparently.
These IoT companies are akin to an attack of your home by an army of hipsters.
They’ll invade your home and kill all of its resale value.
Hm, that tone-dialling EPROM… Just had a look, and the difference between ASCII “T” and “P” is one bit (the 4s bit). Needs zeroing to turn a T to a P. I realise this is probably a little late, but you program EPROMs by changing their default 1s into 0s, right? So you can re-program an EPROM as long as you only want to insert 0s, not 1s.
So a bit of faffing about with wire, and it could have been done by some guy in the field with a chip socket and a few wires.
Probably not practical, at all… but technically possible! And that’s the best kind of possible.
I remember using a similar technique to salvage a six-month supply of EPROM’s. We needed to change something minor in the binary and it luckily only required zeroing a few bits. It was more work to figure out how to make the checksum come out right (code checksummed itself on powerup). We had unused space left that was all 0xFF, so we just zeroed enough bits there to make the calculation work.
That was actually considered but they were in a swamp with practically nothing.
NASA has done an impressive job of remote upgrades – in some cases with spacecraft millions of miles away.
Boils down to test. Was the release well tested? or was the update rushed out because a big customer or marketing really really wanted it out the door quickly
Test is *often* considered second fiddle to core development in small to medium size companies as it doesn’t directly generate revenue, but lack of test can very quickly lose revenue/reputation. (although Nest should have scaled by now). Testing current usage over a few cycles/configuration to predict battery life isn’t hard, so I’m not sure what the excuse is.
With automated updates rolled out to millions of devices you can’t cut corners when it comes to test. Accidentally brick millions of thermostats, that could easily break a company (maybe not google) but somebody would be out of a job.
J. Peterson comment on NASA, he is absolutely right, I’d guess they will have some of the tightest test plans, some out of this world. (sorry)
Why is it that no one is talking about the economic loss to the end user of these wonderful IoT devices? Since these devices all require a connection to the mothership for full functionality, if the vendor goes out of business or pulls their servers offline, you now own a BRICK! Consider also that not one of these vendors will be honest about it.
None of the IoT devices will actually last long enough to worry about that. 10 years and all the smart thermostats and wireless lightbulbs will be broken and obsolete.
How many 1980’s home automation systems still work, and how many have been simply jumped over with wire because they stopped working and there were no spares?
I think a lot of original systems still work, and since most were relay-based the spare parts weren’t that hard to find.
Check this one out:
http://imgur.com/gallery/Jb6jW
> Linux used to be better, but lately, I dread updates, especially major ones because they sometimes will stop my machine from even booting, triggering a big debugging session.
I can’t tell if you are upgrading the kernel (always has been fraught) or using a terrible distro.
Internet Of Shit ™ approved post
Lets just forget about the “internet of things” – that just seems like folly. It should really be all about the “network of things” and around here that means a /local network/. Anyone connecting their hvac system directly to the internet deserves to have their temperature raised to 78 while they sleep.. Kidding, nobody deserves that.