In this installment of Minimal MQTT, I’m going to cover two loose ends: one on the sensor node side, and one on the MQTT server side. Specifically, I’ll tackle the NodeMCU’s sleep mode to reduce power and step you through bridging MQTT servers to get your data securely out of your home server and into “the cloud”, which is really just other people’s servers.
If you’re just stepping into this series now, you should really check out the other three posts, where I set up a server, then build up some sensor nodes, and then flesh-out a few ways to control everything from your phone or the web. That’s the coolest material, anyway. This last installment just refines what we’ve built on. Let’s go!
Using Less Power
The ESP8266 units are awesome. They pack a decent amount of computing power in a tiny package with WiFi. That makes everything easy. Everything except battery-operated use, that is. Without spending some effort on taming it, the ESP8266 is a power hog.
Let’s quantify that really quickly. I’ve connected the ESP8266 to my power supply through a roughly one ohm resistor, and am measuring the voltage across it; every millivolt on the scope is a milliamp of current passed through the ESP. The first scope shot shows the high current draw during startup, the second zooms in on a standby pattern, and the third shows startup again with the voltage scale zoomed out so that you can see the nearly 350 mA peaks.
When using WiFi, the ESP8266 on my desk draws a roughly constant 70 mA from the 5 V supply. (As many people have noticed, though, it occasionally peaks up to 350 mA for milliseconds at a time, which is what causes brownouts with poor power supplies.) While it’s connecting to the base station, it uses quite a bit of power. Once it has connected, it settles down to a cruising altitude of roughly 15-20 mA, but still exhibits the bumps and spikes. We’ll call it 20 mA on average.
There’s not all that much we can do about the WiFi standby and transmission current requirements — physics demands power to make electromagnetic waves. For an application like a long-running temperature and humidity monitor, however, we can reduce the duty cycle — taking measurements every ten minutes or so instead of every ten seconds and powering down into sleep mode for most of the time. The less time the ESP spends awake, the longer our batteries will last.
To put all of this in perspective, I’m going to frame everything in terms of a 1000 mAh battery because it makes the math easy and is about right for a small Li-Ion pack. If it’s just idling, that’ll be 50 hours — we’ll call it two days. Sending data will only shorten that lifetime. That’s no good.
But if we can take infrequent reports and turn it off in-between, we’ll save on most of that constant 20 mA. Imagine that a connection and temperature report takes about ten seconds, and measurements are made every ten minutes. Because this includes the high-current connection phase, let’s assume something like 50 ma average across the ten seconds. That would only be 20 hours of run-time if it were on constantly, but since it’s on only 1/60th of the time, it could last as long as 1200 hours, or 50 days. That’s getting somewhere.
Sleep
But we can’t quite turn the ESP8266 off: what would turn it back on again? Instead, it can go into a deep sleep mode which shuts everything down except an alarm clock that will wake it back up. When my ESP8266 is in deep sleep, it only uses around 145 microamps. (I’ve seen reports as low as 78 microamps, and I assume the difference is standby consumption of the built-in voltage regulators on mine. Or mis-measurement.)
Putting an ESP8266 with the NodeMCU firmware into deep sleep mode is very easy — just use the node.dsleep() function and tell it how many microseconds you’d like it to sleep. Easy enough.
There’s one catch, with two consequences. To wake up from deep sleep, the ESP8266 needs to reset itself, and to do this pin D0 / GPIO16 has to be connected to the reset line. (It’s the one labelled wake in the pinout diagram.) This is necessary because the only thing left running in deep sleep mode is the real-time clock (RTC). Just before entering sleep mode, the RTC is configured to toggle the D0 / GPIO16 after the elapsed time.
Connect your node to your computer and test it out right now. Don’t forget to hook up a wire from D0 to the reset line. Type in node.dsleep(10*1000000) and watch the chip become entirely unresponsive for ten seconds and then spring back to life.
If you followed along, you just experienced the second consequence: the chip completely resets itself after deep sleep. This means that each time it wakes up, it’s forgotten everything, and it is starting over again with init.lua and a clean slate. For something like a simple sensor node that does the same thing each time it wakes up, this is no problem. But sometimes our processes will need to store some state.
Memento
Life for the ESP8266 is like the movie “Memento”. And if you saw that movie, you’ll remember that the solution is to leave yourself good notes. Short of adding non-volatile memory to your setup, you can abuse MQTT’s persistence feature to stash data for reboots. Designate a topic (or two, or more!) to store the data that should persist after deep sleep. Before going to sleep, write the data to the topic using the “retain” flag and wait a few hundred milliseconds for the message to be delivered. On waking back up and connecting to the MQTT network, subscribe to home/powertest/memento. The first value read from that topic is going to be the data left over from the last session.
As an aside, there is a function, node.bootreason(), that you could in principle test to see if the chip came out of a power-up or a deep-sleep reset, but there is a hardware bug and workaround that prevents this from functioning correctly. The upshot is that you always get the same return code, 2, and that’s essentially useless. Bummer.
A workaround for the bootreason() bug can be found in the powertest example in my MQTT GitHub repository. That code also demonstrates setting up status and command channels to talk to the node. To play around and see how it works, open up a window to listen to the MQTT traffic: mosquitto_sub -t home/powertest/# -v -h 192.168.1.49 and then send a command to the node: mosquitto_pub -h 192.168.1.49 -t home/powertest/command -m "sleep".
To conclude: the trick with using deep sleep on the ESP8266 when running NodeMCU is to write your code so that it does the right thing on reset, and stores any long-run data somewhere. And don’t forget to connect the wake pin to the reset pin — the chip will stay asleep until you pulse the reset line.
Peripherals
For the temperature sensor nodes, it’s worth mentioning that they’re going to be battery-drainers in the standard configuration. My powered-up DHT-11 module draws a continuous 486 microamps — let’s call it half a milliamp. That’s a big dent in our energy budget now that we’ve cut the duty cycle down by sleeping. We got our average energy consumption down from around twenty milliamps to just under one milliamp. It’d be a shame to spoil it by leaving an unnecessary peripheral device burning half of that away.
The same often goes for other peripherals, sensors, and LEDs. Get rid of them if you don’t need them, and reduce the on-time to the bare minimum for those that are necessary. A useful method is to power the peripherals through a PNP transistor or P-MOSFET and turn that transistor on using a GPIO pin on the node.
In the end, you should be able to get a few months from batteries if you’re careful and making infrequent measurements. If you need the node to be always on, or need longer life, a 2.4 GHz radio module and a MQTT gateway might be just the ticket.
Thinking About Security
Finally, I got savaged in the comments at the beginning of this series, because some readers mistakenly thought that, just because something can be put on the Internet, it must. And this would mean that our sensor nodes need encryption. But that’s the wrong approach, in my opinion.
The system we’ve built up so far — a Raspberry Pi talking over a home WiFi router to a bunch of ESP-based nodes — is about as secure as you could ask for. All communications are encrypted and authenticated by your WiFi router, and constrained to the local area. Indeed, half of the reason to run your own MQTT broker at home is that your data never leaves the confines of your own four walls unless you need it to. Not connecting devices to the Internet that don’t need to be connected to the Internet is security rule number one. But don’t take my word for it, see what Shodan — troll and entertaining presenter Dan Tentler has to say.
As it stands, everything is (hopefully) behind your router’s firewall and encrypted in transit by WPA2 with a good strong random password, right? And you turned off UPnP? Great. That’s good enough for any but the most paranoid. Or you can pound your head against encryption on the ESP8266 if you want to, but since Espressif has announced that they’re not coming out with TLS 1.2 support until 2017, you’re probably wasting your time.
Configure MQTT to Bridge
Still, you might want to get your data out into the big wide world, or you might want to control your home system when you’re away. There are tons of neat services out there that speak MQTT. And if the data is even the least bit sensitive, you’re going to want to encrypt it in transit.
MQTT broker bridging does exactly what we need in this case: it duplicates topics between brokers. By bridging, we can take our home network which is already secured through our WiFi’s WPA2 encryption and access control, and securely forward data on to another MQTT broker “in the cloud”. In this case, we’ll use the test.mosquitto.org server that we used in the first installment.
To bridge a topic, we need to add a new configuration file on our home Pi-based MQTT server. In this example, I’m calling the file /etc/mosquitto/conf.d/bridge.conf:
connection TestBridge address test.mosquitto.org:8883 topic home/outdoors/temperature both 0 bridge_cafile /etc/mosquitto/ca_certificates/mosquitto.org.crt
The “connection” names the bridge and provides a client id for the remote broker, so pick something unique and descriptive. The “address” is the remote broker’s address, and in this case port 8883 is the port that test.mosquitto.org uses for SSL/TLS connections.
For every topic that you’d like forwarded between the two brokers, you’ll need a “topic” line. In this case I’m forwarding my temperature topic in “both” directions (“in” and “out” are the other choices) and using QoS 0. See the manual for mosquitto.conf for all the bridging options. Since the bridge is really functioning as a client on the remote broker, you’ll find that most everything works intuitively.
Finally, since we’re using TLS to secure the connection, we need the remote broker’s certificate authority file. You’ll have to get this from whatever remote service you use. In this case, I downloaded it here and saved it in the certificate authority directory that came with the Mosquitto installation.
After a sudo /etc/init.d/mosquitto restart everything should be good to go. Try publishing to the temperature topic on either server and you’ll find it appearing in both places. Voilà, your local network has been securely forwarded on to the cloud, without exposing the individual devices that lie behind your Pi-firewall. That’s the easy way to do IoT security. And if you bought one of those new Raspberry Pi 3s with WiFi built in, you could even run the IoT network on the built-in WiFi.
Wrap-up
I really like MQTT, if you haven’t guessed. It’s a data-agnostic transport mechanism with a little bit of memory that allows you to do a lot with a little. You can very quickly hack together a system that lets one device control another, integrate data from multiple sensors, and wrap it all up nicely with a web interface (for instance). It’s an open standard, with an implementation for any programming language or platform you can think of.
There are a lot more ways to expand this system outwards. We haven’t yet touched on storing, analyzing, or summarizing the data — that’s the job of applications on your server that can listen in as clients to specific topics. An MQTT-to-REST bridge is a few lines of Python code, and again you can encrypt it outbound however you like. Once you start writing server-side client scripts, you’ll find that creating triggers that depend on other inputs are pretty easy and soon you’ve got your own, very flexible version of IFTTT.
A nice but not very well known feature of the ESP8266 is the ability to save a small amount of data in the RTC memory, which is persistent between deep-sleep cycles. This enables in example collecting a few values without sending them immediately, saving a lot of wifi power (as stated, the connection setup draws a lot of power).
A nice video demonstrating this:
https://www.youtube.com/watch?v=r-hEOL007nw
Good resource! I’m hardly an ESP expert, so thanks! 512 bytes of RTC memory is pretty handy, and NodeMCU looks like it supports it too: http://nodemcu.readthedocs.io/en/dev/en/modules/rtcmem/
I still think it’s cute to store the data in “the cloud”. You’ve got WiFi / MQTT. Use it. :)
You can also use the “retain” flag to issue commands to the sleeping node that you want it to execute when it wakes back up. Store/forward is the perfect complement for low-power sleepers.
I didn’t get too much into using MQTT as a control channel for the nodes, but it’s super handy, especially for devices that are inconvenient to reach, etc. I did toss a little into the code this time, because I couldn’t resist, including a ping/pong demo. I find the ability to ping nodes even more useful than MQTT’s last-will functionality. But I’m going far afield here…
i just stumbled on your comment and i am hoping that you might be able to help me. I am using nodmecu to control sensors and am using mqtt broker to send instructions to the nodemcu, my problem is that when using nodemcu wit sleep mode its doesn’t resubscribe to the mqtt broker after waking up from sleep an thus doesn’t receive my intopic. I would like to know the reason for this issue and if there is any solution to it. thank you in advance for any useful information
Good question! I’m nowhere near a NodeMCU expert — I only used it b/c it was the quickest way to cobble something together.
But: https://nodemcu.readthedocs.io/en/master/en/modules/node/#nodesleep has a callback function that you can invoke when the WiFi comes back online from a sleep. So I would try wrapping all of your MQTT subscription stuff into a single function (“mqtt_sub()”?) and passing that function as an argument when you put the thing to sleep.
Again, I haven’t tried it, so maybe start with something like turning off an LED and then turning it back on when it wakes up, just to be double-sure the mechanism works. Then sub in your subscription function and you’re good to go!
thank you I will try to use that
great article for learning about MQTT and the limitations of the venerable ESP, but I cannot help think that for this use case something like a Moteino would be much better. You can get 6 months plus from a 9v battery with a Moteino transmitting temperature readings every 15mins and sleeping in between.
Yup. And (IMO) the really right way to do a sensor network like this is a bunch of 433-900 MHz radio nodes and a radio-MQTT bridge. The Moteino or Jeenode or whatever is all good. If you really over-engineer stuff, you can get by with a button cell.
But for a super-quick deployment that’s just about good enough, it’s hard to beat a four-buck ESP8266 and some batteries. I was totally anti-WiFi (b/c of power budget) until I started playing around with it. There’s a convenience factor that’s hard to deny, and batteries are rechargeable.
Yup, that’s a great point. And if you want something more common to not need a custom bridge, I’ve been using Bluetooth Low Energy as well for sensor nodes. Less range, but with most phones and tablets being able to read BLE there’s some convenience there too.
What I’ve done is encode my sensor data (plain or encrypted) into the advertisement packets that are sent over BLE to not require a connection to the sensor node. You just listen for BLE broadcasts kind of like you would for iBeacons. Been getting over a month of life with a broadcast every 2 seconds from a tiny CR1220 battery and Nordic’s nrf51822 chipset.
At some point i thought about doing the same thing. Any idea how much range you could get if the receiver had a nicer antenna?
Well that depends on your power budget. BLE chips will let you decide how strong of a signal you’d like to emit, but of course the stronger it is the more energy is used for the transmission which shortens your battery life. At the +4dBm maximum I’ve gotten a range of 30+ meters through a wall or two in my home. But I usually don’t need that much and extending battery life has been more important.
This was just with a phone and laptop, so I wasn’t really able to tune the receivers.
@Elliot Williams With local server and manual IP, how fast can you power up send data and sleep?
Can the RTC memory store the calibration data and do calibration…say just 1 of every 10 transmissions? (skip that 300ms startup calibration).
It is actually not that hard to beat the $4 ESP. It is just more user effort and more complicated. For $4 you can get a micro of your choice + some 2.4GHz low power radio. But you need a gateway with a big antenna, choose which micro, design a bigger pcb to hold both, more soldering etc.
I can totally understand why people can choose to simply swap batteries on the ESP more often than doing all that .
@Bogdan
I read on the Internets that you could speed up association with manual IP address specification, but I didn’t find that to be the case. I got association in 5-6 seconds pretty reliably regardless of static/dynamic IP. The first time with a new device would take longer (10s-ish). Maybe DHCP lease something-something? I haven’t looked into that at all.
So roughly a second or so startup + 5-6 for association + 1-2 for data transmission. It might be 7 sec, might be 10. The variable part is “only” running 20ma or so, so a second here or there doesn’t change much.
WiFi is a double-edged sword. The radio needs juice and the channel takes a while to set up. But it gets you security and simplicity/compatibility for free.
I would love to see the ESP8266 being used as a MQTT bridge for a cluster of nodes using low cost nRF24+ clone modules.
https://forum.mysensors.org/topic/1870/esp8266-wifi-gateway-port-for-mysensors/2
Thanks for the link, but it is not a MQTT gateway. The only MQTT gateway on ESP8266 is done by Mickey who was commenting and has no pointer to his project. He has some links to a starting point in the comment. The “mysensors” framework as pointed out in the thread is C++ based and not so easily ported to work with MQTT on the same ESP.
oh, you mean like MQTT-SN gateway? Haven’t seen people do that or that many implementations of MQTT-SN nodes. I think it will still burn too much battery because the nodes still need to do way more communication than sending a packet when they feel like it.
Or one could use the eeprom on board?? Too simple? Yes, probably not something you should write every cyclez, but perfect for persisting less volatile state or config info
I just adore the ESP8266 and have been working up a battery powered temperature monitor. Not using MQTT, but it talks via TCP once a minute to a server (quickly thrown together using ruby) running on my linux machine, sleeps except when taking data. I want to put the RTC memory tips to use …. thanks! I will note that the DHT22 clone I use (the RHT03) consumes only 50 uA in standby and then 1.5 mA once you wake it up to do a conversion. Now I presume it goes back to sleep when done, but should check. I also presume the DHT22 is similar. Dunno about the DHT-11.– I anticipate this running for 6 months or more in the final version from a single 18650 Li-ion cell harvested from an old laptop.
I have a few ESP nodes using MQTT for my home automation system. They have proven to be very reliable over the last year.
This little MQTT series has been great, almost a small book load of content shared, thanks for the effort you put into it Elliot.
Thank You Elliot for this short MQTT series. I had been postponing implementing some of the automation at home which I wanted to do for a long time. MQTT is perfect for my requirements, I have the broker running on my home server and I am slowly adding devices to the network. Thanks for getting me started on this.
I know this maybe a little outside the topic, but it is power saving, so its related..
I can seem to find any documentation of if its able to use a pin change interrupt to wake the mcu, increment a counter and then sleep again ?
I would like to use the esp8266 to count pulses, and transmit them every so often, but cant find any one doing the counting from sleep ? :)
Have any of you worked with pin change inturrept during sleep ?
I want the esp8266 to count pulses, and transmit them every so often.. (once every 2-3 min)
But cant find any using pin interrupts while sleeping :(
Not sure. You can certainly use the reset pin to wake it up though. Can you workaround to get your signal to pulse that line? Maybe a diode-or with the RTC clock?
I can do that yes, and then just some logic to find the reset fault, can I maybe see if the it was wakeup timed or reset pin ?
Use a low power ATtiny to count the pulses and tell the ESP when it wakes up and interrogates the tiny?
Thats my thought too, but would be nice to use a single mcu :)
The ESP is a system on a chip anyway, but yeah all the required logic on one bit of silicon would be more convenient, then it is just a software issue for you.
It depends on how hard it is to set up the logic, else the attiny13 would be easiest, I think :)
The ESP has to be in a quite high power mode (mA) to be able to be woken by any pins. You can add a binary counter such as the CD4060 and then read that upon wake every 3 minutes. Needs pins, but no programming.
If there just was a serial out chip like this one :D
I cannot say I know one. You can always use that and a I2C GPIO chip to read the counter.
But to be fair, slapping a micro near it will get you more stuff: you can even do wake up on pin change without much power, have adc etc.
Thank you, [Elliot] for that series. This will get me started with that topic! Perfect timing ;-)
I also have to add my thanks Elliot for this series I have been working through it to create a sensor network for monitoring my Solar hotwater system
The easiest way to avoid doing something is to handwave it saying “you don’t need that” – I strongly disagree that there’s no need for _proper_ security. One might have a need to keep an open WiFi, or common subnet with the wired LAN (all one needs is one of those routers that turns out to have a hardcoded backdoor and an attacker who notices that), or simply have guests accessing the “secure” WiFi that one wouldn’t like to see mess up the garage door opener or the water heater. There’s no need to fiddle with messy certificates and signatures left right and center to do it either: Mosquitto supports TLS with pre-shared keys quite nicely – but how one can get the ESP8266 to play along is anyone’s guess. Unfortunately, even after this series ended.
Depends what you mean by security and how skilled you attacker is. You can always jam the wifi and then no TLS or LAN separation will save you.
I assure you its much easier to avoid doing it because I don’t care.
Getting an ESP/Arduino/small chip to play along with big-computer networking security protocols is a hassle when it’s even possible. So you have two simple choices: 1) keep them off the Internet 2) end-to-end encryption.
Get a Raspberry Pi3. It has its own WiFi. Restrict it to only device traffic. Then you have a segregated, secure device network with only one point of (arbitrarily secure) contact for $40 and no more work. Done. This is, IMO, the best way to do a home network.
If you insist on running data over insecure channels, why not end-to-end encryption? The NodeMCU security module is buggy ATM — the version I have the hashes work but not the AES encryption/decryption — and so I didn’t mention it.
But honestly that’s what I’d do in your shoes: hard-code a long-long key into the ESPs and your Rasbpi. There are tons of known/tested ciphers that you could code up/port in an afternoon.
@Elliot Williams there is a typo in the text, the lowest current is 70uA, not 7.
Thanks!
What iz MTQQ
Hi Elliot, many thanks for this MQTT tutorial briefly. I have a question about accessing local MQTT server from outside of local LAN such as internet. Can we handle it via port forwarding, instead of bridging local MQTT server with other one ?
thanks in advance,
Taner
Great Tutorial, well staged and clear. Many thanks!