Last time on Minimal MQTT, we used a Raspberry Pi to set up an MQTT broker — the central hub of a home data network. Now it’s time to add some sensor and display nodes and get this thing running. So pull out your ESP-8266 module of choice, and let’s get going.
For hardware, we’re using a WeMos D1 Mini because they’re really cute, and absolutely dirt cheap, but basically any ESP module will do. For instance, you can do the same on the simplest ESP-01 module if you’ve got your own USB-serial adapter and are willing to jumper some pins to get it into bootloader mode. If you insist on a deluxe development board that bears the Jolly Wrencher, we know some people.
NodeMCU: Getting the Firmware
We’re using the NodeMCU firmware because it’s quick and easy to get running. But you’re not stuck with NodeMCU if you want to go it alone: MQTT has broad support. [TuanPM] ported over an MQTT library to the native ESP8266 SDK and of course there’s espduino, a port for an Arduino-plus-ESP combo. He also ported the MQTT module to NodeMCU that we’ll be using today. Thanks, [TuanPM]!
Anyway, back to our story. If you haven’t tried out NodeMCU on at least one ESP8266 module, you should give it a try. Here’s the painless way. Head over to NodeMCU custom builds and build yourself a custom firmware with only the modules compiled in that you need. Here, I’ve included dht, file, gpio, mqtt, node, tmr, uart, wifi, and ws2812. (I’ve also included this image as a download in the GitHub repo for this series.)
To upload the NodeMCU firmware to the ESP8266, I use esptool.py, which should work on any platform that Python runs on. Follow the instructions on the esptool GitHub, or just esptool.py --port /dev/ttyUSB0 --baud 57600 write_flash 0x00000 firmware.bin if you’ve done this sort of thing before. (Windows, use COMx for your serial port. Mac people use /dev/cu.usbserial-whatever.) If you have an ESP module without USB-flashing support hardware, you might have to pull GPIO0 to ground on restart — you might have a “flash” button that does this. Again, the instructions at the esptool GitHub should get you squared away.
NodeMCU: Getting Around
If you’ve never used NodeMCU before, don’t fret. It’s a little bit confusing at first because many functions run asynchronously. We’ll handle that when we’re looking at the code. Other than that, Lua is not a difficult language to learn, and you could do worse than to have Programming in Lua open in another browser tab.
The cool feature of NodeMCU is that it’s run through an interactive serial terminal, so plug up a USB-serial converter if your ESP module doesn’t already have one onboard, connect at 9600 baud, and you’ll be greeted by a friendly > prompt. To get the module registered on your local WiFi network, try:
wifi.setmode(wifi.STATION)
wifi.sta.config("ESSID","password")
Wait around ten seconds and you should be set. Type print(wifi.sta.getip()) to confirm that all’s well. We’ll upload a script to automate this before we’re done, but it’s good to see how simple and interactive things can be right from the start.
MQTT in NodeMCU
So let’s get a little experience with MQTT on the NodeMCU platform. The online documentation is pretty good, so open that up to follow along. The MQTT library centers around the mqtt.Client() object and its methods, connect(), publish(), subscribe(), and close(). When you create the client, you can also register some callback functions that you’d like to be triggered by certain events, namely connect, offline, and message for when the client connects, goes offline, or receives a message.
If you’ve got your broker set up from last time, let’s send it some messages and monitor them on the broker. Open up a window on the Pi and type mosquitto_sub -h localhost -v -t home/#.
The node’s MQTT client needs an ID, keepalive time in seconds, username and password, and a clean-session flag. Connecting to the server can be as simple as specifying its IP address, and publishing takes the usual topic, message, QoS, and retain flags. Here’s a minimal demo:
m = mqtt.Client("myNodeName", 120, "", "") -- blank user and password
m:connect("192.168.1.49") -- my local broker's IP
m:publish("home/test", "hi from node", 0, 0) -- no QoS, not retained
Bam! Your first MQTT publish from an ESP8266 in three lines of code. With the addition of some callback functions, we can make it user-friendly and start receiving MQTT data as well.
m = mqtt.Client("myNode", 120, "", "") -- blank user and password
m:on("connect", function() print("connected") end )
m:on("message", function(client, topic, data) print(topic .. ": " .. data) end )
m:connect("192.168.1.49") -- my local broker's IP
m:subscribe("home/#", 0) -- no QoS
The client code continually listens for messages or state changes, and sends the appropriate strings to the client:on() function that then runs your code when the named event occurs. It’s a lot like the way Javascript uses callbacks, if you’ve done any of that.
Now if you send a message to the broker, mosquitto_pub -h localhost -m "heya" -t home/test, you’ll see it printed out at the node’s serial terminal. There’s a bit more to say, but that’s the gist. You’ve got publish and subscribe functionality on the ESP8266. And if you’d like to subscribe to or publish data to the bigger Internet, there’s nothing stopping you. (Although note that everything you send to test.mosquitto.org is public by default.)
m:close()
m:connect("test.mosquitto.org")
m:subscribe("hackaday/#", 0)
m:publish("hackaday/mychannel", "your message here!", 0, 0)
Let’s work this base into a realistic two-node, home-automation example.
Sensor Node
The WeMos module I used has DHT-11 and DHT-22 shields available. The sensor is hooked up to NodeMCU pin D4, which is GPIO2 in the ESP8266 datasheet. The NodeMCU DHT library docs for using the DHT units are short and sweet, with one function that you’ll care about, dht.read().
Anyway, if you’ve got a DHT-11 or similar hooked up to the NodeMCU pin D4 (ESP GPIO2), it should just work. print(dht.read(4)) returns five values: status, temp, humidity, temp_decimal, humidity_decimal. Verify that, and you’re set. You can wrap it in a function like this:
temp = 20
function updateTemp()
status, temp, humid, temp_dec, humid_dec = dht.read(4)
end
If you don’t have a temperature sensor hooked up, but you’d like to play along, you can code up a random-walk temperature simulator in short order.
temp = 20
function updateTemp()
temp = temp + math.random(3) - 2
end
Type print(updateTemp()) a few times to verify that it’s working.
So, whether real or fake, we’ve got temperature data stored in a variable. We already know how to set up an MQTT client and publish data to a given topic. What’s left? Making sure it happens every few seconds, and wrapping everything up nicely into a file to upload to the node.
Timers
As mentioned above, NodeMCU runs many commands asynchronously. That means that we can’t just write code that says “do this, wait for a return value, and then do that” like you often do in C or C++, because the first function doesn’t block. Everything happens almost at once. The upside of this is that we call m:subscribe() and it just runs in the background. The downside is that when we don’t want everything happening at once, we need a timer.
Say we want to publish the temperature of our sensor every ten seconds. We set a repeating timer, for ten seconds, that calls our publishing function, and we’re done. The timer command runs as soon as it is called, but the function that the timer contains is only called periodically. Let’s test that out.
-- assumes MQTT client setup and connection
function post_temp()
updateTemp()
m:publish("home/outdoors/temperature", temp, 0, 1)
end
tmr.alarm(0, 10*1000, tmr.ALARM_AUTO, post_temp)
You should now see the current temperature (real or imaginary) showing up on the topic on your broker every ten seconds. I also set the “retain” bit for the temperature report, because you probably will only want to report a room temperature once every few minutes, but when you connect with a client you’ll want to see the last value without waiting. (And note that the timer is set in milliseconds, hence multiplying by one thousand.)
Display Node
In the next article, we’ll talk about data logging and graphing and all sorts of fancy server-side stuff. For now, let’s make a quick device that lets us gauge the outside temperature on a human scale, encoded in color: blue is cold, green is brisk, yellow is warm, and red is hot. To do so, we’ll connect a WS2812B LED to another ESP8266 node, read the temperature from the broker, and display it.
There’s actually not much to add to the display node. We already know how to subscribe to a topic and receive data as it comes in. All that’s left is a display function, and hooking it into the received-message action. A fleshed-out version is available in mqtt_display_node.lua, but here’s the important bits:
function display_temp(data)
data = tonumber(data)
r = math.min(2*math.max(data-20, 0), 40)
b = math.max(40-2*data, 0)
g = 20 - math.min(math.abs(20 - data), 20)
ws2812.writergb(ws2812b_pin, string.char(r, g, b))
end
m:on("message",
function(client, topic, data)
display_temp(data)
end
)
Writing out to a WS2812 pixel (or chain) is as simple as calling ws2812.writergb() with a string that corresponds to the RGB data that you’d like to set for each pixel in the chain. Here, we’re just setting a single one. Otherwise, the argument would look something like string.char(r1, g1, b1, r2, g2, b2, ... ).
That’s it for the display node.
Upload Code
Typing code directly into the nodes clearly won’t do. We need to store our code on the device. NodeMCU has an emulated filesystem that we can use to upload our code to. You can call these files manually with the dofile("filename") command, but you can also use the fact that the system automatically runs a file called init.lua on reboot to kick everything off. My initialization files usually set up my WiFi configuration, wait until it’s connected, and then start running the code that makes the device do what it does. (Examples in the GitHub.)
Getting your Lua source files onto the ESP8266 is best done by using uploader utilities that essentially just type them out across the serial line. I use nodemcu-uploader.py myself, because it works nicely with any editor and a command-line build environment. You’ll end up going back and forth between trying something out on the node in the serial console, and writing down what works in a file. Occasionally, you’ll upload this file to the device, maybe restart it, debug, and repeat. This can be a little clumsy, so we can see the appeal of something more integrated like esplorer. Try ’em out and see what works for you.
Wrapup
That’s it for now, but it’s a lot. You’ve got a central home server running, with a temperature (and humidity) logging node that connects to it, and another that pulls out the temperature and displays it on an RGB LED. There are a million ways to elaborate out on this simple system. Motion sensors, light sensors, and relay- or SSR-drivers could flush out a complete home automation solution. If you don’t already have a new-Hackaday-article notifier on your desktop, you probably need one.
Next time, we’ll have a look into server-side extensions that run on the Raspberry Pi broker that we set up last time: this gets us data logging, web pages, and weather forecasts. Following that, as a final installment, we’ll take care of some (cool) loose ends, like connecting your cellphone and other human controls into this whole system, and thinking a little bit about security.
My issue with **m:on(“offline”, function(client) print (“offline”) end)** has always been that once the thing goes off-line it stays that way forever. What’s the point of having an event driven function for offline if the function itself can’t do anything to re-connect (just try re-connecting from withing the offline function, the thing starts rebooting, etc crap). The client clearly has no internal mechanism to keep the connection alive so once the ESP drops off for whatever reason (wifi, reboot the broker, etc) then the client is dead until reboot?
Hmm…. I’ve never seen it reached, so it’s a bit of cargo-culting on my part. I set the reconnect-on-drop bit in the connect() method, so it should try to reconnect.
Honestly, though, I no NodeMCU expert. Anyone who’s tested the edge cases care to comment?
OK, so tested it out a little. I killed the MQTT broker and watched what happened.
With the sensor node, it would error and reboot continually while the server was down. It’ll do this unless you kill the timer that makes it publish in the “offline” handler. Once the MQTT broker was back up, though, auto-reconnect worked as it should.
With the (subscriber) display node, it would disconnect, and then try to reconnect as per auto-reconnect. But once the MQTT broker came back on, the node connected but didn’t resubscribe — meaning that it never got messages (temperature updates) that it should have. For whatever reason, commenting out the “offline” handler made it work again.
In all of these situations, sending a periodic status update from the node to the broker would help with management. I have a debug channel that I run alongside my main one. So the “home/outdoors” node has a corresponding “debug/outdoors” topic where it sends a periodic “everything’s ok” message, using the NTP module and a clock-time, FWIW.
In the real world, it only wakes up once every 10 minutes, though, so these statuses are few and far between.
I’ll have to think more about reliability — the NodeMCU units are still new to me and haven’t been malfunctioning. :)
I like that any serious fault causes a reboot, which is kind of like a watchdog timer. Anyone have any more reliability tips?
Yeah, the NodeMCU is a good Expressif board. The only real problem with ESP chips are that they are power hungry. Running them off of battery won’t get you terribly far, unfortunately…. But if it’s IoT in the house, you’re going to have nice stable power for the most part.
There are areas where IoT may be wanted that you also don’t want to plug in: Bathroom, moveable-buttons (ala amazon button), doorbell ringer, anything in contact with water, and outside perimeter of house.
Those areas, an Arduino and a nRF24L01+ chip do much better. The Nordic Semiconductors are awesome little chips at low power and great bandwidth. I use MySensors library, and have a gateway that posts all data to and from to Mosquitto. And then I use Node-Red to glue it all together, so that any sensor, actuator, or logic can change the state of what’s going on in the network :)
Yep, i have the same issue, working on a sensor network. For some situations…the ESP won’t do. However, since I don’t wan to bother with 2 options, i stuck with only the low power one, there’s nothing the ESP can do that a micro cannot, for my applications.
I went with an XMEGA32E5 micro and a RFM75 instead, since I can get them easier from good suppliers. The micro is slightly more expensive than your typical MEGA328, with tone more features. Together burning 2uA in sleep, 20mA active, so coin cell compatible. Did not get to node red yet…
“However, since I don’t wan to bother with 2 options”
wan? WAN? I see what you did there! B^)
It is possible to put those ESP chips to sleep for any amount of time. I’m currently working on hacking wifi into an old busted weather station, and I’ll have that running off a 100mA solar panel and 18650 battery.
I’m currently programming it through the Arduino editor (boo!) but after reading this I might give Lua a try. In C++ at least, the function is ESP.deepSleep(sleepTimeS * 1000000);. You have to connect GPIO16 to the reset pin so it resets after the sleep period has finished (it triggers this within the function), otherwise it just sits there after the sleep has finished doing nothing.
My dodgy multimeter reads 0.0mA when it’s asleep, so I’m happy with that. If it wakes up every minute for a few seconds, the average power draw should be quite sustainable for my set up.
You can, yes. There are plenty of applications where the ESP is good enough. Sometimes you can slap a solar panel and be done with it. Sometimes bigger batteries are ok.
Other times you want a sensor to be small and work on some batteries for a long time and cannot get a solar panel. Or you want a sleepy sensor but don’t want to wait a few seconds of reaction time while the ESP connects to AP and server.
If you are a company making these devices you can use different solutions for different situations. If you are a hobbyist with limited time, you might want to take the solution that works for both.
Just curious, what Make/Model of wx station?
Re: Power. I’ll get into power reduction and security in the fourth, final, and kinda catch-all installment. Short version: you can go months on AA’s if you play your cards right and only need infrequent updates.
But still, point absolutely taken. Simple radio chips are better for the simple radio tasks! And can last years on a pair of batteries, which is a big difference.
I’m interested in what people do for radio/MQTT gateways. I’ve built my own ad-hoc system, but I’m still on the lookout for something “standard” / librarified.
You use this? https://www.mysensors.org/build/ethernet_gateway ?
Cool, I can’t wait to see the power consumption on IoT sensors. And for my purposes, I tend to use more 18650’s myself. They’re cheap and ubiquitous (especially if you salvage cells from dead laptop batteries). I use a NiteCore charger that can handle all sorts of chemistries, and charge the LiIon batteries well.
And I’m no way a naysayer for this. There’s times where you want IP enabled endpoints with wifi. I’d worry some about network degradation (due to not supporting N), but that’s just nit-picking.
Yep, the backbone of my IoT setup is:an old asus eeePc 701SD as my house server, MySensors gateways (using USB serial gateway), a pile of nRF24L01+ chips (with some SIF24 clones), Mosquitto, MongoDB, Node-Red, and Tor.
Mosquitto is obvious; MQTT broker (server).
I use MongoDB for data storage and time series calculations.
Node-Red is my flow engine. I use it to route data, compute data, store data, and do logical operations on my flows.
I then network multiple areas (hackerspace, house, friend’s shed where my 3d printers are) to Tor Hidden Services. This allows me to treat all my computers as machines on a huuuuge ethernet hub.Every machine has its own onion address, and auto-connects to Tor when on. It also allows you to ignore the traditional “static internal IP, port forwarding, Dyndns, firewall” hell of having a consumer internet connection offer services.
I wrote my own listener/command line interface using tmrh20 RF24 library running on a RPi and built sensors using atmega328p and nrf24l01+. I need a 1year + battery life and don’t have wifi access for my application, so I store the data in a SQLite database. I also have a 7″ touchscreen and wrote a python/kivy program and use gnuplot to view the data on the RPi and have it set up as a wifi access point so I can download the data with my phone too.
Thanks for writing this series. I’m looking forward to the rest of it.
Nice write-up. I’ve been looking at http://owntracks.org/ recently and wondered if it can be tweeked so a cheap android phone can be used as a gateway broker for mobile sensor clusters. Use case: you have a field monitoring task for a period of time and nothing other than a mobile data connection available, and because it is not fixed you need the geo location data associated with the sensor recordings.
I don’t see why not, android phones can be used as an AP using the mobile net connection. From the ESP point of view, it;s the same thing as using a router.
I mean merging the GPS data off the phone with the streams from the sensors so the data is tagged, say for mobile environmental monitoring where you go to a site drop you sensor boxes around the area then use a central point for collection and transmission?
the simplest way i can think of is to use google location API and then on the server correlate the two in time. This would work with a stock android with no special app.
Of course you can make your own app so the sensors actually talk to the app and then the app gathers all the data and sends it. Might be a better solution, in case you have no net connectivity, but still wish to collect data.
well, in this example a local broker is used, with a fixed IP address. I would assume for your app you will need a public server, reachable through some DNS service. Nothing impossible here.
Minimal MQTT is running a full MQTT broker on an ESP8266 like in here: http://myesp8266.blogspot.com/2016/04/mqtt-broker-on-esp8266-3.html
Do you know ESP EASY ? http://www.esp8266.nu/index.php/ESPEasy
I am am going on with it for my house automation process.
note : you should compile the latest revision
Could a small mqtt broker (mosquitto?) be setup on a beaglebone black?
Why not? It has been implemented on esp8266.
I just want to thank you for this series! I wouldn’t have heard of MQTT or been able to get one of these esp8266 modules up and running without it.
Super sweet!
I ordered a couple of these to play with and follow along with this tutorial. I am using windows and I did install the driver from the wemos site. But it seems I just cannot get a flash to work. Using esptool.py it completes and says leaving… and then the blue light onboard goes solid. At that point if I connect at the baud rate it was flashed at I can see the following error repeating Fatal exception (28): epc1=0x4000228b, epc2=0x00000000, epc3=0x00000000, excvaddr=0x000000b5, depc=0x00000000 So I tried flashing at a few different baud rates still the same issue. So next I tried the nodemcu firmware tool, the flash seems to go through and the light does not go solid but still nothing when I connect. Connecting to the one I didn’t flash works fine and the prompt comes up right away. Going to try to dump the working one and flash it back to the nonworking at this point. Did some google searching but came up with no specific answers, anyone got any ideas?
Not sure what’s going wrong, and I’ve never used the WeMos drivers. (What would they be for? The things just speak straight serial.)
On the other hand, if you’ve got chips that already have NodeMCU installed, why reflash at all?
When it boots up, check that mqtt is among the modules compiled in. If so, you’re set.
After a new install, the chips do take a few seconds to reformat some space to use as a “filesystem”. That can take a bit — it feels like 30 sec or so.
So will the WeMos boards work without modification?
How would you check mqtt is installed when it boots up? This is my first foray inot this stuff, be gentle. Thanks.
When you start up a NodeMCU board, it lists out the modules that are compiled in. Easiest way? Connect to it over serial and hit the reset button. You’re looking for a line like:
modules: dht,file,gpio,mqtt,node,tmr,uart,wifi,ws2812
Thius sounds so nice. Ok so I have never used Lua or MQTT or anything like it. So what do I need to do to get going? I have 2 WeMos D1 minis. I need to change their firmware? Is there a step by step I can follow? Thanks so much for the information.
Just plugged in a stock Wemos board. It reads:
SSL: true
modules:adc,bit,bmp085,cjson,coap,crypto,dht,enduser_setup,file,gpio,hx711,i2c,mqtt,net,node,ow,pwm,rc,rtcfifo,rtcmem,rtctime,sntp,spi,tmr,u8g,uart,wifi
In short, they seem to compile everything in there. You won’t have as much memory for your own code with such a build, but at least you won’t have to worry about code not being built-in. (MQTT is on the list.)
So, I may be asking an even MORE basic question. I’m using a Raspberry Pi for the whole bit (no windows) In the bulk of this post, you mention the following.
So plug up a USB-serial converter if your ESP module doesn’t already have one onboard, connect at 9600 baud, and you’ll be greeted by a friendly > prompt.
How do I connect (using what term program)? How do I get to the point of see the “friendly > prompt”? And finally, using nodemcu-uploader, what is the command line that I would type to upload my files so that I can get this WeMos to do what you are doing in this project?
Thanks,
SuperNoob
I know this is a old topic but I have been fallowing along to teach my self about these boards. The only problem I have run into is when I get to this point
**temp = 20
function updateTemp()
status, temp, humid, temp_dec, humid_dec = dht.read(4)
end **
Im stuck! Every thing has worked great up to that point. When I type it in all I get is a blank line (nothing there) useing print(updateTemp()). not like I get from using the print(dht.read(4)). Im new to this, so the learning curve has me a little. Any help would be greatly appreciated. Im just not seeing what Im doing wrong. Wemos D1mini, Wemos D1mini pro, Raspberry pi 3 and firmware from this page.
Thanks for the article which I found helpful to get started. However, the minimal example fails for me because the connection is not ready when publish is called. This causes an error:
PANIC: unprotected error in call to Lua API (main.lua:24: not connected)
Solution: Use “connect” callback to call publish.
Working minimal example:
m = mqtt.Client(“clientID”, 120, “”, “”) — blank user and password
m:on(“connect”, function ()
m:publish(“topic”, “payload”, 0, 0) — no QoS, not retained
end)
m:connect(“192.168.1.224”) — my local broker’s IP