Running a data center takes a lot of work, and even making sure the ambient temperature for hundreds of boxes is in the proper range is an arduous task. When faced with the prospect of installing hundreds of temperature sensors in an EMC data center, [Vivek] had a better idea: put just a few sensors on a robot and drive around the racks. With the right software, it’s a breeze to automate the process and build a near real-time temperature monitoring solution for a huge data center.
The data center robot is based on a iRobot Create, basically a Roomba without a vacuum. Attached to the robot is a netbook, Arduino, and a PVC mast housing three temperature sensors and a USB webcam.
Using the floor of the data center for navigation, the robot canvasses the racks sending temperature data back to a server via WiFi. From there, the temperatures can be graphed to make sure the racks aren’t too hot or too cold.
You can check out a video of the robot in action after the break.
[youtube=http://www.youtube.com/watch?v=4zZoev0IWIc&w=580]
I would have gone with something using a little less power than a netbook. Olimex makes some nice low power SBCs, (grumble) Raspberry Pi, BeagleBone, Some of the Olimex OLinuXinos have wifi built in as well as plenty of I/O, addons for ADCs, etc.
A hacked Wifi router or Pogo plug would also be options.
Using the laptop means not having to build a battery system. Plus they might have had one. I do not know what the Arduino is for at all. the DS18B20 is a one wire device. No need for A2Ds at all. Dallas even makes a USB to one wire bridge. If they where willing to drop the driving mode with webcam they could just use the Arduino without the laptop. Add a XBee and then make an XBee to lan bridge.
Over all it is really cool and does the job so kudos to the folks that made it.
I thought the batteries in these things lasted like under an hour? I’d imagine you need to monitor the temps 24/7 right? With the lag of waiting for the bot to travel around the whole DC and the hassle of replacing the batteries, I don’t see this as a good way of doing things..
Kinda cool and geeky, sure, but practical? doesn’t seem like it..
When its vacuuming it might lasts around one hour, but with vacuum and sweeping motors off it’s capable of running around for up to 4 hours.
I really hope they don’t actually use this. It’s fun, but a terrible idea to use in practice.
Terrible why? Battery life issues are one, but what else are you thinking?
I can think of one reason.
Boss: Why did our servers fail
Guy: Well……the roomba got stuck and didn’t alert us to the temperature rise in that rack row. Honestly, it would have sensed the temperature rise in another 30 minutes but by that time it was too late.
Boss: I should fire you right now for not buying industry standard and common sense items like temperature sensors and monitoring server for less than 2000 dollars. We just lost all our work and our client.
Our server rooms use a small box that connects into any switch, configurable IP and web interface. There are PoE kinds available so you don’t need extra power, if your switch supports it (granted, core/data center switches aren’t usually PoE). This, along with the battery issues mentioned by SteveHaD, are good reasons why this isn’t the greatest idea and the “it’s daunting to install a bunch of temp monitors” is invalid.
HOWEVER, I definitely want to work somewhere where little robots are just running around, like on the Death Star. So I like this hack.
Problems/comments:
The biggest part I am confused about… Why not just get the servers to each monitor the temperature of itself? Pretty much all motherboards made these days have thermal monitoring capabilities.
Why use an Arduino and a Laptop? Why not just use the Laptop with a USB temperature sensor? You can buy them from eBay for about £3 (cheaper than an Arduino).
What good is monitoring the temperature BETWEEN the racks and on the floor? That isn’t even the place most of the heat will be.
At that speed surely it is too slow for “real time” monitoring of the data center.
In fact, to quote XKCD: “Arduino for blog cred”.
Servers can only monitor their internal temperature. Trying to write code for 100’s of different hardware and install it doesn’t sound easy. And internal temp dosen’t necessarily convert to ambient constantly.
But if the internal temps of the CPU and HDDs are fine, what use is the external temp?
And aren’t the sensors neatly abstracted (at least in Linux) so you don’t have to write code for 100s of different hardware?
I agree with you on that, if the CPU’s/GPUS’s/HDD’s/ etc. are all fine then I don’t see much point of monitoring the temperature of the FLOOR like this robot does.
And yes they are all abstracted. You can write maybe 4/5 lines of shell code to be able to get the temperature.
…and on Cisco devices along with most other manufacturers, it is a simple snmp integer value with corresponding temp. So it would be trivial for even a novice programmer to write up a dynamic webpage that lists all temperatures for all servers and networking equipment and update it in real time using a simple script.
Looking further into the issue, if everything was well documented the first device monitor could be up and ready to go with less than three hours work. Each device after that would take less than a minute to add manually. There may even be a way to dynamically poll for snmp enabled temp values, but I don’t know of a way.
I would not be surprised at all if someone has already made some OSS or a library/module for this already.
Wow. This looks like as much of a hacked-together kludge as EMC’s NAS servers. We used to have one that was literally 5 different servers all glued together with ethernet and serial cables — and in included and “embedded” desktop-class mini-switch that handled its internal networking. All this in a 5-figure price supposedly enterprise-class storage system. Boy were we happy when we threw it out for a NetApp. 12-Us of space replaced by a 2-U box.
This is the stupidest thing ever. Temp probes cost about as much as ethernet cable, and there are piles of even open source monitoring and alarm/alert programs. All turn-key. IN a datacenter, this is absolutely a trivial cost.
You shouldn’t cover this shit on this website. Anybody doing this to actually monitor temps will not actually be monitoring temps: they’ll have a fucking toy robot running around to show off, sometimes, while it isn’t recharging.
To answer the question of why servers themselves don’t monitor – the servers are what is trying to be protected. Those can and will go down if temp is too high. You need something independent to monitor.
As a proof of concept showing a data center robot capable of going to a specific location and performing measurements, this is cute. As a precursor to a data center robot capable of going to a specific location and rebooting a designated server, this is awesome.
In practice, this robot has more use in mapping the boundaries of a hot spot that the in-place temperature sensors have detected (is it a failure of the underfloor airflow right there or has a server’s heat output mysteriously tripled). Endlessly circling the DC is just a gimmick.
I think it could be done more efficently with some off the shelf sensors connected to rs458 or simply the servers internal temperature sensors, but I think it is still a awesome hack. It whould bring a smile to my face to see one driving around :D
Did anyone actually click on the link? This is a project for an annual innovation showcase for the company the author works at. This is a proof-of-concept prototype for a competition. Pull up your pants, guys. Your ass is showing.
I did read Sean and I agree with you. This project is to monitor the ambient temperature for power management not server temperature. Just like any computer servers have fans that turn on and off when needed so just monitoring the server temps would not give them the answers they needed. The robot was used to make a temperature map of the data center. Over all very need but I have no idea why the used the Arduino. The temp sensors they used are digital one wire devices so no need for a2d and Dallas makes a USB to 1-wire adaptor.
would be more practical to attach a rail to the ceiling and just have the sensors be moved by a steel cable (should i patent this?)
Too late. You presented the idea in an open forum. Now it’s in the public domain. :)
Seeing what Apple has done, I don’t think it works that way anymore.
Hmmm…X/Y axis on the ceiling with simple cables and pulleys moving a SBC “head” that feeds temp data back wirelessly. That would definitely allow you to map the temps overhead but unless you had a temp sensor hanging down into the rack space it wouldn’t measure between the racks obviously.
OMG that music ruined it for me. The worst background music ever on a half ass idea…..
As already pointed out already temp sensors internally or externally aren’t a factor of cost also the fact relying on this alone as single point of failure which anyone in networking knows is a no no.
But what people are forgetting what this guy hasn’t forgotten is that no all equipment isn’t made equal the equipment doesn’t all run at the same temps, He also demonstrates which i find interesting the change of ambient temperature as the Robot progresses down each Rack.
I this guy is on the right track but a temp sensor alone isn’t the best application and again this should only be used as an additional layer of monitoring there’s no such thing as one single all in one
My suggestion though would be to make the irobot create or hacked Openwrt router? into a web security camera (not the temporary laptop’s) and sensor besides the robots collision sensors add sensors into the room itself for the robot, other things that could be added fire, smoke, gas sensors and a light sensor.
The light sensor for instance could detect low level lighting and send an email to the administrator(s) that there might of been a critical Brown/Black-out at the same time this event might of occurred administrators could look at UPS logs and compare the time stamp from the robot.
The robot could the save all this information to a SD card and push it over the WIFI throw into a pretty graph this would also be useful to know when its time to upgrade and where and explaining to the boss(s) Sir this is “why” we need to upgrade and here’s the proof a little ROBOT told me. “The little green one?”
DS18B20 1-wire temp sensors are cheap, easily bussed, and can be fanned out across 50m or more. It would be easy and fairly cheap to make a Christmas light like strand of these and string them down an aisle, and use one USB or RS232 bridge per string. I use a single one of these in our server room and it’s already proven valuable in diagnosing some building airflow problems.
Of course the Roomba solution takes samples in the middle of the row, which is a better average than anywhere it would be practical to run this sensor strand.
Wow, the stupid factor is so high I don’t know where to begin.
Temperature monitoring is a key factor in running a healthy DC. Having a dorky little robot wander around and get what basically turns into a random sampling is NOT temperature monitoring.
Whoever signed off on that project should go back to their McDonalds job.
I applaud the creativity these guys showed. I would like to see a version of this that used a toy train. If you’re going to follow a set path…
The reason you use temperature sensors in every rack is because you need real time readings from all racks simultaneously at all hours of the day. By the time that thing makes its way back to a rack, a computer could be overheating for many hours.
Kudos for the fun hack but businesses really need to understand that spending a little money on the right product makes a big difference.
robots are neat, but not really efficient here. environmental managers have alot more power to flip relays, send emails, turn on alarms. The APC rack managers are brutally effective, i’m amazed there’s large companies out there that don’t have monitoring equipment like this
mixing cold and hot air is a nice way to throw away money.
Roomba would be more useful if it can straighten out those messy racks ;)