The city of Oldsmar, Florida was the source of disturbing news this week, among reports that someone gained unauthorized access to a water treatment facility. In an era where more systems than ever are connected to the Internet, the story is a sobering one for the vast majority of people reliant on grid utilties.
The hacker was first noticed to have gained remote access to a computer system at the plant at 8 a.m. on February 5. An operator at a workstation controlling chemical dosing at the plant observed a remote connection, though did not initially raise the alarm as such access is common practice at the facility for troubleshooting purposes. However, at 1:30 pm, the hacker connected again, this time commanding the dosing system to raise levels of sodium hydroxide in the water from 100 to 11,000 ppm – dangerous levels that would make the city’s water unsafe to drink. The increased level command was immediately overridden by the operator, who then raised the alarm.
The city notes that other safeguards such as pH monitors at the plant would have triggered in the event the original intrusion went undetected. However, the event raises renewed questions about the level of security around critical utility systems connected to the internet. In the last decade, cyberattacks on physical infrastructure have become a reality, not a vague future threat.
Nothing’s known yet about the perpetrator, or how secure the system was (or wasn’t?) before the event. It’s been long known that a lot of infrastructure is simply connected to the internet, as Dan Tentler has been showing us since at least 2012. (Video, ranting.) Indeed, it’s amazing that we’ve seen so few malicious attacks.
Friends don’t let friends install TeamViewer on their critical utility infrastructure…
Yeah… How is it possible to “raise levels of sodium hydroxide in the water” with a remote connection? Are they *trying* to get us all killed?
I can see cases where you’d want to control stuff remotely (especially in the current pandemic/”pandemic” scenario), the more precise question is “How is it possible to raise levels of sodium hydroxide to dangerous levels?” (remote or otherwise)
Yeah, but still…why is is reachable via the public internet? Why isn’t it protected by a Firewall of sorts? Why isn’t it only reachable via a jumphost or a VPN tunnel, or even both?
And I just friggin bet that it’s got some well known default credentials, if at all.
And it makes you wonder what else is like this. Any nuclear buttons?
@BT: Funny enough, it doesn’t. I remember there being a site somewhere that’s crawling the IP space for open admin UIs. There’s heaps of chemical plants and similar out there offering their admin access to the public internet.
Which site are you referring to when you mention “ there being a site somewhere that’s crawling the IP space for open admin UIs.”
@BT for various not particularly good reasons, the first Permissive Action Links for nuclear devices were set to a bunch of zeros as the code
That would require someone with knowledge of network security, and that costs money.
I can say this as someone that used to work at a different municipality not to far from the one in this story. low ppm injections are just for ph balancing, but if you have to break a line for a repair you need to be able to sanitize it, we used our bleach system but caustic injection is the better route and is used in the food industry in combination with acid washes( in a separate step of the wash cycle). But for the line washes you the other end of the line opened at a relief or fire hydrant to flush it while the high level (lye, bleach, acid) is pumped in. then after the chemical pumps are returned to normal operation you continue flushing until the level coming out of where you are flushing drop to normal.
Interesting, thanks
But the ability to adjust levels of anything above what’s safe should only be possible by people physically on site, and there should be purely mechanical locks and/or switches that have to be opened or toggled to enable going above safe levels. Like a big lever labeled LINE SANITIZE and LINE FLUSH with padlocks to lock them in the off position. Nobody could override that remotely with anything. (Except a remote operated Robot Lock Picking Lawyer.)
Just a guess: the specific hardware is able to supply 0 up to 11000 ppm of sodium hydroxide to stabilize pH value of water. And 11000 may just the max value.
I think the question might be better worded as “Why is the maximum value two orders of magnitude above the normal level?”
Wild goose guess. But maybe they use concentrated version of it when cleaning the tanks? I mean they could fill the tank with it and then drain the water somewhere else instead of tot he city’s powe supply
For sanitization if something is detected in the water and it needs to be sanitized and will never be released.
Import note: there is no comment here on whether or not it would be successful. Just that the remote connection requested it.
Maybe it’s entirely impossible, maybe it is restricted by credentials. Maybe there is a 2factor system.
The only think we know is the remote connection requested it and that the operator noticed that.
Maybe that’s just the generic specs of the hardware which in other uses might be totally normal.
If so then it might have been nice to throw another layer between the complex, hackable and crashable computer system and the hardware. Just a simple microcontroller maybe that takes a value in from the computer and verifies it is sane before sending it on to the hardware.
No, they have just heard about that “basic diet” quack and the fear of “hyperacidity of the body due to malnutrition, smoking, stress, “. So they wanted to counteract by increasing the pH level of the water to increase public health. :-)
No good deed goes unpunished ;)
Probably because the equipment is a highly tested and well built at a sensible price generic device for injecting something into a pipe, and it might be reasonable to inject 1100 ppm of X into Y in another situation. Rather than a custom built device with physical interlocks designed specifically for the water supply, which would have cost 10x the amount and left people saying “why didn’t they use this far cheaper device?”
Look like TeamViewer has been used. I’m willing to belive that this was only due to a poor password. Maybe even an internal thing, ex-employee and such.
And why do the always scream ‘hacker’? How I hate this misuse of the term hacker.
https://www.reuters.com/article/us-usa-cyber-florida-idUSKBN2A82FV
https://www.vice.com/en/article/88ab33/hacker-poison-florida-water-pinellas-county
“it’s amazing that we’ve seen so few malicious attacks.”
…that we know of.
Exactly. It’s easy to imagine a hacker doing this, then quickly setting it back to normal, thus keeping the capability ready for future use. Putting a few hundred houses out of water wouldn’t be a big deal in the grand scheme of things, but what if it was half of a state at the same time?
My guess is former employee, former contractor, etc. If they knew enough to go specifically for that setting it hints at a knowledge beyond what you’d see with remote access, and we don’t seem to have a shortage of crazy in Florida.
In defense of Florida, it doesn’t have a higher level of crazy compared to other states. What it does have is one of the more open records law in the country. The law makes it difficult for state and local government to keep things secret, which makes it much easier for reporters to write stories.
https://www.miaminewtimes.com/news/how-floridas-proud-open-government-laws-lead-to-the-shame-of-florida-man-news-stories-7608595
NotTheFlordiaMan?
B^)
My guess:
– nobody changed the default password because we would forget it when someone needs to jump in because of an emergency in the middle of the night
– someone scanned the internet, got the admin panel, tried a common password and it worked, or searched for the default password
– did nothing because he didn’t knew what was that
– went back to Google, found the manual
– went back again with knowledge and changed the parameters
I believe it was a kid on school, and went back home to continue “hacking” the water supply.
“Based on our cybersecurity incident response book which we proactively created 35 years ago, the profile suggested is that of a 13 year old who has a commodore 64 and a prodigy account.”
More likely an ex-employee as the article suggests. If they didn’t bother setting up a VPN then they probably didn’t bother changing the default teamvewier password, wichi is pretty strong. Ergo, must have been someone either working on the system or who got hold of the credentials from another employee.
Since they knew (a bit) what they were doing, I doubt it was just someone reading the manual and doing EXACTLY one thing. A 13yro would probably click around.
Agreed. There is some intimate system knowledge to connect and go right to work without some usual sniffing around and figuring out how the system is set up, what to pull up and what values to set. Kudos to the operator who immediately overrode the command. But yeah total angry ex employee Teamviewer credentials fits pretty well outside of a patsy involved with an active domestic terror group. Lotsa crazies out there these days on both sides. Happy to stick to my workbench while they beat each other up.
Internet connected municipal ANYTHING is bad news.
Yeah that’s the real story. There’s no excuse.
See also: municipal workers refuse to go into work during a pandemic when they could work from home.
Also surprising that there was an employee just hawkishly watching the monitor for signs of remote access…
I was an industrial engineer for several years. First thing every morning I’d walk around and check the readouts on my machines to make sure everything was working properly. You’d be surprised how long a machine can work like shit before an operator would get around to reporting it “oh yeah, it’s been making that burning smell for weeks”. Plus sometimes things are a little off and the machine still works, but a trained eye will know that it’s about to fail.
Well it usually goes like…
Operator: “Number 3 combobulator is making a slight burning smell.”
Supervisor: “Don’t worry about it.”
Next day…
Operator: “Number 3 is still making that smell.”
Supervisor: “Get back to work.”
3 weeks later…
Operator: “Number 3 is on fire.”
Supervisor: “WTF, you should have shut it down at the FIRST sign of trouble!!!”
can confirm major automakers run entire casting plants like this
management are perpetually stressed because they can’t figure out how to stop machines going down every few hours
And workers perpetually stressed walking the line between figuring if it’s a minor problem they will get yelled at for hitting the red button on and written up, a major problem they will still get yelled at for hitting the red button on for 4 hours until the plant engineers with lots of fancy equipment finally declare it was, then 3 minutes of being a hero, until next minor problem, or by sight, smell and touch failing to diagnose the next major thing in time (that takes plant engineers 4 hours) before it breaks and sends the whole plant home for a week.
I’d be less surprised to find that someone went to change a setting from 100.00 to 110.00 and forgot the decimal point.
“Uh, Fred? Why’s that set so high?”
“Shit. Blame hackers.”
SCADA is a dirty.five-letter word! InfoSec people have been warning about for three decades!
SCADA with probably 12345 as the password and no VPN is a dirty five letter word more than SCADA itself I’d imagine, considering how well it works most of the time
The reply will get held up for having a link, but Dan Tentler did a talk about 5 years ago named something like “115 b**s**t crazy things to put on the internet” at I believe a DefCon and it’s utterly amazing what he shows there – and that’s just the funny stuff. I’ll put the link in the reply but it comes up easily if you search youtube with Dan Tentler and crazy thing on internet. He’s also done talks on ShoDan, which is one easy way to find such things. Small water plants are the least of it…
Link mentioned above: https://www.youtube.com/watch?v=hMtu7vV_HmY&ab_channel=Devoxx
A much more sinister move would be to flash VFDs controlling pumps with a custom firmware that would vary motor speed to constantly play Gourmet Race, Tsurupettan or Caramelldansen regardless of actual speed settings.
a hack like that would put Stuxnet to shame…
The strangest thing about this was that this was predicted by Weird Al Yankovich in his 2006 song “Virus Alert”
“So just trash it now, or else it will
Decide to give you a permanent wedgie,
Legally change your name to Reggie,
EVEN MESS UP THE PH BALANCE IN YOUR POOL
ITS GONNA MELT YOUR FACE RIGHT OFF YOUR SKULL…”
A true rival to Nostradamus 🤣
That’s about as close as Homer dumping manure in Lake Springfield in the Simpsons movie, or Prof Frink contaminating the city water in a comic book spinoff…. or the original pollution of Lake Springfield with nuclear waste, before it was cleaned up and Homer dumped manure in it.
I wonder how the system worked before remote access, surely the world existed before the intenet became mainstream. The answer to this is relatively simple in thought, people do bad stuff, like all the time, combine that with zero conscience and you should have come up with a critical system designed first with said people/entities in mind. A shame we always seem to have to wait for a major disaster to perfect a system. Sort of reminds me of that intersection that common sense dictates a traffic light, but waits on a disaster first.
In the before time you had people on multiple shifts. They wanted compensation for those unholy hours. So they were replaced. And the taxpayer is happy (for a while anyway).
I work for an electrical company. External (Internet) access to the control system network is not allowed. We grumble a bit as it sometimes makes it tougher to get our job done, but we understand ‘why’, so we deal with it. When there is a problem, you come in to work and deal with it (covid blah blah or not). Even when you are in the ‘buildings’ access is still layered to keep users/intruders at bay.
This all relates even to home automation and IOT applications too. Any of my projects stay in house on my home network which is separate from internet network. No plans to ever connect remotely with a cell phone or anything (Google voice comes to mind). Not worth it. I cringe when people show me turning on lights or something with their phone, or videos of inside their house while at a restaurant or where-ever…. Paranoid? Maybe. But I sleep better.
“Any of my projects stay in house on my home network which is separate from internet network.”
I have been coming to the conclusion that I need to have an internal playground for projects and vintage stuff, even behind a couple of firewalls they give me the heebie geebies.
This entire story is being presented as a “malicious hacker” in the media, but I have taken this story with a grain of salt. A remote connection doesn’t seem to be disputed, but if you look closely at some of the early stories, the value went from 100 to 11,100. That increase is curiously like someone leaning on the 1 key while the field is highlighted.
I see several possible scenarios:
– There was a legitimate breach of a remote access protocol, the person who breached it was a curious onlooker and decided to “change” a value to see if they really had access.
– There was no remote access and someone accidentally poked the 1 key or actuated in some manner.
– There was a “h4x0r” who intended great harm and used their l33t skills to penetrate a municipal water supply.
Maybe the truth is actually a gradation of those scenarios?
“the value went from 100 to 11,100”
Others report it as 100 to 11000.
Like if someone was changing a setting from 100.00 to 110.00 and forgot the decimal point…
There are “git like” version control systems like versiondog and autosave which could at least detect some modifications.
I wonder if they could also mitigate or detect such attacks? (Someone working with such tools?)
Normally pH correction chemicals are controlled based on plant flow via a large flow meter and trimmed via at least one pH monitor, normally 2 to allow for them drifting and to take one out for maintenance.
For caustic soda lots of plants use peristaltic pumps with much higher turn down ratios than the older diaphragm pumps, 255:1 isn’t uncommon so increasing the dose from 100 ppm to 11,100 is probably setting the pump to run at its maximum speed.
The operator probably noticed all of the pH monitors after the dose point going into high level alarms and the dosing pump speed being set to manual and 100%. Wouldn’t take long to work out what was going on.
(Process engineer designing water and waste water treatment plants)
If it was done by someone local, the headline will read;
“Florida Man (blah, blah, bla)”
And this week in “WHY THE F*CK IS THIS SYSTEM NOT AIR GAPPED?!?!?!”
It IS air gapped!
(Bluetooth is like over the air to another device, so that would be air and a gap)
B^)
Because it’s a water system. It only needs to be water tight.
Also, it probably was until a couple of months ago when they realised that if they got COVID they’d have no water supply so hurriedly connected it all online so people could work from home.
Government Systems should be on a closed private Network or Air Gaped. At least have DARPA create a new Networking Protocol that an average civilian can’t access.
A retired Senior Chief once told me “You would be surprised how many times a day someone was trying to hack a simple terminal to a machine shop”.
I think this is not a real hack as much as security through obscurity. I have friends who have ‘graduated’ to remote system access in the last years (farms, shipping docks, …) and the security is close to zero. The only ones which have increased security are the ones who have been hacked or had to provide some successful security audit. Of course, there are ones that audited but did not want to spend the money for hardening, since it didn’t provide an immediate cashflow increase.
Some years ago I got hold of a sevice software for some prosumer device. It was fun to dissassemble it and look at the bad code and security. There was one login dialog which you could just cancel and the entire database was visible with just a single unauthenticated SOAP request. Not just one device, but all the devices, some of which were security-related (industrial trackers with geofencing). DB modifications were also done via simple SOAP requests.
I prepared a report for the company and alerted them to all the security issues. They responsed quite some time later and asked me if I would like to apply for a position (they are not SW-centric). I said I can consult in hardening the system, but either way, they should harden their access. Never heard anything from them and years later everything was still unsecured.
Just one of those facepalm moments in life…