Now it’s official. The particular website that was hit by a record-breaking distributed denial of service (DDOS) attack that we covered a few days ago was that of white-hat security journalist [Brian Krebs]: Krebs on Security.
During the DDOS attack, his site got 600 Gigabits per second of traffic. It didn’t involve amplification or reflection attacks, but rather a distributed network of zombie domestic appliances: routers, IP webcams, and digital video recorders (DVRs). All they did was create HTTP requests for his site, but there were well in excess of 100,000 of these bots.
In the end, [Krebs’] ISP, Akamai, had to drop him. He was getting pro bono service from them to start with, and while they’ve defended him against DDOS attacks in the past, it was costing them too much to continue in this case. An Akamai exec estimates it would have cost them millions to continue defending, and [Brian] doesn’t blame them. But when Akamai dropped the shields, his hosting provider would get slammed. [Krebs] told Akamai to redirect his domain to localhost and then he went dark.
The Democratization of Censorship
[Krebs’] takeaway from the whole event is summarized in his blog post (now that he’s back online): “The Democratization of Censorship“. It’s worth a read, and we’re not going to try to one-up [Brian Krebs]. His basic point, however, is that it used to take a nation-state to censor information on the Web — strongman regimes or agencies with spooky contacts in big ISPs. But if any script-kiddie can leverage IoT devices with hardcoded passwords to pull selected websites off the Net, the game has fundamentally changed.
You’d have to be a fairly dedicated anarchist to say that this is a good development. After all, we haven’t traded government censorship and surveillance for private censorship. There’s just another actor on the stage, and what’s worse, that other actor is criminal. We understand that [Krebs] meant it in an ironic sense, but “democratization” is such a nice word that we hate to see it used here.
[Krebs] also makes the case that sufficiently motivated groups can now effectively silence journalists, and makes the case for thinking about how we can protect free speech on the Internet. For his part, [Krebs] is now hosted as part of a Google project (Project Shield) that aims to mitigate such attacks. (Ironically, Google still thinks of its adversary as “powerful institutions” rather than “some dude in his basement”.)
The timing makes it look like it was the “vDOS” folks who were selling DDOS services, and two of whom are now in jail. They had a beef with [Brian] and they took him down. But while in [Krebs’] case it probably was personal and an issue of censorship, in the majority of cases it’s just about money.
In the last few years, ransomware has become so widespread that people outside the security community have even heard of it. But DDOS ransom attacks are the true growth industry. And these extortionists even have cute nicknames now: “booters” or “stressers”.
[Krebs] estimates that getting DNS services that will protect him in the event of a similar attack would cost him $100,000 to $300,000 per year. Clearly, he’s not able to fork out that much for legit protection, but the cost of protection against this sort of attack should provide an upper bound on how much ransom these criminals can ask for. As another data point, the ransom note delivered to ProtonMail suggests that the actual street price is as low as twenty Bitcoin — around $12,000. (They got hit, customers complained, and they paid.)
The point is that one could make a good living running a botnet of DVRs, threatening to knock websites off the Internet for a day or two. We see this as a much more likely threat than [Krebs’] fear of censorship. DDOS extortion is illegal and wrong, but where there’s money, there’s going to be a criminal to fit the crime.
Why? Why Not?
Given that botnets of DVRs can be converted into cash, [Krebs] was asked why he thought anyone would do this. Before the attack, whoever was running the IoT botnet had 100,000+ computers under their control, all of which were entirely under the radar. But now the IP addresses of all of these machines are known, and someone might get around to patching the devices someday. Who would burn a gigantic botnet just to make [Brian] mad?
[Krebs’] answer is terrifying, but probably spot-on. It doesn’t matter who launched the attack. There are tens of millions of insecure IoT devices out there. Using up 100,000 here or there is a drop in the ocean. Of the bazillions of IoT devices coming online this year, how many have hardcoded administrator passwords right now? How many will be found to be vulnerable to yet-unknown attacks in the near future?
We also cynically think that hitting [Brian Krebs] is good advertising for the groups who are selling DDOS extortion — if there was a single sysadmin who hadn’t heard of the concept, they will have now. Akamai touting the cost of defending against this sort of attack is the best publicity that the “booters” could have hoped for. Scale your botnet up, hit a rich target, and maybe you can approach the $100,000 payoff. (Not an actual suggestion.)
Whatever the motive, there are millions of unpatched routers and DVRs out there waiting to enlist in the next botnet. In June 2016, Sucuri wrote about defending against a “large” botnet of only 25,000 CCTV appliances. In August, Level 3 wrote about vulnerabilities in over one million units of one brand of DVR. What counts as a “large” botnet has quadrupled over a few months, and the amount of traffic that one can generate has kept pace. And all of this is just the tip of the iceberg.
Tiny Headless Servers Everywhere
The problem is one that we’ve written about before, more or less obliquely. IoT devices contain headless computers that are connected to the Internet and talking to the outside world without human oversight. They’re what the layman thinks of as servers: a “box” somewhere with no GUI, accessed remotely, and dishing out data 24/7. The important difference between an IoT device and a traditional server is that the bigger server has an administrator who can apply patches and software tools that help him or her keep an eye on things.
With IoT devices, the ability to update, upgrade, audit, and administer is still in its infancy. The root passwords to some of the DVR devices used in this attack have been known since 2013, and scriptable attacks against the devices are included as a Metasploit module. A competent sysadmin would have patched that by now. (And a competent manufacturer never would have let that out the door.)
Instead the devices are administered by (millions of) people who don’t even really know that there’s a tiny little computer inside. These are people who have no idea about downloading and flashing firmware upgrades, or don’t understand they need to do so for a webcam.
Stereotypes abound, and as relatively sophisticated users we might feel smug. But are you 100% sure that there have been no firmware updates available for your router in the last couple months? We have better things to do than babysit our devices.
How to Fix It?
The security problem of IoT appliances is real, and it has nothing to do with Big Brother using your Nest to tell what temperature it is now in your living room, not that we like that either. Exploiting botnets of IoT devices has become a viable criminal option. Unpatched IoT appliances are the (pre-service-pack-two) Windows XP machines of the moment: they’re a public menace because they enable criminal activity. And it’s going to take both industry involvement and user education to get us out of this mess.
One solution is remote-push firmware upgrades. Of course, this is its own avenue for malware distribution, but it might be less dangerous than leaving hard-coded administrator passwords in place, or running outdated software with known exploitable bugs. There are a number of known bad ways to implement this: a single key for all devices “hidden” in the EEPROM, for instance. What are the good ways?
People don’t like change, though, and heavy-handed (hello, Windows 10!), late, or failed push updates give the whole mechanism a bad rap. And companies go out of business or simply decide to pull support for their products. Other firms just don’t care. We can’t rely on businesses to secure our devices in perpetuity when they have no financial incentive to do so.
In short, the consumer IoT botnet problem is a thorny one, and it’s not one that we’ve heard the last of. What do we do?