Early today, some party unleashed a massive DDoS attack against Dyn, a major DNS host. This led to a number of websites being completely inaccessible. DNS is the backbone of the Internet. It is the phone book that turns URLs into IP addresses. Without it, the Internet still works, but you won’t be able to find anything.
Over the past few months, security professionals have suggested — in as responsible terms as possible — that something big could happen. In early September [Bruce Schneier] wrote, Someone Is Learning How To Take Down The Internet. The implication of this very general warning is that someone — possibly a state actor, but don’t be too sure about that — was figuring out how to attack one of the core services of the web. The easiest way to effectively ‘turn off the Internet’ for everyone is a Distributed Denial of Service attack against root servers, DNS servers, or some other service that plays a key role in the web.
Dyn is responding well to the attack this morning, and the Internet is safe from attack for the time being. As for who is responsible for the attack, what the goal is, and if this will happen again, no one knows. An attack on this scale is most certainly someone with a very large pocketbook or a state actor (Russia, China, the US, UK, Germany, Israel, or the like) but that’s not a given. It’s also not given the DDoS attacks have stopped. You might not be able to read this, but if you can, it might be a good idea to find a shortwave radio.
No comments? WTF.
short wave radio shop here I come.
A ghostly voice filters through the hiss of static and faint distorted screams. You hold your breath while gently adjusting the dial the tiniest amount, not wanting to lose the voice hovering on the extreme edge of detection. It grows clearer and you can finally make out the words: “Not a hack.”
…ould’ve…one it…with…555…
…uino… ..treet cred…
typos… in the post…
You made me laugh in a library. Thanks, haha.
spilled my coffee, lol
They are desperately trying to find a pencil to write down the HAD ip adress.
That’s the thing. With the modern web, it’s not unusual for a website to have a few different IPs, as your traffic is load-balanced to one of several servers. To say nothing of the plethora of content in a page grabbed from various sources. It isn’t just one page being served once from one server anymore.
I’m definitely not an enterprise web dev so bear with me, but it’s the same site being served from multiple ip addresses, right? I’d have thought that unless all internal resources and links etc are using absolute urls then the site would still function pretty normally.
Anything wordpress based is NOT relative URL based. They justify their design choice, but I still think it’s wrong ;)
I only visit hackaday in reader mode so I will survive the DNS post apocalypse just fine.
Oh god – this and the replies – too funny. Genuine lol moment, which startled my wife.
Trouble with shortwave is that you’re too likely to fall across the (Trump booster) Alex Jones telling you how the Dyn failure is part of the conspiracy…
I know this from the shortwave radios I’ve built. Hacking!
You will need a power supply to do that after you generator or batteries run out, and you would need someone to talk to!
I heard somewhere that the issue was concentrated mostly on the East Coast USA area.
I’m in North Carolina and things appear to be working for me…
Well I’m in France and Github, Twitter, Quora, SNCF (french train company) are “down”. Well, if only I had the IP adresses I’m sure I could access them :p
Does android use a ‘hosts’ file? And can you access it without rooting? Probably not eh. But then again you always want to root to have android be usable. And how about iOS? Because sometimes using the IP doesn’t work, if the same IP hosts multiple sites for instance, so you need to supply a local DNS and use the name.
Anyway a tech savvy user knows there are tons of places to get the IP online, and as long as one of them works you’d be able to find the IP. Also ISP’s and other DNS providers should have the sense to monitor outages and to use a damn old cache during the layover of a DNS disruption, but then again maybe it can mess up the certification now everybody uses HTTPS and then browsers will refuse the connection.
It was out for a few hours and ended about 9:30 eastern. Affected most of the eastern seaboard and extended west as far as Chicago.
It impacted parts of Canada as well, notably Montreal and Toronto
There’s an outage map here:
https://techcrunch.com/2016/10/21/many-sites-including-twitter-and-spotify-suffering-outage/
It ended at 9:30, but then a new attack with new outages started around noontime, and that was fixed fairly quickly.
This is breaking news ATM, at any point this afternoon there could be new attacks and new responses.
I read this as “outrage map” and that made sense to me.
Im in aus and i lost access, that map is wrong
Im in Aus and i was affected, it seems that the map is incorrect.
and besides, would you even be able to make a land based map?
the internet isn’t based on geographical location.
There are physical servers that would be affected, but if they were load distributed,
the distribution of users connecting probably wouldn’t be based accurate on physical location anyway
I thought it was my router acting up again. Hope we find out more! :/
Further update, it’s 3:00 Eastern, and the “isitdownrightnow” site is offline, and the twitter “API status” link is also down. These might be leftover un-fixed from this morning, or it might be ongoing or new attacks.
In any event, we’re not back to normal quite yet.
http://www.isitdownrightnow.com/twitter.com.html
https://dev.twitter.com/overview/status
Still having some outages. 3:11 in the Atlanta area, and I can’t log onto Digikey. Isup.me seems to be able to access their site, though.
And further further update, Twitter is still having problems and there is breaking news (with photos) of a heavily armed police action outside the Ecuadorian embassy in London. Right now (3:11 P.M. Eastern, US)
Not making this up, but I can’t verify because I can’t get to the post in question. Can anyone else read this and tell us what’s going on?
https://twitter.com/wikileaks/status/789493599121604609
That was supposed to be a link, but apparently the HAD site replaces twitter links with the hot link.
The link should be:
(https)(colon)(slash)(slash)twitter.com/wikileaks/status/789493599121604609
“PHOTO: Heavily armed ‘police’ appear outside Ecuadorian Embassy in London where Julian Assange has political asylum (photo, Tuesday morning)”
The photo isn’t much, just hands inside vehicle.
We are all in the dark about this.
Apart from the assault rifle, armoured glass and checkered hat band, no nothing there in that picture…
Okay, but “assault rifle” is a misnomer.
My point was that the photo doesn’t say any more than the text does. An image of police sitting in a police car is useless.
Did anyone happen to notice the last 3 words “(photo, Tuesday morning)” on this tweet?
That’s about the time his internet access was cut off, but a full 3 days before this DDOS attack. If WikiLeaks wanted to suggest this attack on Friday was a smokescreen for violent action against Julian Assange, which might be under way, wouldn’t it make more sense to claim the armed cops appeared on Friday rather than Tuesday?
Can any of you americans gun owners enlighten a european fellow in gun safety? Is it really a good practice to rest the muzzle end of a sub machine gun in your crotch? Seems a bit odd to me but I know nothing about firearms.
No. It’s very, very bad practice to point a weapon, loaded or otherwise, at anyone you don’t intend to kill.
Sadly, it’s just another graphic reminder of how poorly Metropolitan Police firearms officers are trained.
Maybe the officer want’s to add some bullets-o-steel to the bag?
(S)CO19 carry their weapons on a chest harness, which is a bit awkward for sitting down with, the barrel is probably digging into the seat…
https://relentlesslife.files.wordpress.com/2012/04/article-2136289-12cd080e000005dc-432_634x416.jpg
The Met police may not be wonderfully trained – I’ve seen their shooting when I’ve been at the range at the same time as them – but they’re capable of not shooting black guys on traffic stops, which seems beyond the average US cop…
I wonder if this has any connection with this:
https://www.lawfareblog.com/someone-learning-how-take-down-internet
Let’s look on the bright side here. No one actually reads anything on the Internet. Case in point.
Meanwhile, nothing being mentioned on ‘regular’ news networks.
Does using an alternative DNS at the user end help with this?
(Might be a stupid question, I don’t completely understand how the Internet works.)
I just heard about it on NPR. It was on the BBC News Hour, IIRC.
sometimes… googles are 8.8.8.8 and 8.8.4.4 if you want to give them a try.
I’m more wondering about the prior day or two’s HAD post about /someone/ spreading a virus across IOT devices.. maybe they got spooked and launched their plan early because of it?
I’m in california and cannot access a number of sites including Digikey and Mouser… :(
same here in Los Angeles can’t access Mouser & Digikey
The hosts file shall be our savior!
I can’t get on .io pages now from my normal ISP. :/
Mobile works though. Best of luck, interwebs.
Trust me, we have people working on it.
Civ IV was just released, so you have that…
Civ VI ;)
yeah that too
Oh well it was nice while it lasted, I’ll miss the cat pics tho
Can I ask a stupid question? Why don’t more high profile sites multihomed their DNS to make these kinds of attacks harder. If they had primary DNS at Dyn, and secondary DNS at XYZ Company and a 3rd DNS provider at ABC company and everyone used different companies to do this wouldn’t it prevent this kind of issue? My DNS server would try the primary, if it failed it’d go to the secondary, etc…
I think the problem isn’t the sites listing with multiple DNS outlets, it’s on the user end to know when an alternative DNS server needs to be used.
No it shouldn’t be up to the user. DNS records allow for the domain holder to set up multiple Nameservers. This data is stored in the root servers. The way DNS works is if you make a request to your domain server for X domain it’ll go to the root server and find out what the name servers are for that domain name. It should in theory be able to go from server to server until it finds one that is available. Since the DDOS attack is against Dyn and their DNS servers having a multihomed setup should allow you to prevent this kind of attack as long as everyone doesn’t decide to use the same nameservers for their backups….
after talking with a friend the issue why this isn’t done is that nameservers are chosen randomly and that it could take up to 30 seconds to timeout before switching to the other, which wouldn’t really fix the problem..
That depends on your definition of fix — 100% of users see a problem until you fix it, or 50%/33%/25%/20% of users have a problem where it has a 30 second delay before it works, and DNS is cached, so once it works it will remain working for that user for the TTL times it out.
theoretically since you ask your local DNS to get it it should be cached for anyone else that uses DNS server, but yes I was thinking about that as well.
Its not how DNS works.
Problem is with companies that went down today, they all used only _one_ DNS provider (all eggs in one basket, often listing couple dns addresses all located in same rack) AND very low TTL pretending to be clever in order to GeoDNS. Low TTL means every DNS server on the planet must frequently check with primary. Knocking out primary meant all those records with LOL TTL (1-5 minutes) expired immediately and were gone.
Yeah. It doesn’t make sense to me that there isn’t more ‘distribution’ of the most commonly requested DNS data… Does everyone new page request need to go straight to Dyn (or other primary) every time to get IP for amazon.com, ebay.com, hackaday.com? Also, you’d think that if the last successful access to the primary was authentic, then even after the TTL, if the primary is unavailable, just keep serving the last good result.
I’m not expert at DNS so perhaps there’s reasons this wouldn’t work. Nonetheless, the strength of the internet is its distributed, so maybe it’s time to improve the DNS process.
That’s not really how it works. Your PC checks its local DNS cache and if not found (or expired), checks your router’s DNS cache (if it has one), then checks your ISP’s DNS servers and then keeps working its way up a tree of more authoritative servers until it reaches the root servers (/the/ authoritative source) or finds what it is looking for.
Usually this tree traversal is pretty limited (only a few hops, rarely past your ISP’s DNS for the bulk of requests), but it’s become trendy to use absurdly low TTL values to quickly invalidate DNS entries that would usually be cached quickly at lower (faster and more plentiful) nodes of the tree. Instead, requests are commonly propagated further up the tree until they hit a server that is controlled by the company the DNS is associated with, often so they can do stuff with the request information (DNS region blocking, etc.). It really does make the entire DNS operation more vulnerable when your system relies on the majority of DNS requests propagating way further up the tree than needed as you’ve just created a much easier target for attackers.
“An attack on this scale is most certainly someone with a very large pocketbook or a state actor (Russia, China, the US, UK, Germany, Israel, or the like) but that’s not a given.”
What makes you think that given the recent 620 Gbps record DDoS attack on KrebsOnSecurity using huge numbers of IoT bots and the recent release of the source code required to do so:
Spreading the DDoS Disease and Selling the Cure
https://krebsonsecurity.com/2016/10/spreading-the-ddos-disease-and-selling-the-cure/
Yeah, that was my first thought too. I don’t want to rule anybody out at this point, but I’m sure we’re all aware of the rising tide of IoT devices with poor security.
This interesting development:
WikiLeaks @wikileaks
Mr. Assange is still alive and WikiLeaks is still publishing. We ask supporters to stop taking down the US internet. You proved your point.
http://www.zerohedge.com/news/2016-10-21/wikileaks-reveals-heavily-armed-police-gathering-outside-ecuadorian-embassy-london
I run a caching DNS server at home and my phones VPN there. Hackers can take all the root level servers down and not affect me.
If it were only that easy. The best defense against a DDoS attack is often to move a site to a different IP address. But you can only do that if other DNS servers take heed of the Time-To-Live information in the DNS records on the servers that are in control of their domains.
So for example if you browse to hackaday.com, your browser looks up the IP address from your DNS server. The DNS server checks if it has the address in the cache and gives you the cached address if the cache is still valid according to the Time-To-Live data. But if the cache is stale, it’s still going to go through a lookup on a forwarding DNS server or (eventually) a root server.
If you set up your own DNS server and somehow make it ignore or falsify the Time-To-Live information, it may help to work around an upstream DNS server that’s offline, but the computers that look up hackaday.com on your DNS server will get stale information. And if hackaday.com moved to a different IP address to alleviate a DDoS, you won’t be able to reach it. But in the worst case if the hackers have also broken into the routing system of the Internet, they could possibly create a fake hackaday.com and run it at the original address. So now your computers will go to the new fake hackaday.com and you won’t even know it.
A hack of the DNS system would be very bad, that’s why it was designed to be extremely robust. I’d be surprised if this succeeds, even with state/country funding.
===Jac
Well it *is* that easy. This DNS attack is only effecting the US. In my country we cache the root name servers and every DNS below that is also cached so this can only possibly effect band new domain names and not any others.
I am actually surprised that the US is effected so badly when other countries are fine. You guys actually have the root name servers and you have this problem ??? What are you doing wrong???
What country is that? Elbonia? Every DNS server is a cache, but they all respect TTL, otherwise you will have problems (including security ones like hijacking services by buying server with old IP).
The root name servers are not a cache but they are a SoA. It seems that it is only the US that has been so silly as to put all their eggs in one basket.
Here is what the rest of the world has done so that we have *no* dependence on US root name servers –
http://www.root-servers.org/
Each of these non-US root server caches emulate SoA and attempts to honor TTL but will keep using old records should they fail to contact the actual root that they are caching by locally adjusting TTL / Refresh. From there and below is the normal domain based DNS and domain SoA servers.
That is why this only effected US / Europe. You guys need to fix that problem the same way because it’s a gaping vulnerability. I am pretty well shocked that this could even happen. I expected that the US had solved this vulnerability like the rest of the wold has.
Almost true. Except DNS has never been “extremely robust”. All the robustness that you can find in its design is the distribution. If you look carefully you will find out that every few years yet another problem with DNS was discovered, including some security issues. DNSSEC still isn’t mandatory for DNS providers.
Technically, DNS is the weakest link of the whole “web stack”. Most other issues are political.
Quote: “Except DNS has never been “extremely robust”
Well that was just proven to be the case in the US but even though the *rest of the world* uses the same NS/DNS hierarchy they were *not* effected!
DNS isn’t the “weakest-link” anywhere but the US. It is however a *critical* link so the rest of the world has mitigated risk.
So the “internet’ goes down ? BIG FRIGGIN’ DEAL! …. what did we all do before all this ‘tech’ ? Oh, that’s right we all somehow managed to survive. Perhaps if it did go down – hard – for awhile it would be a great learning lesson and a retro skill exercise. Of course the ham radio geeks will now chime in with the “when-all-else-fails” rubbish…(rolls eyes)…and that’s coming from a ham radio geek (and ‘prepper’).
We’ve lost the skills of our ancestors by being too reliant on technology.
Sorry friend, you’re barking up the wrong tree. The point here isn’t that we can’t live without our Internet. Had this been a storm that took down infrastructure we would be doing just fine.
The point is this is a deliberate act that is denying use of an important utility to a large population of people. Personal communication, education, business, and entertainment have all been built up around this utility (the Internet). The deliberate disruption of Internet service is an attack on society and that is why this story matters.
Not to mention, whether a disruption is intentional or not, there’s an awful lot of stuff that’s very important riding on the internet. The argument that wiping it out for a bit would “put us back to our roots” or something is downright fantasy. When all is said and done, a large disruption wouldn’t change people’s ingrained habits unless it was truly huge, and at that point the damage done definitely won’t be worth it.
It’s a tired trope often trotted out by caricatures of revolutionaries in fiction. “We just need a revolution to set things straight. then the world will be perfect” Except now you’re at the bottom of a hole, without even basic utilities like running water and often a ton of dead people, and when you finish crawling your way back up to where you were you’ll find that the situation hasn’t changed. You still have to do what you always had to do from the start: Small incremental change, one step at a time.
It took a long-ass time to build stable civilizations from lawless barbarianism/short-lived warlords (thousands of years!), and it seems people are quick to forget that. Most think a bomb-shelter worth of food will last them until society has been rebuilt, but in all likelihood and with arbitrarily many supplies, you’d be long dead and buried by then. The primary objective in any apocalyptic scenario is to prevent it from becoming an apocalyptic scenario, because you’ll literally pay for it a thousandfold.
That’s why it’s a good idea to have backup communications channels and local systems that can still operate if the Internet goes down.
If you connect something mission critical like the power grid equipment to the Internet you almost deserve what’s coming for being so stupid.
True
Hmmmm…
Internet over ham radio, anyone? ;)
Can I haz the internets at one Baud(s)????
:)
But no! Really!…
Bring back ASCII porn!!!! I missed out on an entire generation :(
I’m only about a quarter century old BTW.
Then there is ascii bubble porn where you just take a random page of text, blank out random circles of it and let your dirty mind do the rest.
as great as that sounds the HAM bands in specific are disallowed for this as most modern HTTP has been moved over to HTTPS and any sort of encryption is disallowed on amateur andor HAM bands, there are alternatives though, but you’d need internet to find out what equipment/modulation it uses
Not coming from HAM myself, a blanket ban on any form of encrypted/private communication seems downright insane.
Radio bands are not an unlimited resource. Especially on the frequencies that ham radio uses. The HF bands can have a signal that bounces around the earth on only 10W. If the signal is encrypted then you have no idea what kind of traffic it is carrying. Even on higher frequencies, you can use more power which can mean a signal that travels significant distance. Consider 1500W on wifi. I don’t think anyone has ever done it. I think 2 or 3 of the lower “channels” on 2.4ghz fit into the ham band.
HAM radio is not for commercial traffic and it’s not for anything that you would need encryption for, period. There are EMCOMM people who might disagree, but they don’t recognize the damage that encryption would do to the bands.
They should allow cryptographic signing (encourage it, even) if they don’t already. Then anyone can see the contents and verify that they’re really from what it claims to be from, but a hacker would have a hard time trying to forge a message without somehow getting a copy of the signing key.
Signing should be fine, in most countries the rule disallows ‘methods to obscure the meaning of the message to third parties’. A digital signature does not obscure the meaning in any way.
https://www.wired.com/2016/10/internet-outage-ddos-dns-dyn/?mbid=nl_102116_p3&CNDID=42858046
so THATs why people just can NOT drive today!
the second they try to pull up to the first corner after leaving thier house they just CAN NOT access the usual URLs that they usually do when going around the corner right outside thier house!!! so that leaves them “stranded” in the middle of an intersection half turned and looking down at thier legs/cellphone! caught red handed! … next time maybe the pages stop completely and they are stuck in thier driveway! no more cars!
PS: to all you web-while-driving people out there;
DRUNKS DRIVE BETTER THEN YOU
(when not looking at a cellphone)
I’m glad my network has a DNS server that caches everything for a while. Nice that nothing happened to stop this before hand. :-/
What’s the chance this was something to do with the US announcing it will respond to Russian cyber attacks in-kind? The U.S F’d something up and nuked their own DNS by mistake maybe? wouldn’t be the first time.
So that’s why the guys in the office couldn’t download there porn…
or watch the stuff.
We can’t have nice things.
so this is why mom got me up at 5am because etsy wouldnt load.
The internet is down? Yeah, no one can read what i write! I HAVE A SMALL WEEWEE AND I THINK EVERYONE SHOULD GO TO HELL. AND I VOTE FOR TRUMP.
Wow, feels good to be alone out here
Meh, Internet works again.
14 year old kid is on it. Hasn’t has caffeine yet today. Give it a few minutes.
Did he try turning it off and back on?
SE12 Episode 6, one of the best South Park episodes ever, short clip, can’t find clip for Kyle doing the reboot…
https://youtu.be/FeH4oDirZJA
I have an idea lets build a firewall a huge firewall that covers the whole internet, A big beautiful firewall and the best part is we are going to make hackers pay for it. Make The Internet Great Again
We shall set up our own network, discreet and discrete.
There will be a big beautiful port forwarding rule right in the middle, but your packets can only get through after Extreme Vetting.
I love it! #MakeTheInternetGreatAgain Ohh god I can’t wait to grab those packets by the pussy.
wired is now also reporting that IoT malware “Mirai” might be behind it..
https://www.wired.com/2016/10/internet-outage-ddos-dns-dyn/
meanwhile this wikileaks twit asks supporters to “stop taking down the US internet”..
https://twitter.com/wikileaks/status/789574436219449345/photo/1
Usenet works.
Who needs www?
Usenet leans pretty heavy on DNS these days. It’s decades since leased lines and diallup between roots and hubs had any significance.
* reluctantly bids farewell to cherished Hayes modem *
Goot times… good times.
Bring it all down pls
It’s probably George Soros trying to save Hillary’s campaign by disrupting WikiLeaks….
How about a discussion on what to do to help prevent such attacks. First the IOT should be architected not to need DNS and should be on separate networks from general computers. IOT objects generally do not move and with IPv6, would have fixed addresses. It is not quite that simple but that provides a basis for intelligent traffic monitoring. Next, router manufacturers should build in intelligent outbound traffic monitoring so that users can be made aware if they have a bot and can fix the infected devices. The next element Is for ISPs to monitor incoming traffic and be able to notify users of malicious traffic, with the ability to cut off that traffic at the edges for users who do not fix their devices. A good way to do this is to charge for excess data volume as is done for cell phones. ISPs who do not manage would, in turn, be penalized if they do not manage their traffic . There is a solution but commercial and political factors inhibit the ability to fix cybercrime and cyberwarfare. It is funny that the Internet was designed to withstand nuclear war but not cyber attacks. I guess the assumption was that the Internet only needed to survive a few minutes in case of nuclear war. I am working on a highly secure and available IOT architecture partially to help address the bot issue but mainly to ensure my IOT system is resilient when the Internet is attacked.
How about companies and users simply patch and secure their s**t! Also We wouldn’t be in this mess if tech companies selling IOT devices/cameras didn’t treat security as an afterthought.