NTP is one of the most interesting and important, but all too forgotten, protocols that makes the internet tick. Accurate clock synchronization is required for everything ranging from cryptography to business and science. NTP is closely tied around a handful of atomic clocks, some in orbit on GPS satellites, and some in laboratories. So the near-failure of one such atomic clock sparked a rather large, and nerdy, internet debate.
On December 17, 2025, the Colorado front range experience a massive wind storm. The National Center for Atmospheric Reassure in Boulder recorded gusts in excess of 100 mph (about 85 knots or 160 kph). This storm was a real doozy, but gusts this strong are not unheard of in Boulder either. That is no small reason the National Renewable Energy Laboratory (now the National Laboratory of the Rockies) has a wind turbine testing facility in the neighborhood.
Winds of this nature would not terribly interesting. However, the wind storm brought with it a particularly dangerous red flag warning outside of Boulder, a first for Colorado. Such high fire danger combined with damaged infrastructure prompted the local utility, Xcel Energy, to shut off power for hundreds of thousands of customers starting on December 17. Power was not regained until December 21 for many customers.
This outage came with all sorts of headaches to research institutions across Colorado. Not least of which was the National Institute of Science Technology’s (NIST) Boulder campus which houses a rather precise atomic clock. Due to predicted failure of NIST’s heat exchange, much of the normal monitoring equipment was unavailable to the scientists, further complicating the situation.
As was designed, once utility power failed, backup generators took over. But as the outage dragged on, indications came to the scientists in charge of the atomic clocks at NIST that one of the generators had failed. This prompted scientists to warn against relying on the Boulder NTP sources. The scientists running the clock feared complete failure of the hydrogen source clocks. Such failure would require a lengthy and complex re-start procedure once power was returned in the long term, and complete failure of a stratum one NTP source in the short term.
Further complicating the already bad situation was the fact that due to the dangers involved, the scientists could not reach the campus. So not only could they not confirm with certainty what issues the clocks may be experiencing, but they were unable to shut down the NTP servers. Fortunately, power was returned and the main source clock only drifted by a few microseconds. This is still far too much drift as would be preferred on a clock normally accurate in the range of nanoseconds, but perfectly usable for NTP which is only accurate to within a few milliseconds.
So this prompts the question, if such a key time source had failed, what would have happened? In short, not much. By nature of being so distributed, most servers have multiple NTP sources, often including GPS satellites. However, there would most certainly be any number of servers without multiple NTP sources configured. Websites hosted on such servers would be rendered inaccessible as HTTPS encryption handshakes require synchronized clocks. TOTP passkeys and FIDO hardware authenticators would likewise be unusable as both protocols rely on accurate time sources. So any two computers would be unable to properly execute protocols requiring synchronized time. Beyond the limited failures outlined above, its difficult to say what more the damage could be, but the effects are unlikely to be terribly dramatic.
If harnessing atoms to tell time sparks your interest, make sure to check out this atomic delay clock next! [Jeff Geerling] also has a nice discussion of this power outage that you might like.
s/Reassure/Research/
“Winds of the nature in and of itself is not terribly interesting” should be corrected somehow, but my ECC has merely reported a multi-bit error on this one, failing to offer a correction.
The whole thing needs a good going over.
Wut?
Someone also needs to mention that it is the NCAR – National Center for Atmospheric Research. Somone’s autocorrect appears to have gotten in the way. Can the OP please correct this.
For context, at time of writing my comment above, it was referred to as the National Center for Atmospheric Reassure, hence the comment.
Or Numerical Computation of Amalgamated Reasoning.
Written by AI? Especially the first sentence there – “but all to forgotten” WTF?
Fenix’s hallucinating, better call 112 ASAP as it might be a stroke.
HAD needs to re-boot their AI that writes articles.
AI is not reliable yet or for the foreseeable future.
Glad I have my own NTP server.
At first I was glad to see typos, at least it’s written by a human. But then I thought, what if the AI is inserting typos to appear human? I’ll take typos over AI any day.
Basically redundancy is the gift that keeps on giving.
??? Really?
AFAIK at least TOTP only need several second accurate accuracy (if not much less like 30 to 60s).
Where do time differences below a few seconds factor in with HTTPS? I thought it was more or less just the expiration date of certificates.
Isn’t timing during the handshake more important than the “actual” time on both ends?
Why should the time on both ends even be compared?
If I set my computer’s time a few hour’s back or forth and disable NTP I couldn’t surf the net anymore???
How do FIDO hardware keys get millisecond accurate time? (“know” even less about FIDO so this might be a bonkers question)
Not a weird question in the slightest. As it turns out the HTTPS handshaks (not the encryption scheme itself) do depend on accurate clock synchronization. TOTP is actually in most cases can have more like 90s of clock desynchronization as most clients will take the code before and after the current code. And FIDO is a lot like TOTP in a hardware package so that a bad actor can’t just grab and emulate the hardware key’s signature.
What’s described above is more of an absolute worst case failure of clock drift on the order of minutes. That is a highly unlikely, but not completely inconceivable, situation in which a quantum source fails and erroneous readings cause large time drifts.
Fortunately, we didn’t find out what exactly would have happened!
I can’t find anything on that. Nothing on requiring “synchronized clocks” from the article nor any dependency on “clock synchronization”.
All I can find is that the time/date must match “roughly” for the certificate validation to work reliably (if near the end or start of it’s lifetime). Same goes for checks against the CRLs.
Pretty sure several seconds or even minutes apart does not count as “synchronized clocks”.
If that were true, lot’s of computer would not be able to browse the “interwebs”, but only rough accuracy is required to verify that the certificate has not expired yet.
Absolutely correct on all the points, especially the ‘human’ 90 seconds of clock time. Worked for a big global bank, 430,000+ people authenticating and clock sync on a global basis especially with global servers distributed becomes critical to architecture & engineering teams.
Hope your 2110 clocks aren’t just relying on a single source. PTP time is everything. A whole lotta fun if that clock goes in the port – O – John
News stories credited an old diesel generator with the recovery of the critical atomic clock. I keep a 2 KW military diesel unit around myself.
Another IA post without any hack?
I want a WWVB time server(for NTP). I don’t think they are made anymore???
It’s all GPS based now(disaster waiting to happen)
NTP is available in a variety of packages for different hardware and applications, so long as it can see the internet periodically.
World time normal ist distributed across the world, in satellites (GPS, Galileo), their operations centers, and research and standards facilities all over the world.
Within the US you forgot Fort Collins, Colorado and Gaithersburg, Maryland.
This is an topic core to information security, correlation, authentication. Clock skew occurs on 100% of information technology due to the fact that it’s not designed to be a hydrogen or cesium stratum 1 clock source. This is why NTP syncing is key, to correct skew/drift and is done automatically usually across a distributed ‘voting’ arrangement of NTP configuration for a piece of equipment. When that NTP sync is no longer available, skew/drift and what my team called ‘runout’ would start accumulating – we never got to the apocolyptic 5 minutes for Microsoft Active Directory authentication where things would get Really, Really, bad but resilient NTP design using GPS satellites, and distributed NTP sources and voting are key to maintaining 100% uptime.
I have worked in a lot of datacenters and every one of them has an NTP locally that is synced to multiple sources. The servers in the datacenter are pointed to a local NTP server. So, if the higher stratum sources all go down, then your local NTP free runs (with presumably some minor drift) and your servers see pretty much nothing so there was never any risk to any services. Remember that if you lose NTP completely, you rely on local clock without correction, more than close enough for all web services including HTTPS.
Yeah, anything with a quartz clock keeps pretty good time, and you can’t synchronise clocks all that accurately over a packet-switched network anyway. If there were parts of the internet that depended on atomic-clock accuracy, they’d be failing all the time.
You do still need a central benchmark in the long term though, even if it’s just a sundial. Plus there are things like astronomy where it is important to have a correct absolute time.
Meanwhile someone is taking steps in the right direction. Potential small version of such that won’t need backup generators and separate/dedicated facility.
https://spectrum.ieee.org/silicon-clock
December 18/19 I had a dream about NTP failure, although I hadn’t known it happened. I was talking with my friend and looked at my watch to check if I have to go, but the hour kept changing. I asked him what time it is, but his watch had the same problem. Then I woke up to another dream and clocks were behaving the same way, except of my mum’s watch that was offline. It occured that I felt the NTP failure while dreaming.