NTP is one of the most interesting and important, but all too forgotten, protocols that makes the internet tick. Accurate clock synchronization is required for everything ranging from cryptography to business and science. NTP is closely tied around a handful of atomic clocks, some in orbit on GPS satellites, and some in laboratories. So the near-failure of one such atomic clock sparked a rather large, and nerdy, internet debate.
On December 17, 2025, the Colorado front range experience a massive wind storm. The National Center for Atmospheric Reassure in Boulder recorded gusts in excess of 100 mph (about 85 knots or 160 kph). This storm was a real doozy, but gusts this strong are not unheard of in Boulder either. That is no small reason the National Renewable Energy Laboratory (now the National Laboratory of the Rockies) has a wind turbine testing facility in the neighborhood.
Winds of this nature would not terribly interesting. However, the wind storm brought with it a particularly dangerous red flag warning outside of Boulder, a first for Colorado. Such high fire danger combined with damaged infrastructure prompted the local utility, Xcel Energy, to shut off power for hundreds of thousands of customers starting on December 17. Power was not regained until December 21 for many customers.
This outage came with all sorts of headaches to research institutions across Colorado. Not least of which was the National Institute of Science Technology’s (NIST) Boulder campus which houses a rather precise atomic clock. Due to predicted failure of NIST’s heat exchange, much of the normal monitoring equipment was unavailable to the scientists, further complicating the situation.
As was designed, once utility power failed, backup generators took over. But as the outage dragged on, indications came to the scientists in charge of the atomic clocks at NIST that one of the generators had failed. This prompted scientists to warn against relying on the Boulder NTP sources. The scientists running the clock feared complete failure of the hydrogen source clocks. Such failure would require a lengthy and complex re-start procedure once power was returned in the long term, and complete failure of a stratum one NTP source in the short term.
Further complicating the already bad situation was the fact that due to the dangers involved, the scientists could not reach the campus. So not only could they not confirm with certainty what issues the clocks may be experiencing, but they were unable to shut down the NTP servers. Fortunately, power was returned and the main source clock only drifted by a few microseconds. This is still far too much drift as would be preferred on a clock normally accurate in the range of nanoseconds, but perfectly usable for NTP which is only accurate to within a few milliseconds.
So this prompts the question, if such a key time source had failed, what would have happened? In short, not much. By nature of being so distributed, most servers have multiple NTP sources, often including GPS satellites. However, there would most certainly be any number of servers without multiple NTP sources configured. Websites hosted on such servers would be rendered inaccessible as HTTPS encryption handshakes require synchronized clocks. TOTP passkeys and FIDO hardware authenticators would likewise be unusable as both protocols rely on accurate time sources. So any two computers would be unable to properly execute protocols requiring synchronized time. Beyond the limited failures outlined above, its difficult to say what more the damage could be, but the effects are unlikely to be terribly dramatic.
If harnessing atoms to tell time sparks your interest, make sure to check out this atomic delay clock next! [Jeff Geerling] also has a nice discussion of this power outage that you might like.
s/Reassure/Research/
“Winds of the nature in and of itself is not terribly interesting” should be corrected somehow, but my ECC has merely reported a multi-bit error on this one, failing to offer a correction.
The whole thing needs a good going over.
Wut?
Someone also needs to mention that it is the NCAR – National Center for Atmospheric Research. Somone’s autocorrect appears to have gotten in the way. Can the OP please correct this.
For context, at time of writing my comment above, it was referred to as the National Center for Atmospheric Reassure, hence the comment.
Or Numerical Computation of Amalgamated Reasoning.
Written by AI? Especially the first sentence there – “but all to forgotten” WTF?
Fenix’s hallucinating, better call 112 ASAP as it might be a stroke.
HAD needs to re-boot their AI that writes articles.
AI is not reliable yet or for the foreseeable future.
Glad I have my own NTP server.
Basically redundancy is the gift that keeps on giving.
??? Really?
AFAIK at least TOTP only need several second accurate accuracy (if not much less like 30 to 60s).
Where do time differences below a few seconds factor in with HTTPS? I thought it was more or less just the expiration date of certificates.
Isn’t timing during the handshake more important than the “actual” time on both ends?
Why should the time on both ends even be compared?
If I set my computer’s time a few hour’s back or forth and disable NTP I couldn’t surf the net anymore???
How do FIDO hardware keys get millisecond accurate time? (“know” even less about FIDO so this might be a bonkers question)
Not a weird question in the slightest. As it turns out the HTTPS handshaks (not the encryption scheme itself) do depend on accurate clock synchronization. TOTP is actually in most cases can have more like 90s of clock desynchronization as most clients will take the code before and after the current code. And FIDO is a lot like TOTP in a hardware package so that a bad actor can’t just grab and emulate the hardware key’s signature.
What’s described above is more of an absolute worst case failure of clock drift on the order of minutes. That is a highly unlikely, but not completely inconceivable, situation in which a quantum source fails and erroneous readings cause large time drifts.
Fortunately, we didn’t find out what exactly would have happened!
I can’t find anything on that. Nothing on requiring “synchronized clocks” from the article nor any dependency on “clock synchronization”.
All I can find is that the time/date must match “roughly” for the certificate validation to work reliably (if near the end or start of it’s lifetime). Same goes for checks against the CRLs.
Pretty sure several seconds or even minutes apart does not count as “synchronized clocks”.
If that were true, lot’s of computer would not be able to browse the “interwebs”, but only rough accuracy is required to verify that the certificate has not expired yet.
Hope your 2110 clocks aren’t just relying on a single source. PTP time is everything. A whole lotta fun if that clock goes in the port – O – John
News stories credited an old diesel generator with the recovery of the critical atomic clock. I keep a 2 KW military diesel unit around myself.
Another IA post without any hack?
I want a WWVB time server(for NTP). I don’t think they are made anymore???
It’s all GPS based now(disaster waiting to happen)
World time normal ist distributed across the world, in satellites (GPS, Galileo), their operations centers, and research and standards facilities all over the world.
Within the US you forgot Fort Collins, Colorado and Gaithersburg, Maryland.