The phones! They were one of the basic utilities of the 20th century, and were just about as reliable as death and taxes. Even when then power grid went down, you still had a fair shot of getting a phone call through thanks to the reliability of the Plain Old Telephone Service.
Today, we eschew the simplicity of copper and mechanical switches for the supreme bandwidth and capability of high-speed cellular connectivity. With that, we accept that the additional complexity comes with a risk of complicated failures that bring everything tumbling down. Australia’s largest telecommunications provider found that out to its peril just a few short months ago.
Networked Failures
Generally, we expect our telecommunications networks to be supremely reliable. There is no moment of the day whenย someone doesn’t need to make a call, particularly in emergencies, and the wheels of industry and commerce depend on constant connectivity these days. Tolerance for failure is generally very thin. Despite this, and the efforts of engineers to maintain uptime at as many nines as possible, Telstra fell badly short on July 8th, 2026. The company had a nationwide outage that affected 8.8 million people, leaving them unable to make calls or connect to the network at all.
The cause of the outage would prove to be particularly embarrassing. Telstra owns and operates a highly advanced cellular network, offering 4G and 5G service across the nation’s cities and much of its outback areas. The company may outwardly appear to be a shining beacon of modern connectivity, but there was something dank lurking in the company’s server closets. Namely, three aging network time servers that had the capacity to bring the whole system to its knees.
The culprit? A Microchip Technologies SSU 2000 NTP server. The model dates back to the early 2000s. Twenty four years later, Telstra still relied upon three of the units to provide network time protocol (NTP) services across its network. The servers were generally perfectly adequate in this role on any given day. That was, until the Melbourne server had a wobble.
A technician was working in the early morning to replace a backup power feed in the chassis housing the server. This caused the server to be rebooted at 3:38 AM, which normally would not be a problem. However, at some point in the last two decades or so, the server had gone through a configuration change. While it was originally intended to be a Stratum 3 NTP server, getting its time reference from a Stratum 2 unit, that process had failed at some point. It had been reconfigured instead to use its internal GPS card to gain time directly from the satellite network instead. Unfortunately, the server was also remarkably old, and suffered from a well-documented GPS date rollover bug, such that when it rebooted, it reported the time as 2006 rather than 2026.
The problem that stemmed from this was because time is critical to authentication. An endless cascade of devices downstream of the NTP server picked up the wrong time, and started using it to sign digital certificates and the like. This immediately caused other systems on the network to reject the spurious traffic with certificates that were 20 years out of date. The impact was swift and vastโTelstra was quickly facing a nationwide outage affecting millions of customers.
The issue was first detected at 4:20 AM. The naughty server was isolated by 7:11 AM, but it would take until 10:30 AM to identify all the network components which had received erroneous time data. It took several hours furtherโuntil 4 PMโto properly quell the NTP issues. In the meantime, a significant portion of the country had seen its phones offline all day, and entire rail networks had ground to a halt as their Telstra-based communications systems went completely offline.
Later submissions to a government inquiry would reveal Telstra had received two reminders to patch the GPS card, in 2020 and 2022. The vendor itself had issued warnings about the GPS rollover bug as early as November 2000. A decision not to fix the bug had been taken as recently as January 2026, because the undocumented change to have the server rely on GPS time was unknown, and thus the update was considered unnecessary. Simply patching the system would have prevented the issue from ever occurring in the first place.
The issue once again brought telecommunications availability in Australia to the forefront of the conversation. Repeat outages across Australian mobile networks have led to particular concerns about the ability for people to reach emergency services by calling Triple Zero from mobile handsets. The latest failure on Telstra’s behalf has led the local telecommunications industry to issue new guidance to the public on what to do when a call to Triple Zero doesn’t go through.
Modern handsets are designed to switch to a different cellular network in the case an emergency call can’t be connectedโa process called emergency camp-on. However, this process takes time, and the caller will often hear silence on the line while the phone is attempting to connect. The new advice is that callers should hang up and try again straight away if their first call to Triple Zero doesn’t connect within a few seconds. On the second call, though, the phone should be given up to a minute to find another network to get the call through.
In the case of this outage, camp-on functionality workedโsome 3,200 Triple Zero calls were passed to Optus and TPG networks when Telstra’s failed. However, there were some ongoing issues that saw a further 604 Triple Zero calls fail over the period to 2 PM the next day.
Overall, Telstra’s failure was a major one. It’s rare for a major network to go down so completely and over such a wide geographical area. The fact that it happened because of an undocumented change to an ancient network appliance is all the more embarrassing. It will drive home the message that documenting even seemingly minor changes is important, with the lesson likely to be told in the halls of the Australian telco for some decades to come.
Ah, yes. Who doesn’t miss the reliability of the old POTS lines. During extended power outages we would use electricity from ours to heat the house and run the well pump. It wasn’t enough for full power but it kept the showers going and the pipes from freezing. Yah, you weren’t technically supposed to do that but no one bothered about it so long as it was emergency only.
I’m calling BS since the line voltage is too low and the current capacity is minimal.
90V? The trick was to pair up with a neighbor and call one another over and over. The ring voltage was where the power is at.
This is good trolling
Mark
So what would you guess the Ringer Equivalence Number (REN) of your system was?
No idea but it was very high. A tech did mention it once. We just told him the inlaws live with us and we have to have a bunch of those extra loud extension bells. Can you imagine if they fell and couldn’t get up!
Correct. We did that too but primarily to charge our electric car and power our cable modem because the kids cant live without internet.
During the Iberian power outage (Spain + Portugal, both completely offline) the coordination between plantswas completely over the phone, according to my reading of the entsoe report.
I’m astounded that this seems to have worked without -seemingly- as much as a single hitch.
If the power grid depends on the phone network and vice versa, we are a few small generator failures away from a prolonged outage!
I wonder what other things rely on this GPS time code reference.
Considering a fair amount of cryptography seems to need a shared clock.
It might only be a matter of time before someone tries spoofing a GPS signal in range of one of these lynchpin services. Perhaps even with code injection or followed by a jamming signal, if there’s no fall back system in place, it could prove somewhat of a weak link.
It is just the normal.
23./24. Juni 2026 at 22:30 Deutsche Bahn changed a network switch. It failed. The entire German rail network was totally down for 2 hours, with massive aftereffects the following day.
All good. They are always late anyway.
That was a symptom of people making the rules without a basic understanding of the system: as reaction to an accident which may eventually perhaps have been prevented if there had been continuous radio communication along the track, a 100% coverage by GSM-R was mandated. Thus all the cables, phones and radio were replaced by some special cellphone net, and having an outage of that was defined as an emergency event in which everyone has to stop.
In another country not far away the rule instead is, that the safety of the railway must not depend on establishing a communication link. The safety of railways here[tm] traditionally is ensured by the signalling, not by phone.
I could write books about the unintended consequences adding lots of friction in day-to-day life.
Its a bit like someone got sick from a bottle of spoiled milk, and now you have to burn all your food when the fridge fails, things like cookies or conserves being discontinued due to their incompatibility with the mandated fridge.
i’m often frustrated by how complicated ntp (and programs like ntpd and chrony) are. the remote server always has the right time. and the local system time is always wrong (RTC time was never set, was set to wrong TZ, and/or drift), and a lot of things (like VPN) don’t appreciate having a radically wrong time. so i’m often frustrated by a system that won’t even start to adjust the time until it gets 3-5 remote servers to agree within a milisecond, and then only performs micro-adjustments to maintain a monotonic clock whether it has to make up a gap of 5 seconds, 5 hours, or 5 years. it seems like it’s always either within 1ms or completely wrong…when really, reliably within 500ms would be generally more than sufficient for my uses, and easily accomplished with a more naive client (or with the right configuration options, natch).
but the prospect of a single server going off by 20 years sure does make all that complication look clever! a conservatively-configured ntpd would never be fooled that way
Quite a number of US municipalities/counties fell victim to THAT EXACT failure, but it was conveniently omitted from the US news. Local news covered it, but as-matter-of-fact a opposed to full investigation, yet when omething comparable happen elsewhere very suddenly bam, it is big deal.
If I remember right, year 2016, MULTIPLE occurrences, to be exact, and it is NOT the only one. Providers (you know who they are the two largest monopolies, or communist parties, no difference) were sued, etc etc.
It is predictable and can happen again because internet is not covered by the telecom regulations the way the old copper line were. If internet goes down (netsplit, etc), providers are like yeah, sure, we don’t own it, we merely sell the connection for profit, aha, whatever.
One notable downtime I personally witnessed was in the middle of the working day, almost 4 full hours, EVERYTHING was offline, police, hospitals, etc. Trade, too, don’t forget that, all the online payment processing, the outage was just plain massive. The oddity of it all? The county affected literally SITS on the northeastern backbone running alongside I95, it is not like middle of Wyoming or in the Rockies, right there, atop of the fiber.
(among earlier and rather funny stories from decade earlier – excavator was digging up leaking sewage pipe and happened to pull the fiber connecting local office buildings of no particular/significant importance to the already mentioned backbone along I-95. Yes, you guessed it, SOMEHOW almost entire segment of this part of the county happened to run through that particular fiber, which was put there in like early 1990s – I personally knew the founder of the company (now defunct, sold off and closed) who paid for THAT exact splice into the backbone in the 1995 or so, out of his personal investment moneys, back then it was one of the first local splicings, btw. That particular connection point, it literally became #1 local connection, not aware it is important/critical, but once broken, disconnected not just few city blocks, but almost entire zip code).
Obviously, competing providers do NOT want to pick up their competitors’ traffic for them, which is where competition should be forced out of the critical infrastructure planning, but who am I to know any better. All I know police, hospitals/ER, EVERYTHING was offline and there was no way to reach anything or anyone for help. (I wasn’t affected, most of my work is reasonably offline, so I can continue working unabated, just delayed a bit).
Sat might have done a sit-in for some things even before Starlink.
Me too little serverโฆ
same…
4:20 and 7:11 in one article?
Something is afoot!
Over fifty years ago a friend told me about the time his father, a contractor, inadvertently cut a communications cable in rural Kansas (if I recall correctly). In short order, his boss, and his boss’s boss, and his boss’s boss’s boss all arrived at the site. Turned out he had broken communication between all eastern and western US military bases. I’ve always suspected that incident was one factor leading to the creation of the internet, which after all was first promoted as a communication system with no single point of failure that could be attacked in a nuclear war.
Hey, I had a bunch of those SS200s in my plant — had to retire them all years ago because they had a y2k-like bug that would have caused the radios to go offline on a specific date. Vendor provided adequate warnings and everything.
No excuse for running units like these more than 3-5 years at a time anyway — especially since the roof-mount antennas and in-line signal amps are both exposed to weather and are powered continuously anyway.
As someone who has worked as a telecommunications technician I can certainly tell that the real problem would not have been, The technicians repeatedly decided not to fix the problem. It would be “this server will be replaced in the next convenient Budget period”.
‘We have to wait for the new ownership’ ‘Next budget cycle’ ‘Maybe next year’ the road to failure heck is paved with idiot managers and bean counters
Yes, technicians love to update and fix things. Bean counters, not so much.
I expect more of this as systems get older, more complex, and the people who built the lower levels are no longer around. A lot of people are not full stackers in any tech domain these days, most float around on the highest levels of abstraction and the bare metal people are rare.
reminds me of this old yarn from years past:
https://www.thebdr.net/western-unions-time-was-up-years-ago/
Small correction- the device referenced in the article is the SSU-2000 (see https://www.microchip.com/en-us/products/clock-and-timing/systems/telecommunication-clock-systems/ssu-2000 ), not the Syncserver S200 shown in the adjacent picture (the SSU-2000 is a different form factor and doesn’t have a display).
I used to rack 1u timesevers for secure datacentres but they always were set up to pick up from a RF based time source so everything worked when the network was cut off from the internet.
I wonder how many national telco’s are still quietly running aging netnumber titan’s for call number mapping on nfv networks, unsupported because they have been out of support for decades. I know of at leat one ;)
Telco’s are a interesting thing to work security deep inside of.