Inside The Mysterious Global Navigation Outage You Probably Didn’t Notice

The entire world has come to depend on satellite navigation systems in the forty or so years since the first Global Positioning System satellites took to orbit. Modern economies have been built on the presumption that people and assets can be located to within a meter or better anywhere on, above, or even slightly under the surface of the planet. For years, GPS was the only way to do that, but billions have been sunk into fielding other global navigation systems, achieving a measure of independence from GPS and to putting in place some badly needed redundancy in case of outages, like that suffered by the European Union’s Galileo system recently.

The problem with Galileo, the high-accuracy public access location system that’s optimized for higher latitudes, seems to be resolved as of this writing. The EU has been tight-lipped about the outage, however, leaving investigation into its root cause to a few clever hackers armed with SDRs and comprehensive knowledge of exactly how a constellation of satellites can use the principles of both general and special relativity to point you to your nearest Starbucks.

Navigation and Then Some

The idea behind Galileo goes back to the late 1990s, when the powers that be within the European Union decided that a global navigation satellite system (GNSS) run primarily for the benefit of civilians and without the potential for disruption by foreign powers would be a smart idea. The basic structure of the system combined the space and ground elements needed to provide positional precision of 1 meter free to all users and without fear of military preemption short of large-scale conflict. Customers in need of high-resolution positioning would pay to have access to a 1-cm precision encrypted signal, making Galileo the most precise native GNSS by far. Other features were to include a search and rescue capability, wherein the satellite constellation listens for signals from emergency transponders from vessels at sea or backcountry hikers, dispatches emergency responders, and transmits an acknowledgment to the sender.

The Galileo constellation. Note the inclinations of the orbits, which are configured to ensure optimal coverage at higher latitudes. By Lukas Rohr CC BY-SA 3.0

The Galileo project was officially launched in 2003 but suffered from years of funding issues, squabbling with competitors like the USA over frequency sharing, and general governance woes. These issues were eventually resolved enough that two experimental satellites were launched, one in 2005 and another in 2008. These served as testing and validation platforms, and once all the boxes were checked, more satellites joined them, ground stations were built, and the infrastructure needed to run a modern GNSS was assembled.

By the end of 2016, Galileo reached its Early Operational Capability (EOC) phase, with about 18 of the planned constellation of 30 or so satellites aloft. The EOC was sort of a beta test, with early adopters offered only a weak signal. Even though more satellites have been launched — there are now 26 birds in orbit — and the signal strength has been increased, Galileo has not yet progressed to the Full Operational Capability stage, or FOC. That transition is planned for sometime in 2019.

It’s About Time

A Galileo Full Operational Capability (FOC) satellite. About 19 m span across the solar arrays. Source: The indispensible Gunter’s Space Page

During its EOC phase, Galileo has suffered from more than one serious problem. In early 2017, problems with some of the high-precision clocks aboard four of the satellites surfaced, with some of the birds suffering an outage in both the rubidium atomic clock and the hydrogen maser time standard. This caused an understandable stir in that the clocks were supposed to have a 10-year on-orbit lifespan. The clock issues did not result in any service outages or degradations, since every satellite carries three backup clocks, but it did cause a shakeup and a review of yet-to-be-launched hardware.

While Galileo emerged from the clock issue relatively unscathed, it would not be so lucky the next time. On July 11 of this year, the Galileo system suffered a major outage, with 24 of the 26 satellites listed as “Not available” for reasons unknown, with the other two listed as “Testing”. While the GSA, the EU agency responsible for the Galileo system, at first issued advisories to customers that the system was to be used with caution, and later that the system was not to be used at all, they were not particularly forthcoming as to the root cause of the outage for at least four days following its start. Even now that they have announced that service has been restored, there has been no explanation of the issue beyond it being a “technical incident originated by an equipment malfunction in the Galileo ground infrastructure.”

With effective silence apparently being the policy of the GSA, it’s up to hackers to try to determine what happened with Galileo. Knowing that any GNSS is basically a set of really precise orbiting clocks that need to be carefully synchronized and updated, and with the admission by the GSA that the problem originated in the ground segment, it seems logical to go looking for the problem there.

Stuck in the Past

That’s what Daniel Estévez (EA4GPZ), a radio amateur and a GNSS engineer, did with a Lime SDR receiver and GNSS-SDR, an open-source platform for looking at data from navigation satellites. In an excellent write-up, Daniel details what he found. During the outage, the signals coming from the satellites were fine in terms of signal strength and modulation, and basic data, like the week number that has been the source of problems with the GPS system before, were all as expected. But in reviewing the ephemeris, or description of the orbit of each satellite, he discovered something strange: the satellites seemed to have the wrong idea of where they were in space.

All satellite orbits vary slightly over time, and since the position the satellite is in when it transmits a time signal is used by GNSS receivers to determine location, the ephemeris needs to be precisely known. By what Daniel was seeing, the ephemerides for the Galileo constellation hadn’t been updated since July 11 — pretty much when the degradation started. With the satellites unaware of where they were, the location solutions calculated by receivers got worse and worse as they drifted further and further from their positions on the 11th.

Daniel’s data agrees with what an anonymous source within Galileo who cited “a problem of the PTF [Precise Timing Facility] in Italy” as the root cause of the outage. The PTF is the source of ephemerides for the whole constellation, and if there was an issue there it would certainly manifest itself in just the way that we saw. There has been no official confirmation of this from GSA, but given that they’ve kept their cards close to the vest all during this last week, don’t hold your breath waiting for more information.

In the end, casual Galileo users probably never even noticed the outage, since most phones that support Galileo also support GPS and GLONASS. As long as everyone on the planet is getting along, each system serves as a backup for the others, which in cases like this failure is a welcome level of redundancy. And as GSA officials are quick to point out, Galileo is still officially in EOC testing, so things like this are expected to happen. Still, a little more transparency might be in order in the future, so that hackers like Daniel and the others who probed the mysteries of this outage can turn their talents to better pursuits.

[Featured images: European Global Satellite Navigation Systems Agency]

19 thoughts on “Inside The Mysterious Global Navigation Outage You Probably Didn’t Notice

  1. Galileo and GPS aren’t in competition, since they operate seamlessly using BOC modulation and GPS officially provides the missing coverage for when the Galileo constellation is not in view with too few satellites in orbit.

    BOC is basically a split spectrum modulation that allows the two systems to operate on the same band, so receivers can switch seamlessly between. It’s kinda like listening to a radio station with two DJs talking simultaneously on top of each other, and you can select between them because one is talking in bass and the other in falsetto.

  2. I wonder if tight-lippedness has anything to do with not wanting to expose the seeming ridiculousness that these devices, which inherently communicate with earth, base both their temporal and spatial self-awareness on dead-reckoning.
    Worse than that, dead-reckoning set to universal-coordinates rather than earth-coordinates, which are what really matter, when determining coordinates *on earth*.

    [Sidenote, why don’t I get subscription-request/notification-emails anymore?]

    1. A satellite in orbit has a very predictable track. Ground stations determine the orbit (which is uploaded as ephemeris data to the satellite, along with synchronising the on-board clock). It therefore has no need to fix its own position. The results are good enough, in the case of the Galileo cluster, and when it is working correctly, to give you a fix accurate to within centimetres.

    2. The Galileo site says that they’re going to start up a formal inquiry shortly. They will want to get it right more than get it done fast, though, so I still wouldn’t expect answers any time soon.

      But when they do come, they’ll be definitive.

  3. One interesting thing about this outage is that it suggests Galileo is much less robust than GPS. Even the original GPS satellites could continue updating their ephemeris data with stored estimates for between 14 to 60 days after the last contact from the ground segment; obviously there’s still some loss of accuracy over time with this method but it’s better than leaving stale data in place, especially if ground stations might mistake it for fresh once the timestamp wraps. (I don’t think this feature was ever needed.) The current GPS satellites can also autonomously calculate new ephemerides and clock information using cross-links between satellites without any help from the ground, meaning that in theory they can keep on working at almost full accuracy for months after a complete failure of the ground segment. Galileo doesn’t seem to have anything like this.

    1. Call in planned obsolescence. There’s barely enough Galileo satellites as it is to function as a global positioning system, and they’ll be EOL before the whole constillation is up, so the first thing they have to do after it’s finally “done” is to replace the entire thing.

  4. The bigger problem thatn the described lack of data update is that there is a backup facility for exactly this data-updating in Bavaria (Germany). The handover from Italy to Germany hasn’t worked as supposed (which could all be read in newspapers here in Europe, so it didn’t take any hacker to figure it out – and it has been widely discussed in technically interested public, so literally everyone who is interested in sat-navigation knew about the issues).
    If that handover had worked, the control data would have been sent up in time.
    What the Galileo project needs to figure out – aside from its obvious short-term hardware perspective – is “European Cooperation”, which is not a thing in any matter. Whenever a slightly larger project involves more than one country around here, things are likely to go belly up rather sooner than later.

    1. Of course they didn’t manage to handover between Italy and Germany. There’s a major flaw in it that it relies on cooperation between EU nations.
      About the only thing they’ll agree on is to find a way to prevent it working in the U.K. post Brexit.

      1. Seems like it’s been even worse than what we both thought.

        They didn’t even bother to plan the system REDUNDANT. They obviously only had TWO static systems, when one fails and the other was just now updating (which happened now, as it seems) – bingo: Bullshit.

        They might have wanted to ask someone who has done computer work before. Like any grammar school kid. But it looks like they only had academics at hand – so things were PLANNED to go southways.

  5. I kept an eye on this after the GALs started flagging as unhealthy on that first day. Even after they posted an advisory notice on their site, they took days before they posted a notice on their twitter account…

  6. Galileo sounds like a repeat of European color television standards. Rather than learning from, and building upon, what the Americans had accomplished, Europe implemented another television standard, that solved few problems and introduced many others. (Actually *several* competing standards, mutually incompatible.)

    And so with Galileo, with the added insult that it works “really well”, except with gaps in coverage, when it then relies on American GPS to fill in the gaps. GPS which was implemented 30~40 years earlier.

  7. For the ones that think that prediction of spatial position and timing is so simple, let me just give a little glimpse on the challenges:
    ever heard of the Einstein and his relativity theory? Also thought about the speed of these satellites relative to the Earth?
    I know, relativity is not an issue on Earth, but these satellites are travelling at speeds where relativity needs to be taken into account, especially when dealing with timing… While travelling, the clocks in the satellites are experiencing a significant time shift relative to the clocks on Earth that needs to be corrected – despite being precise, the satellite clocks need to be adjusted regularly in correspondance with the clocks on Earth

    1. The relative speed should make them 7 usec/day *slower* than a clock on Earth’s surface, but actually they’re running 38 usec/day *faster* because they’re also affected by being higher up the gravitational well, and that gravitational effect for them is 45 usec/day *faster*.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.