The Trouble With Hubble: Payload Computer Glitch Stops Science At The Space Observatory

The Hubble Space Telescope’s remarkably long service life and its string of astonishing contributions to astronomy belie its troubled history. Long before its launch into low Earth orbit in 1990, Hubble suffered from design conflicts, funding and budgetary pressures, and even the death of seven astronauts. Long delayed, much modified, and mistakenly sent aloft with suboptimal optics, Hubble still managed to deliver results that have literally changed our view of the universe, and is perhaps responsible for more screensaver and desktop pictures than any other single source.

But all of that changed on June 13 of this year, when Hubble suffered a computer glitch that interrupted the flow of science data from the orbiting observatory. It’s not yet clear how the current issue with Hubble is going to pan out, and what it all means for the future of this nearly irreplaceable scientific asset. We all hope for the best, of course, but while we wait to see what happens, it’s worth taking the opportunity to dive inside Hubble for a look at its engineering and what exactly has gone wrong up there.

Above It All

The idea of putting a telescope in space, high above the roiling atmospheric soup we all live near the bottom of, was not exactly a new one even in the early 1960s, when humanity’s first tentative steps into space actually made such an audacious plan feasible. Before they had even put a boot on the Moon, NASA aspired to put a large reflecting telescope in Earth orbit, with a tentative goal of making it happen by the end of 1979. They recognized that such an installation would require frequent visits to maintain and upgrade it, and so the future space telescope’s design proceeded alongside NASA’s intended replacement for the mighty expendable rockets of the Apollo era: the reusable space plane that would come to be known as the Space Shuttle.

The parallel development of the Space Shuttle and the space telescope, which would eventually be named after American astronomer Edwin Hubble, resulted in design decisions that would pay enormous dividends down the road. Hubble was designed from the outset to be serviced, and specifically by space shuttle missions. As such, the spacecraft carrying the telescope and all the various scientific instruments that can be switched into its optical path has abundant handholds, tether points, and easily accessed hatches and compartments, all designed to make it easier for spacewalking astronauts to perform their tasks. Inside the spacecraft’s many equipment bays, instruments are attached with standardized fasteners designed to be manipulated with gloved hands. Hubble even came with a complete kit of specialized tools, to be used by future service missions.

All the best vehicles come with a nice toolkit. Hubble’s specialized service tools. Source: NASA

Ever mindful of the march of progress, Hubble’s designers knew that the instruments they could think up and build in the 1980s would pale in comparison to what would undoubtedly follow. To prevent built-in obsolescence, Hubble was specifically designed with not only repairs in mind, but upgrades to its scientific packages. Really the only part of Hubble that has been constant over its service life has been the Optical Telescope Assembly, consisting of the famously misground primary mirror, a secondary mirror, and supporting structures like the trusses that keep them in alignment and the baffles to control reflections.

The instrument bay behind the primary mirror was designed to accommodate five scientific instruments at a time. The first service mission to the Hubble, which went up on Endeavour in 1993 and made Story Musgrave and Kathyrn Thornton household names, included the COSTAR corrective optics package, which restored Hubble’s fine focus and allowed it to explore the faintest and furthest objects in the universe. A dozen other spectrographs and cameras have occupied the equipment bays over the years, some being left in place for just a few years; others have been permanent parts of Hubble, generating data since the very beginning.

NASA Standard Spacecraft Computer: Built to be Modded

The foresight Hubble’s designers showed in recognizing the need to swap out science packages on Hubble influenced their designs for other areas of the spacecraft as well. As early as the mid-1970s, NASA realized that building custom subsystems for every mission, which had worked so well for the manned space flights of Mercury, Gemini, and Apollo, would be a hindrance to developing and fielding a wide range of spacecraft to explore near space. To that end, the NASA Standard Spacecraft Computer-1 (NSSC-1) was developed, with the goal of standardizing the tricky business of keeping pretty much any spacecraft that carried it stable and in orbit, as well as to run whatever science the mission planners could dream up, and to do all of that with basically just software changes.

The Hubble SI C&DH. The NSSC-1s are called out as CPUs at the back of the tray, flanked by the stacks of primary and backup memory modules. Source: NASA

NSSC-1 flew on a number of missions before finding its way into the Hubble equipment bay. By modern standards, the computer seems primitive, with discrete TTL chips and a mere 64k of 18-bit core memory (plated-wire memory was also used, and appears to be what was originally flown on Hubble’s NSSC-1). But the design was robust, radiation-hardened, and redundant at the module level, and most importantly, it gave mission planners for the first time what had largely been absent from prior spacecraft controllers: in-flight programmability.

On Hubble, the NSSC-1 forms the core of a subassembly called the Scientific Instrument Command and Data Handling Unit, or SI C&DH. This is a largish tray upon which the NSSC-1 module and other modules, like power regulators, interfaces to the scientific instruments, and remote control interfaces are attached. Everything is redundant, and the entire assembly is built to support being swapped out by spacewalking astronauts. In fact, the final service call paid to Hubble in 2009 included the replacement of the SI C&DH with a new unit, one with the original core memory modules upgraded to CMOS. The video below shows the SI C&DH undergoing ground tests before heading upstairs.

Trouble Upstairs

The current issue with Hubble centers around the SI C&DH assembly, which the media have mercifully dubbed “the payload computer.” On June 13, the NSSC-1 in the unit went into safe mode, which was the widely reported “glitch”. Initial diagnostics led operators to suspect that one of the memory modules had degraded enough that the computer couldn’t continue operating. Exactly how a single degraded memory module is causing this issue is a bit of a mystery; the NSSC-1 only needs one memory module at a time to work, leaving the other three — as well as the four memory modules in the backup NSSC-1 — in reserve. But given the available data, it seemed like the most logical explanation.

The initial attempt at recovery from the error state was pretty much what we’d all try first: turning it off and back on again. The commands to do that were sent up on June 14, to no avail. The next step was to switch the NSSC-1 to one of the three backup memory modules, but that command was met with a failure as well. This and other diagnostics led NASA to believe that the problem might not lie in the NSSC-1 or its memory modules at all, but rather in other modules on the SI C&DH, specifically the Standard Interface (STINT) hardware, or the Central Processor Module (CPM).

Testing that hypothesis required a radical step: turning on the backup NSSC-1 for the first time in space. The backup had only ever been tested on Earth, and has been sitting dormant since it was installed twelve years ago. This step was taken on June 23 and 24, and the data indicated that the backup computer suffered from the exact same fault as the primary computer.

It’s unlikely in the extreme that both the primary computer and a backup computer that has been sitting idle for more than a decade would fail in exactly the same way at exactly the same time, so that test provided strong evidence that the fault lies somewhere else. Current thinking is that some other piece of hardware on the SI C&DH, something shared by both computers, is actually causing the fault, and that the memory error is but a symptom. NASA is looking at the Command Unit/Science Data Formatter (CU/SDF), which formats and sends commands to the various science instruments and data to the ground, as the likely culprit. The other suspect is the Power Control Unit, which conditions and regulates the various voltage rails needed around the SI C&DH.

Is This the End?

While it sounds like NASA has a good handle on what’s going on with Hubble, given everything the spacecraft has been through over the last 31 years, it’s hard to say that the odds of recovery are in its favor at this point. That’s a shame, because aside from the current issue, the spacecraft and instruments are in great shape. But there’s also the fact that since the retirement of the shuttle fleet in 2011, Hubble is without the vehicle it was designed to be serviced by, and to boost it to a higher orbit from time to time to fight atmospheric drag.

What’s even more ominous is one of the last modifications made to Hubble in the 2009 service call: affixing a device called a Soft Capture Mechanism to the aft end of the spacecraft. Originally intended to provide a means for returning Hubble to Earth in the payload bay of a shuttle, the SCM’s purpose is now to provide an attachment point for a future deorbiting mission. Left to its own devices, Hubble will deorbit naturally sometime between 2028 and 2040, taking a trajectory that may leave the more than 800 kg mirror at least partially intact. The SCM will allow NASA to control reentry more precisely, giving Hubble a quick and dignified end to a decades-long career of unmatched scientific accomplishment.

It seems to us that a hypothetical deorbit mission could just as easily be an orbital boost mission, but that presumes NASA will be able to clear the current bug and get Hubble back to work. Here’s hoping that’s exactly what happens, and that the SCM doesn’t have to be used one minute earlier than necessary.

A big thanks to [David Anders] for pulling together a bunch of excellent sources on the NSSC-1 and SI C&DH and posting it on Hackaday.io. It was invaluable in pulling this article together.

[Main image via Wikimedia Commons]

21 thoughts on “The Trouble With Hubble: Payload Computer Glitch Stops Science At The Space Observatory

    1. Hey so I wasn’t the only one to have this issue and report it :) Glad the comments are back. I have *also* forgotten what I was going to write. Must not have been world-shaking then :)

  1. Is the shuttle the only option? Can’t we visit it while it’s still in orbit and repair any issues and do some upgrades? Seems an awful shame to de-oribit it just as we’re getting private industry to a reliable place for getting items to orbit.

    1. If I understand correctly, there are plenty of available crewed spacecraft that can rendezvous with the Hubble, but that’s not the only requirement. To acutally service the Hubble, the craft would need to have an airlock to allow astronauts to spacewalk to the telescope to do the repairs, and I think most (or all) of the current crewed craft are designed for docking with the ISS, so they have a docking port but no airlock. Also, I think it was typical to attach the shuttle to the Hubble using the shuttle’s remote manipulator arm, and there are no current craft with similar capability.

      I remember Scott Manley discussing this in a recent video. It’s probably this one, but I may have picked the wrong one (this one’s certainly related anyway):
      https://www.youtube.com/watch?v=RWUnC2uf3XY

        1. Correct, but the second astronaut had to “suit up” and remain so until the other returned to the spacecraft. Then they re-pressurized the crew cabin and took their suits off. Doany of the available craft have this capability?

      1. And I was thinking a few launches to get a tug to it with the parts, and then the labor up there to fix it. They will miss all the useable fixing and leverage that the payload bay gave them, but they train for that stuff all the time.

  2. Is there no near term option for a man rated SpaceX or other rocket/capsule to have either a Gemini style open door EVA or an add-on airlock like the Apollo-Soyuz adapter tunnel?
    What is the continuing value of Hubble visible light telescope vs Webb IR vs combining many guidestar corrected terrestrial telescope visible light apertures in software?
    What do we loose sending a cheap SpaceX vs deorbiting or high parking a broken Hubble for a future historical monument.

    1. You could send up a satellite that “parasitically” attaches to Hubble, breaks/cuts what needs to be broken, so its sensor is where the Hubble sensors currently are, and uses Hubble’s mirrors for its own sensors. It would have to orient a bunch of dead mass, which is a bit wasteful, but other than that, I would expect it’d be *much* cheaper than manufacturing and sending up a new mirror.

        1. Time is the price.
          The JWST mirrors took years to make each. If the Ariane exploded at launch and we lost them, the mirrors would be what takes the longest to make again for a replacement, definitely.
          By re-using hubble’s mirror, you save a lot of money and time

  3. Send a probe/sat that can grab onto Hubble (with some sort of clamp), then use thrusters to change its orbit so it is a few meters from the ISS (where Canadarm can grab it). Then the astronauts from the ISS can service it there. Then that same robot gets refueled at the ISS by the astronauts, and pushes Hubble wherever it should orbit.

    The beauty of this plan is the technology is trivial, and most of what you need to send up is fuel (which is very cheap).

    Alternatively, you could send some sort of probe with robot arms to do the work an astronaut would normally do, but that’s *much* more work/complexity/risk than the orbit-changing plan.

Leave a Reply to EugeneCancel reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.