Hacking when it Counts: Much Space Station Hacking Saved Skylab

Thanks to the seminal work of Howard and Hanks et al, the world is intimately familiar with the story behind perhaps the most epic hack of all time, the saving of the crippled Apollo 13 mission. But Apollo 13 is far from the only story of heroic space hacks. From the repairs to fix the blinded Hubble Space Telescope to the dodgy cooling system and other fixes on the International Space Station, both manned and unmanned spaceflight can be looked at as a series of hacks and repairs.

Long before the ISS, though, America’s first manned space station, Skylab, very nearly never came to fruition. Damaged during launch and crippled both electrically and thermally, the entire program was almost scrapped before the first crew ever arrived. This is the story of how Skylab came to be, how a team came together to fix a series of problems, and how Skylab went on to success despite having the deck stacked against her from the start.

Hacking a Space Station

Skylab_illustration
America’s first space station

When NASA was designing its manned space program in the 1960s, it knew it was going to need an encore should they succeed in meeting President Kennedy’s goal of landing a man on the moon by 1970. So even while Gemini and Apollo were still rehearsing the moves needed to get to the moon, the designs that would eventually fly to space as America’s first manned space station were being finalized.

Ideas for using the spent upper stages from launch vehicles as the basis of a space station dated back to at least the 1940s, when Wernher von Braun speculated about what would be needed to send a man to the moon during his Peenemunde days. Various iterations of the design floated about during the 60s, including a design which would have the entire living and working areas of the space station inside the active liquid hydrogen fuel tank of an upper stage rocket; after arrival in orbit, the excess hydrogen would be purged and the tank pressurized with an oxygen-nitrogen atmosphere.

The “wet workshop” design eventually lost out to the “dry workshop” idea, and fully fitted-out labs and habitats were built within an S-IVB stage, which normally served as the third stage of a full Saturn V stack. Christened Skylab, the space station provided laboratory and living space for three astronauts for missions of between one and three months. Equipped with a solar observatory and a docking adapter for the Apollo command-service module (CSM) on one end, Skylab’s missions were to study the Earth and the Sun, and to conduct space medicine and other microgravity experiments with the full laboratory inside.

A Mission in Trouble

On May 14, 1973, Skylab was launched as an unmanned mission. Mounted where the third stage of a Saturn V would normally go in the stack, and nestled in a payload shroud, Skylab was in trouble right from the start. Other than a lateral acceleration that was detected early on, Skylab’s 10 minute trip to orbit was uneventful. Once in orbit, the station began getting ready to accept its first crew, scheduled to lift off  shortly after their home was aloft and operational. Skylab’s observatory and solar-panel windmill deployed correctly, but mission controllers soon began to see anomalies. Telemetry told them that the micrometeoroid shield hadn’t deployed, and that temperatures within the workshop were nearing dangerous levels. What’s more, Skylab’s power production was seriously below specifications.

As controllers pored through telemetry, a picture of  what was going on 460 km overhead began to emerge. The lateral acceleration during launch was caused by a structural failure of the micrometeoroid shield, intended to protect the crew against penetration by space debris and to shade the workshop on the side that was always kept facing the sun. While it was tearing free, the shield ripped one of the two solar arrays mounted on the side of the workshop clean off, while simultaneously jamming the remaining array with debris that prevented it from deploying fully.

Skylab, intended to be a spectacular follow-up to the success of the Apollo program, was now in orbit but totally unusable.

The first step in saving Skylab was maneuvering the spacecraft to control the heat build up inside. Everything the first crew would need for a month in space – food, fuel, instruments, film – was slowly roasting at temperatures up to 77°C. Without a heat shield, the only option was to orient Skylab so that the smallest possible surface area was directly exposed to the sun. While that kept the temperatures reasonable, it burned precious attitude-control fuel, plus it kept the remaining solar arrays in the dark for much of the time, resulting in power problems. Still, the maneuvers successfully balanced power production and temperature control and bought enough time to start working the problem.

Eleven Days in May

It became clear that unless the first crew could rig a heat shield to replace the missing one, the entire $2.2 billion Skylab mission would end before it began. Secondary to that would be solving the problem of the partially deployed solar array, so that Skylab would have enough power to run all its scheduled scientific programs. Engineers and the flight crew, now on a launch delay and wearing surgical masks since they were still in quarantine, began working on both problems in parallel, in a marathon of brainstorming and design-build sessions that came to be known as “The Eleven Days in May.”

p72
Don’t forget to pack your umbrella.

The heat shield problem seemed insurmountable without a spacewalk, but NASA was loathe to allow EVAs without proper ground rehearsal. Luckily, Skylab had been designed with a small airlock in the workshop to allow experiments in the vacuum of space. Luckier still, the airlock was located on the sunny side of the ship, and so was a natural port to exploit for a sun shade. Several design were batted about, but the winning design was basically a Mylar parasol with a collapsible shock-tube frame that would be deployed through the small airlock. The design was perfected with tests in the Skylab neutral buoyancy simulator and the final product shipped to the Cape for launch.

Solving the power problem would prove to be a little more on the hackish side. Telemetry indicated that the surviving array was jammed with debris, and a plan was hatched to conduct a “stand-up EVA” through the CSM hatch to clear the blockage. But tools would be needed, and nothing in NASA’s tool crib fit the bill. Looking for inspiration, engineers from Marshall Spaceflight Center raided a local hardware store and found a pole-mounted tree pruner. A flurry of calls to local manufacturers resulted in selecting a cable cutter and a prying tool from a company manufacturing tools for, ironically enough, the power industry. The tools were quickly modified, mounted to a collapsible 3 m pole, and shipped to the Cape.

A Bit of a Fixer-Upper

Skylab_2_crew
The Fix-It Guys: Kerwin, Conrad, and Weitz

The crew of Skylab 2, Pete Conrad, Joseph Kerwin, and Paul Weitz, finally arrived at their home for the next month on May 25, 1973. They found the place pretty much as advertised, and after conducting a reconnaissance fly-around, they popped the hatch and got to work on the jammed array. Despite their efforts and the power amplifying effect of a steady stream of NSFW language over an open channel, the array remained stuck fast, so the crew gave up and docked with their new home.

The parasol was deployed through the airlock and the temperature quickly dropped. A later full EVA would get the stuck solar array deployed, and with this addition of 7 kW to the power budget paired with a now-reasonable temperature inside, Skylab was finally up and running.

1024px-Skylab_(SL-4)
Parting shot as Skylab 4 leaves

Skylab eventually flew three crews of three men for a total of 171 mission days. Skylab 4, at 84 days in space, held the record for longest duration in space for years. While it was never intended to be a permanent station, the plan to have the new Space Shuttle visit the station and boost its orbit never came about, and after almost 35,000 orbits, Skylab finally returned to Earth over the Australian outback in July of 1979. It was a sad and ignoble end to a machine that had beaten the odds with the help of an umbrella and some hardware store items, and the dedication of a team that wouldn’t quit hacking until the problem was solved.

 

 

Know of any more examples of hacking when it counts? Send us a tip for use in a future Hacking When it Counts article.

[All images courtesy of NASA]

46 thoughts on “Hacking when it Counts: Much Space Station Hacking Saved Skylab

      1. Still, it was a working space station for 23 days.

        The Almaz/Salyut program had multiple gems, such as firing an Anti-Aircraft gun in space to test whether you could use on to shoot down satellites, and one of them had a crew-constructed sauna on-board.

      2. The second crew. The first aborted due to docking failure, and aborted.
        The second crew spent nearly as long as the Americans would, the longest at that time.
        They died due to an engineering fault; and that fault is what killed the Salyut-1; it may have been a sauna, it may have blurred the lines between civil and military uses, and it may have set itself on fire at least once.
        The program ended when the Soyuz capsule returning to earth blew a valve on descent, suffocating the passengers. All flights were grounded, and without re supply the Salyut was doomed to run out of maneuvering fuel, being intentially de-orbited instead.
        Ironically it is the more modern Soyuz that prevents the ISS from suffering a similar fate towards the end of the Shuttle program.

        The Salyut was first. It had more than one crew. It was the first successful space station. That was it’s purpose, and that was fulfilled. The article is wrong, and unfortunately offensive in ignoring three astronauts who died on reentry.

        1. I think I remember reading about this in “Leaving Earth” by Robert Zimmerman. Excellent book.

          Is this the one where the cosmonauts did the first hydroponics experiments in space? I forget.

    1. Well you know how it is with americans; they think they did everything first. I bet most(those who know a little about space programs done by NASA) americans still think that Sally Ride was the first woman who went to space. In reality, Valentina Tereshkova was the world’s first and Sally was the first _american_ woman in space.

      1. I think most of us Americans with anything more than a passing interest in space exploration and space exploration history are extremely aware of the many accomplishments of the Soviet/Russian space program, and the importance of the many international partners in all phases of the space station.

        Among my earliest memories of the space programs include seeing Svetlana Savitskaya’s space suit at the Kansas Cosmosphere and Space Center in Hutchinson, Kansas, USA. It was one of many artifacts of the Soviet space program that were on display, including a Sputnik and other hardware.

        From the mid-1950s through mid-1969, most of the “firsts” for humanity in space were happening under the red Soviet flag. Failing to study the history of the Soviet space program would be ignoring much of what drove spaceflight for decades.

  1. Storey Musgrave loves to talk about fixing Hubble. Apparently he wanted to take a small ratcheting strap along in case the doors wouldn’t close. The engineers kept saying ‘no, the doors will close’. Storey took the weight penalty and put the strap in his personal kit. Anyone care to guess what happened?

    1. A good write up on Hubble (including some public domain video, maybe) would be great.

      I also recall there was an extension pole / torque bar used that was somewhat off-cards?

  2. What nothing about the last hack. Before the last mission to the station. They had to software hack skylab. The batteries on skylab had fallen below safe level for startup. Every time they issued a remote startup command the computer would shut it down. (Low-battery startup test fail). The last mission was not going to happen until ground support realized more power went into the batteries than was used by the startup sequence. Hence the hack became 1000’s of startup commands being send until the battery charged enough to continue.

    I remember this hack every time a fluorescent light won’t turn on. Flicking the switch until it does.

  3. I was a design engineer on the Skylab program and supported it from the early days into mission support. As an engineer we always look for root cause and I was there and witnessed the root cause of the problem encountered during the launch. The lesson to be learned was most likely not looked at by the powers that were in charge but I watched it unfold and it did not dawn on me until I watched the movie apollo 13 and then I recalled the event and thought oh my gosh.

    1. Thank you and your colleagues for getting that space station up and running. As a kid in the 70’s, I was inspired by Skylab (Apollo was over before I was aware of it), and held out hope that NASA would rescue it at the last minute. They did not, but what I read about Skylab was part of the reason I became an engineer. (I’ve been an engineer for 28 years.)

    2. This article doesn’t cover the additional hacks on the 2nd and 3rd missions. The small parasol was replaced on the 2nd mission with the larger cover seen in the parting shot picture.

      AFAIK, Skylab was designed for only three or four crews. There was no way to empty the tank where they put their garbage. That’s why NASA never got serious about saving it, or using the 2nd Skylab that ended up literally hacked up as a museum exhibit.

      There are two Skylab simulators languishing in disrepair at other museums. Some people started to restore one but the museum ended the project and pushed it outside under a tarp. The tarp went to pieces and the museum refuses to allow it it to be covered up. For some reason they want that piece of history to rot away.

      I wonder if Paul Allen would buy it and pay for restoration and find a museum that would appreciate it? He’s restored a ME-262 and other things so he appreciates historical artifacts. http://www.airliners.net/aviation-forums/military/read.main/168459/

      1. Correct. I saw the one living at the Smithsonian’s museum. The only reason why that happened is that the amazing thing is too big to be in the room. And I recall watching the launch, and was shocked by the bird arriving like that. To Don Spade, my John Deere hat is off to you for working on the design and supporting her.

        To the Backwoods man, the Space program is the primary reason why I’m deeply involved in electronics and computers, and of course space travel.

    3. I still marvel at the size of Skylabs interior. I showed photos of it to a collegue originally from Moscow. They had never seen the interior photos of Skylab. Thanks for helping create a truly wonderful experience, the look of marvel on his face said it all about the Cold War space race propaganda exchanges.

  4. “…how Skylab went on to success despite having the deck stacked against her from the start.”

    HaD, I DEMAND you put an end to you Misandrous (Misandric?) bias! Your micro-aggression is triggering me! Stop it!

    1. Aren’t most vehicles vessels relegated to female genders in English?

      Ships are female, even if they pack a nuclear reactor or two and enough firepower to overwhelm a middle-eastern state
      Cars are female, even if they have cast-plastic-chromed bollocks swinging from the tow bar
      Hurricanes are female, even if they ruin the east coast of america.

      You are mistaking common English usage for a statement about gender bias.

      Let me give you an actual “misandrist” statement;
      “…how Skylab went on to success despite having the deck stacked against her from the start by men

      1. Well, I’ve been waiting for the comment you’ve replied to to show its ugly face here. Thank you for rubbing it in, special snowflakes “triggered” by words don’t belong here, they forgot the “Sticks and stones” poem.

        As for the naming vehicles, i must admit, my car is female “Elwheeze,” although the last car was male, “devil” he used to get me into trouble, or cause it ;)

      2. Hurricanes are now given male and female names. Used to be male only, until someone decided it was sexist and women wanted their names associated with millions of pounds of property damage and lost lives… Never worked out why they wanted that, I’d feel a bit down if a hurricane with my named killed loads of people, even if it’s nothing to do with me.

  5. The reason that there was a problem during launch was my company pointed out that the vehicle ha not been shake tested with the shroud in place which was the launch configuration. The NASA person in charge of testing ranted that youall contractors always trying to drive up the costs we ain’t running any more tests. So for the cost of a test the mission was nearly lost. Totally forgot the conversation until I saw apollo 13 the movie.

    1. Test, test and re-test. It’s the only way to find the flaws.
      Apollo 13 had an explosion in the oxygen tank. After an ground ordered cryo-stir. What was not known at the time a heater in the tank is activated during the stir. Was designed for a lower voltage, NASA changed the design and prob did not test the system. So when the 68Volts was applied the wiring shorted out.
      Hubble mirror when being ground was not properly tested. So ended up being flatter than desired. Had a final test using laser’s been done as requested (and denied). The problem would have been found.
      Challenger exploded based on a rush to put a teacher in space. Solid rocket boosters problems were known but not fixed. Every launch was a 1/200 shot of 100% failure. So many critical one defects existed to end of project.
      The wings if damaged is a critical one part. We all know how that one worked out.

      NASA has it’s shinning moments, but a dark pass. Look at VW, they cheated the test. And how is that working out for them now.

      1. It wasn’t just the voltage increase, which wasn’t communicated to the supplier of the O2 tank and its components. That tank was built in 1965 and had at some point been dropped, which kinked the overpressure blowoff pipe.

        Dropping it should have gotten it immediately scrapped, or at least sent back for a complete teardown and inspection. Instead someone signed off on it and shipped it out for service module assembly. Had the tank manufacturer been properly informed of the voltage increase (or were they told and just assumed their wiring could handle it?) it would have been taken apart for the change and perhaps someone would have noticed the damage again (it’s known it was damaged from a paper where it was recorded) and perhaps some bright(er) employee would have insisted it not be used until it was repaired and recertified.

        Poor communications and bad safety procedures put a damaged part with inadequate electrical components into the Apollo 13 service module and nearly killed three people. After the Apollo 1 fire and the discovery of so many problems with the Command Module, there should have been a complete inspection of ALL the plans and components that had been drawn and manufactured.

        I assume it never occurred to anyone that if the CM had wiring problems that the SM might too? It would have been a huge effort to trace back every change made to every electrical component, especially changes made in volts and amperage, to discover if there might be anything else, anywhere in the rocket stack that could cause a fire.

        Did NASA do that after Apollo 13?

        I bet SpaceX is still doing that, inspecting *every part* of their rocket design after discovering they had some payload support struts that didn’t meet specifications, which caused the loss of a rocket and payload. If they’re ever going to put people atop their rockets they cannot have any failures. If they’re giving any part a pass they are flirting with disaster. We’ve seen NASA do it at least four times which killed a total of 17 people.

        With spaceflight, especially manned spaceflight, failure is NOT an option and any “Eh’ it’ll do.” attitude has the potential to kill people.

        So, for the next close to real space movie after “The Martian”, “Skylab” or one that brings to the big screen the daring plan that was developed to rescue the Columbia crew? http://arstechnica.com/science/2014/02/the-audacious-rescue-plan-that-might-have-saved-space-shuttle-columbia/

        I wonder how long it would have taken to grab the MMU from the Udvar-Hazy Center and refurbish it? With that available, the rescue Shuttle wouldn’t have had to get so close to Columbia. It would have also made it possibly possible to take up a replacement RCC leading edge panel or other repair stuff and apply it. The Shuttles *could* be remotely piloted but that required a special cable to connect a couple of places in the cockpit. That cable was NOT included in the standard flight equipment.

        1. Rescue the Columbia crew. 2. Patch up the wing. 3. Install remote operation cable. 4. Land rescue shuttle. 5. Try landing Columbia. Odds of success go way up with replacement of damaged RCC vs patch job.

      2. It was known that the heater was activated. The problem was the the tank was damaged during testing. When it was tested, it had too high of a voltage applied. IIRC this made the thermostat contacts stick closed, and the heater ran continuously. During that infamous test, it purged the tank, and kept on heating. The internal wiring and other components were overheated and hanging there with charred leftovers for insulation. It was then installed and flown. As soon as you powered it up in space, the wiring shorted out, locally heated up, and the rest is history given the thing was full of LOX. The voltages were correct both in design and on the spacecraft, but ground testing was done incorrect and damaged the test article without anyone noticing.

  6. I was a principal investigator on the SkyLab cosmic ray experiment flown by the PB Price group. Package arrived back from space circa November 1975. Great research effort. Increased the total cosmic-ray events some 100-fold from prior data.

  7. When I was a kid I read a great book written in the 1970s, “A House In Space”, about Skylab. One of the technical issues was the water tanks (which weren’t refillable, another reason the project had a limited lifespan”.

    Obviously gravity isn’t going to push the water down the pipes. So they pressurised the tanks with air. Of course, in 0G, air weighs the same as water. So what they got, was tanks full of increasingly aerated, bubbly water. One of the astronauts said something like “I think farting 500 times a day is a hell of a way to go”.

    It’s a great book. The third crew actually went on strike! First time ever in space!

    The first 2 crews were all high-performing military guys. They got bored, and requested stuff to do, so ground control added more experiments and increased their workloads. The third crew came from a civilian, more science-based background. They were presented with the high workload as soon as they came on board. This, and the revelation NASA had the station bugged, and downloaded each day’s worth of recordings through a fast-forwarded audio tape radio link, contributed to a feeling of isolation and being under siege.

    Of course that itself was a very valuable lesson for future missions.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s