Soyuz Failure Leaves Questions Unanswered

The Russian space program experienced its first serious incident on a manned mission in 35 years when Soyuz MS-10 failed during ascent on October 11th, 2018. The abort system worked as designed, and crew members Aleksey Ovchinin and Nick Hague landed safely approximately 430 km from the launch site in Baikonur. Beyond being put through unusually high G forces, the two men suffered no injuries and will have their mission recycled for a future flight.

From an abort standpoint, the event went as well as could possibly be expected. The fact that the crew walked away unharmed is a testament to the emergency systems on the rocket and spacecraft, and serve as a reminder of why these functions are designed into manned rockets even if they are rarely (if ever) used. The success is especially impressive considering the Soyuz’s launch abort tower, the solid fuel rocket designed to pull the spacecraft away from the failing booster rocket, had already been jettisoned before the event occurred. The spacecraft was instead pulled to safety by the secondary abort thrusters, which were added to the vehicle’s design in 1975 as a contingency and until now had never been used in a real-life scenario.

What Went Wrong?

But while the safe return of the crew was naturally the first priority for all agencies involved, the questions soon turned to the Soyuz itself. What caused the loss of the rocket? Is it a defect which could be present in the other Soyuz rockets currently under construction? Perhaps most importantly, when could the Soyuz fly again? As it’s currently the only way to put humans into space, the International Space Station is completely dependent on regular Soyuz flights, and a delay in the program could endanger the orbiting outpost.

Now, with the initial findings of the Russian incident investigation being made public, we’re starting to get answers on some of those questions. The official report so far agrees with the conclusions many “Armchair Astronauts” made watching the live stream of the launch, and the evidence suggests that the core issue is the same which doomed previous Russian vehicles.

Soyuz failure as seen from the ground.

The MS-10 Failure

To anyone who’s watched Soyuz (or more accurately, the R-7 family of rockets which the Soyuz is derived from) launches in the past, it was immediately obvious that there was something wrong when watching the live stream of MS-10’s ascent. The four side mounted boosters are supposed to separate from the main stack and perform a synchronized back-flip maneuver, known as the “Korolev Cross” in honor of the R-7’s designer,  Sergey Korolev. But on the October 11th launch, instead of the careful aerial ballet that normally occurs, there was a cloud of debris behind the Soyuz rocket, and at least one of the side boosters was erratically tumbling through the air.

Immediately after the side boosters separated, violent shaking was observed inside the crew capsule, and the Soyuz automatically triggered the launch abort system and pulled the spacecraft from the booster rocket. It was clear to even the casual observer that the side boosters had failed to separate correctly and that doomed the rocket; but the question was why the booster separation, a maneuver the Soyuz has performed hundreds of times since the dawn of the space age, had failed this particular time.

As it turns out, the answer was easier to find than you might expect. Even though booster separation happens at an altitude of 50 kilometers (31 miles) and while the rocket is traveling in excess of 6,500 km/h (4,038 MPH), the Soyuz side boosters hit the ground largely intact. Investigators just needed to track down where the boosters came down in the Kazakhstan desert and examine them. Interestingly the boosters are not usually recovered after a nominal Soyuz launch, and it’s not unheard of for locals to cut them up and sell them for scrap.

Intact wreckage of a Soyuz side booster.

During a normal separation residual liquid oxygen from the propellant tanks is vented out the side of the booster’s nose, which pushes it away from the rocket’s center core and starts the backflip maneuver. This is a simple and reliable method to put distance between the side boosters and the ascending rocket without relying on secondary rocket engines like the Space Shuttle’s SRBs, and utilizes a sensor in the booster which opens a nozzle near the topmost attachment point. But according to the investigation’s findings, the sensor was improperly installed and failed to trigger the oxygen vent.

In addition, a recording of the failed booster separation from onboard the Soyuz was publicly released on YouTube (failure occurs at roughly the 1:24 mark). The official release of this video is considered unusual given the relatively secretive nature of the Russian space agency, and has been seen to some as a way to appease international partners in an era of increased commercial space competition.

Echoes of the Past

If an incorrectly installed sensor causing the failure of a Russian rocket sounds familiar, it’s because this isn’t the first time it’s happened. In 2013, a Proton-M rocket carrying three GLONASS satellites dramatically failed just seconds into its flight. The rocket started to pitch over shortly after it cleared the tower, and despite obvious attempts by the vehicle’s automatic systems to correct for the variation by vectoring engine thrust, the rocket flipped over and powered itself directly into the ground.

Proton-M rocket disintegrating before impact.

An investigation determined the cause of the failure to be the improper installation of multiple angular velocity sensors. Despite the correct orientation for the sensors being clearly indicated and the fact that installing them incorrectly required physically bending the plate they were attached to, both the primary and secondary sensors were installed upside down. With invalid data being fed into the flight control system, the automatic attempts to correct the vehicle’s trajectory ultimately caused it to invert itself instead.

The loss of another rocket due to human error during assembly, this time with human lives aboard, has called into question the viability of the Russian space industry. With dwindling budgets and increased competition, critics point to these failures as evidence that corners are being cut in quality control to reduce costs.

Soyuz Returns to Flight

With a clear cause identified for the MS-10 failure, investigators were able to disassemble the relevant sensors on the rockets currently under construction and verify the sensors had been installed correctly. While all of the other sensors checked were in working order, investigators have not yet revealed if they know the circumstances which lead to the improper installation of the sensor on MS-10.

In the meantime, another Soyuz rocket has already been successfully launched, and at least three more are slated to lift off before the next scheduled manned mission to the International Space Station on December 3rd. If all goes according to plan, the relief crew should arrive to the ISS before the current occupants need to depart, preserving humanity’s uninterrupted presence in space since the year 2000.

83 thoughts on “Soyuz Failure Leaves Questions Unanswered

    1. India has stated a 2022 goal for putting a human into space, and the Chinese space program does not have any crewed launches scheduled until 2020 (and haven’t flown one since 2016).

      If the Soyuz couldn’t return to flight for whatever reason, our best chance would be to fast track SpaceX’s Crew Dragon; there are no other manned space vehicles that are even close to operational status.

      1. I’m no great fan of Boeing or the Starliner, but let’s not ignore its existence. It’s currently scheduled to fly with a crew a couple months after Crew Dragon, which is a small enough gap that a small schedule hiccup could see Boeing’s manned launch come first.

        1. We can ignore Boeing. They’d probably charge more than Russia. And even they don’t know what they want with SLS. I just got word they want to add parts to their contract without paying us more for testing. But they balk when we can’t meet their incredibly ridiculous deadlines and their rushing has already resulted in overlooking many key factors that their old resigned team has forgot to mention. Remember that IG report? I experience their mismanagement 5 days a week. 6-7 days a week during the holidays every year.

          But yes, MAYBE Starliner will fly on schedule. It’s different than everything else.

          1. Oh, it’s already well behind schedule and almost double the cost of Dragon, but NASA will make sure that pig flies at least a couple times. I just figure that, with 10 months left in the planned schedule, they’re probably not going to add more than about a year of further delays. That puts them relatively close to operational status, as these things go.

      2. What a goddamn shame. Here we are, stuck on a rock and we’ve let every single vehicle that can move a person off break down and rust away. We gotta watch our little backyard outpost–already a monument to sad, compromised ambitions–gather cobwebs because we literally can’t reach it even at such a small distance. In the year 2018!

        This is pathetic. We’ve gone right up our own asses with fear and self-absorbed, obsessive information tech and completely ignored the real, bright, physical future we wanted half a century ago. We’re still using those same Soyuz for christ’s sake! Aside from internet and phones and shit like that, our technology hasn’t moved much at all. We’ve got those self-landing space-X rockets which are quite impressive, but they’re still chemical-fueled stacks optimized for launching a small unmanned package into a tiny orbit skimming our own atmosphere. We had such audacious plans.

        We have done so poorly. Now we gotta watch helplessly as our last, laughable foothold in space expires because as an entire species working together and doing our earnest best we can’t maintain one single launch system. Any launch system. What a hateful shame.

        1. ” we’ve let every single vehicle that can move a person off break down and rust away.”

          Well.. the only one we ever had was the Apollo stack and it was all one-use only. Surely you weren’t talking about the STS were you? That would have only ever taken you in circles around the rock with no option of not returning to it’s surface sooner rather than later. Keeping it going only served to use up any funds that might have been used to actually create a vehicle that goes somewhere again. There were some plans to extend the Gemini program into something that could reach the moon and Venus but that still involved building rockets that never actually existed.

          “We’ve gone right up our own asses with fear and self-absorbed, obsessive information tech”

          Yup! That pretty well summarizes it!

          “and completely ignored the real, bright, physical future we wanted half a century ago”

          Do you tear up watching “Tomorrowland” too?

        2. “we’ve let every single vehicle that can move a person off break down and rust away.” What are you talking about? “We”, and by that, I mean, the people of Earth, have a perfectly serviceable launch system, and it’s called Soyuz. The fact that one was built with a defect and failed to accomplish its mission does not diminish it. Unlike the only human-rated launch system built in the U.S. in the past several decades, it did not kill anybody when it failed, because it had an emergency abort system, and that system actually worked. We have several new launch systems on track to make great strides in reducing the cost per kg to orbit, to the point where space tourism may actually become a thing.
          Also, I would like to point out that today’s Soyuz isn’t the same as the ones built in the 1970s. Just like the Americans, the Russians from time to time make improvements to their systems.
          Yes, we’re still stuck with chemical rockets. What have YOU done to overturn Newton’s laws? Chemical rockets are the cheapest and most efficient way to get things into orbit, because THAT’S JUST THE WAY IT IS. We do have (and have had for decades) electrically-powered rocket motors, but these still need reaction mass to throw overboard, and they have their application, which is low thrust over very long periods, usually using solar power. I guess that’s nothing to you. They’re just not as good as chemical rockets when you have a short-period high thrust requirement, mainly due to, you know, physics.
          The future of space travel is good. Which is a good thing, since the future of life on Earth doesn’t look so rosy.

  1. Not claiming to know the answer, but … on a German forum, someone explained that the tool that was used to attach the sensors had two settings (or there were two tools?) and the engineering crew used the wrong setting, which “tried hard” to mount the sensor it was told to mount, even though it would not fit. After the sensor had been mounted, it was hard to tell from Q.A. whether the correct sensor had been mounted (or whether it was mounted the correct way).

    IF that is correct, it would make sense that even with Q.A. in place (which obviously hadn’t done a proper job anyway, but we all know that Q.A. has been considered useless nonsense by most companies worldwide for at least a decade – here’s looking at you, Microsoft …) if mounting a device incorrectly with possible disastrous outcome was possible, humans would find to do exactly that. Even if the machine they told to do it was moaning and screaming in the process.

          1. That’s exactly why ‘near miss’ accidents are required to be reported under British health and safety law. If a near miss is reported and someone subsequently gets hurt you can expect substantial pay outs.

    1. And now we know why designs incorporate things like asymmetry and keys, so things can go only one way. Plus bending a plate should be something visually caught.

      ” Even if the machine they told to do it was moaning and screaming in the process.”

      Impact wrench. Is it moaning and screaming by design, or something done incorrectly? Maybe proper installation should have a nice chime to it?

      1. Making things “Idiot proof” just means that an idiot who is more clever will be hired the next day to prove you wrong, It all comes down to the person doing the job, the right person will not need safeguards as they will do the job correctly, the person who thinks they know the best way to do the job will figure out how to defeat any safeguard in order to do things in their “better” way.

          1. LOL, ever see genetics explained in “Idiocracy”, Genetic engineering will give us more powerful athletes and better looking movie stars but that about as far is it will go. Having intelligence does not increase ones social media score so that will be left on the cutting room floor so to speak.

        1. I agree with what you are saying, there should be a level of competency, however, it is impractical to expect everyone all the time to assemble complex systems (eg, soyuz) together the correct way based on having a particular level of skill. If the sensor is assembled incorrectly, it will cause a complete rocket failure. Is this something that should be left up to installer skill? I think it should be an ‘idiot proof’ assembly method for vital components that can cause larger failures such as in this case, even if it is to catch out ‘good workers’ having a bad day.

        2. I think that the basic issue is still cost cuts. Costs are cut, but quality is attempted to be maintained, by decreasing the size of the team by firing the least efficient ones until you are left with the smartest, most efficient guy. You can pay him more, because he’s smart and efficient, and still costs less than two less mart and efficient guys.

          But what you don’t realise is that the most smart and efficient guy is also the guy who has enough creativity to solve all the problems that he encounters by himself.

          And so the smart and efficient guy will encounter the problem that the sensor doesn’t want to be mounted. His bosses told him that he’s the smartest guy. And instead of calling the designers in to find out what’s going wrong, he finds his own solution and gets the job done.

          But if there would have been 2 guys doing that job, they would have convened and most likely strengthen each other’s suspicions, and conclude that something is really wrong and they need to get the designers in.

          Never let a creative problem solver work alone. Always pair at least two of them up.

        3. Maybe with stuff like rockets this is true, but the process to cheaply get to 99 percent reliability is a lot different than the process you need to make something you can trust with your life.

          Stacking up safeguards for will often result in a system that works most of the time even with untrained users. Look at USB ports. Most users do just fine with those. Or gas pumps, or microwave ovens, or word processing software.

          With manned rockets any failure is unnaceptable, but with consumer gear some level of failure is almost guaranteed because of how little control you have over users, environments, and the tasks the devices are put to. Almost two different ideas of reliability.

    2. Software QA is mostly crap because most software (a) is only used for crap, not for life&death scenarios, and (b) even if it fails, you can push out a patch.

      Software in medical, military, nuclear, etc industries, and where it can’t be updated (e.g. embedded non-networked) is still (mostly) tested very thoroughly.
      I worked on software that couldn’t be updated, and for which failure would have been very costly. We tested the shit out of it. Our QA team were half developers, writing automated tests (completely independent to the unit and integration tests the dev team wrote). We shifted millions of units (you’ve all used software we wrote) and never had a bug found in the field.

    1. Yes, the escape system worked at least! That was the second time a launch escape system was used (The other was during a 1983 Soyuz launch attempt).

      Nice to have when its needed.

  2. Yet another blatant and obvious piece of evidence that NASA’s management really screwed the pooch when they shut down their manned program without coming up with a working replacement first, something they should have been working on doing since 1986!

    1. Maybe if there was Space Oil, we’d invade. But so far the main ones are for science, spying, or communications. And people in space don’t really help the latter two. The first, to a degree, but would a human have done better than the rovers we sent to Mars?

      1. Not directly. 20 was cancelled so the booster could be used to launch Skylab. 18 and 19 were cancelled to direct the funds to STS to make up for budget cuts. Nixon wanted to cancel 16 and 17 directly (as opposed to budget cuts) but was talked out of it.

    2. I’ll concede that the Shuttle did need to get retired: it was monstrously expensive, dangerous, and with the ISS effectively complete didn’t really have a clear use case anymore.

      But it’s definitely a travesty that we didn’t have either commercial crew or some variation of the Ares I working to at least retain US crew launch capability.

      1. There’s currently no need for Helium 3, if and when a need develops (i.e. nuclear fusion becomes a useful power source) I’m sure the Lunar Helium mines will spring into action.

    3. It’s not NASA’s fault. It’s the voter’s fault. They voted for Republicans too often, and Republicans have a long record of depleting budgets to wage wars. How much did the Bush families Iraq war cost? 2.4 trillion dollars, for 7 years of war.

  3. “Immediately after the side boosters separated, violent shaking was observed inside the crew capsule, ”

    OMG! So you mean that cheesy shaking effect that always happens on the bridges of starships in low (and sometimes not so low) budget sci-fi shows actually might be accurate?!?!

    Were there sparks flying from the consoles too? Did any cosmonauts wear red shirts that day?

    1. There’s video from inside the capsule. You can see when the side booster strikes the core, then the engines cut off. Their arms, heads and anything not fully secured flies forward. Then the escape rockets light and everything slams back down.

    1. Actually, there is no such system on Russian rockets because of the remote location of the launch sites. The flight controller is able to shut down the engines remotely, but cannot destroy the rocket as is used in American systems. Incidentally, this is explained in the Wiki link you posted.

      What we’re seeing in that picture is the rocket simply coming apart due to the spin and aerodynamic forces.

      1. This is what they call “mechanically controlled range safety flight termination system”. The rocket just breaks apart if something goes wrong. And sometimes when everything is fine.

  4. “Idiot proof” is rather an oxymoron. I have used Idiot-Resistant for a v long time.
    And, Thsak God the rescue system worked? Mind you, it was a never before used, backup. The primary was gettisoned. Maybe they will add some fuel to allow it to be kept for a few more minutes. But with the money we save w/o the Shuttle, surely they could send 7 guys to watch and co-inspect assembly over yonder, eh? What I’d like to know is, how many cars’ equivalent ckeanish-air would be taken off the road for just 1 launch, gone unlaunched.

    1. A quick word about the launch abort rockets on Soyuz.
      The original Soyuz just had a launch abort motor in a tower at the top (like Apollo etc.). In an emergency this would pull the top of the rocket away from the rest of the spacecraft, before the descent module (containing the cosmonauts) separated and landed on it’s parachutes.
      Of course, on the 99% of launches where it’s not needed, carrying the launch abort tower is wasted mass. Consequently, the designers would like to get rid of it as soon as possible. So, beginning with Soyuz 19, there were additional, smaller, engines added directly to the shroud (faring), which are strong enough to provide an escape mechanism once the rocket has jettisoned it’s side boosters. This way the escape tower can be jettisoned at the same time as the boosters, thus leaving less mass for the core booster to push into orbit, and leaving slightly more room for payload. Then if there is an emergency at this point (as there was on MS-10) the fairing boosters will separate the spacecraft from the rocket.
      Of course, after less than a minute the faring is also jettisoned, and any launch abort is done using the engine in the service module of the Soyuz itself.

      So it’s not really fair to call the engines on the faring “backup”. In every flight they are the primary launch escape system between the launch escape tower being jettisoned (around +115s) and the faring jettison (~160s). This is just the first time they’ve been used as designed. And it’s worth noting that they worked perfectly.

  5. Comments about incorrectly mounted sensor are strange, at least. As I’ve understood official report, the root cause is mechanical bending of the mechanical sensor (kinda end stop switch with pusher rod) because of handling error while attaching failed booster to the core stage. May be rude handling, may be crane operator error, may be protection cap removed in wrong sequence.
    Rate gyro sensor misplacement by incompetent worker, it was much early. It was root cause of epic Proton mishap. This misplacement (wrong orientation) was not detected by electrical test protocols, and missile tipped over just after launch.

  6. What a waste of money. Just so humans can crap in a vaccuum hose toilet. Seems like all their efforts are used to keep themselves alive in space. Nothing gained but pretty pictures of earth.

      1. Just a couple of comments Mr Flanders. 1) You can’t spell. 2) It’s a lavatory. People wash in a bathroom, they shit in a lavatory. 3) How do you think the computers that were used to design a Dyson were developed? Or the integrated circuit technology used in the computers? Or the plastics molding methods used in the cyclones? Or the tool controllers used to make the moulds used to make the plastic parts used to make the …… I would not want to be insulting directly to you Mr Flanders but, if you don’t like technology, please go and live up a tree somewhere. I bet you have a car though, and a phone and a computer and proper clothes and central heating and all the other trappings of civilisation.

        Toodle pip

        1. Just a couple of comments, Dave Graham:
          1) “lavatory” literally means “washing room”, so it’s really no more correct than “washroom” or “bathroom”. And BTW, using a high-falutin’ word like “lavatory” should preclude use of the vulgar “shit” in the same sentence, so even though still incorrect, the statement should have been, “people defecate in a lavatory”, for stylistic consistency. You could also say, “people shit in a shitter”, but that would be kind of self-evident.
          2) Computer development was NOT advanced by the space program. Pretty much all off the technology used in the on-board computers were dead-ends, and all of the ground-based computers involved were commercial off-the-shelf systems. The inception of computing was based in WWII, on two fronts: code breaking by the Poles and Brits, and nuclear simulation systems by the Americans. Further development was almost exclusively for business and other down-to-earth practical applications.

        2. And a couple more:
          3) Pretty much everything in the space program was low-volume, made on traditional machines. This had no effect on mass-production methods such as NC or CNC machining, or even plastics molding. ALL of the things you mention, cars, phones, computers, “proper clothes”, and central heating, predate the American space program, so your rants make no sense.
          4) Mike Flanders gives no indication that he doesn’t like technology. And even if he did, the argument that the space program provided direct benefits to everyday life is a myth made up by people trying to justify the program the wrong way, for people who don’t understand the long-term benefit provided by basic research. Saying that space research provided all of the trappings of civilisation is like saying that art is a great way of covering holes in your walls, and music exists to drown out the sex noises of your neighbors.
          5) Mike Flanders didn’t even USE the word “bathroom”. He said “toilet”. And while “toilet” is sometimes used to describe the room containing a shit receptacle, it usually refers to the receptacle itself.

      1. A person would have to be exceedingly incompetent to ignore the orientation markings then *deliberately damage* the mountings in order to forcibly install the sensors upside down. I very much doubt the Russians were press-ganging rocket assembly workers from the closest bars.

        I’d expect a person smart enough to operate a screwdriver to tell someone “This doesn’t fit!” followed by that person smacking the idiot in the back of the head “Idiot! You have it upside down! You’re fired!”.

        1. Gregg: You don’t know that. Just how much incompetence did this take? Is there anything in the information about this incident that indicates just how much effort had to be expended to install the sensor wrong? It’s quite possible that it took very little more torque on the wrench to bend the pin than it would take for proper installation. And which way is “up”, when you’re presented with an irregularly shaped object on a horizontal assembly line?
          If a device is designed so that it CAN be installed upside-down, it WILL be so installed. The fact that this error was made and not caught means there was a design problem.

  7. I have to wonder if the software at startup could check for sanity readings from sensors. Gravity pointing the wrong way would be a pretty clear indication to throw an error and not allow launch.

    The Russians transport their rockets horizontally on a railroad before standing them vertical for launch, too. This provides a perfect opportunity to check from another orientation, which should then find any incorrect installation. They’d have to add a “attach jumper cables and run POST 1 routine” step before standing the thing upright, that’s all.

    1. So you are suggesting that the solution to an assembly problem involve adding complexity to an already complex machine? Adding complexity seldom results in higher reliability. Furthermore, what makes you think that this error could have been seen from the outside, just by having the rocket in a different orientation? Almost certainly, the sensor involved wasn’t even visible from the outside of the rocket.
      And unless the sensor just happens to be an accelerometer, just how is the software supposed to determine that gravity is pointing the wrong way?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.