Voyager 1 In Trouble As Engineers Scramble To Debug Issue With Flight Data System

Recently the team at JPL responsible for communication with the Voyager 1 spacecraft noticed an issue with the data it was returning from the Flight Data System (FDS). Although normally the FDS is supposed to communicate with the other subsystems via the telecommunications unit (TMU), this process seems to have broken down, resulting in no payloads from the scientific instruments or engineering sensors being returned any more, just repeating binary patterns. So far the cause of this breakdown is unknown, and JPL engineers are working through potential causes and fixes.

This situation is not unlike a similar situation on Voyager 2 back in 2010 when the returned data showed a data pattern shift. Here resetting the memory of the FDS resolved the garbled data issue and the engineers could breathe a sigh of relief. This time the fix does not appear so straightforward, as a reset of the FDS on Voyager 1 did not resolve the issue with, forcing the team to consider other causes. What massively complicates the debugging is that each transmission to and from the spacecraft takes approximately 22.5 hours each way, making for an agonizing 45 hour wait to receive the outcome of a command.

We wish the JPL engineers involved all the luck in the world and keep our collective appendages crossed for Voyager 1.

45 thoughts on “Voyager 1 In Trouble As Engineers Scramble To Debug Issue With Flight Data System

  1. It will be longer than 45 hours. There is the 160 bits per second data rate, so there would be delay on how long it would take to receive a useful sequence of diagnostic commands or codes (total RAM is 69.63 KiB) and then for them to execute ( 80000 instructions per second), or recorded onto tape, and the results generated.

      1. It would take over 15 hours to transmit the entire contents of the 67 MB tape, and NASA captures the data when one of the antennas in its Deep Space Network is pointed at the spacecraft. That’s about 6 hours per day. So the total duration could potentially be whatever 45 plus 15 plus 55 is.

        I know NASA said 45 hours but NASA’s wrong.

  2. 45 hours is nothing ….. I had a telecoms pole in my back yard that needed to be removed and I was politely told by British Telecom that I would have to wait 28 days before talking to the engineering dept. 28 days later I phoned them up again and threatened to cut it down with my chainsaw, which prompted some positive action.

          1. No, you can just buy over. but to use one in a professional context you need to be certified in the UK if you want any business insurance cover, which you definitely do if your job involves a chainsaw.

        1. I like the idea that the Beasties _were required to have_ a license to ill, rather than they just had one.

          Makes me think they had to go to the Department of Illing, stand in line for an hour or so, fill out some forms, wait a couple weeks for approval…

  3. Latency. I used to work in Europe, servicing an IBM mainframe installation in Australia. We would have to type in scripts on the machine in Melbourne and wait 22 seconds for each character to appear on our terminal screen. Just like some modern overloaded web UIs. Crap. Four of us, tired, standing the terminal around at 3am in the morning. And someone misspells an instruction. Aaaaaaggggghhhhhh!

    1. That’s 1970s technology. It was (is) rock-solid.. And it could have been even better.
      The Voyagers weren’t really designed for this mission in first place.

      They’re derived from an earlier probe generation and “quickly” made to not miss the time frame for the grand tour (all planets in a line).

      If more time and money was spent, the Voyagers could have been true multi-generation probes that last a hundred years. Or maybe not – there is a saying that nothing lasts longer than a temporary solution. ;)

      I think the RTG material used was the main reason the Voyagers as to why do suffer so much now. The RTG type caused the power to drop earlier than theoretical possible, it’s not merely the fault of the radioactive material.
      (Still, it outlived it’s original mission just fine.)

      Things like the absent heating and undervoltage do quite stress the probes.
      If power was normal, the failures would be less often, I assume. Heating/a stable temperature would improve reliability, for sure.

      What’s also must be kept in mind, I think, is that the hardware is redundant. Multiple computers (discreet stuff) and circuits of same type must be powered simultaneously.
      So the Voyagers could be consuming even less power in reality, if that redundancy wasn’t there.

      Speaking of, maybe that’s a last straw, if power drops below the (very very) critical limit.
      Maybe Voyagers can be operated with one backup system less than now?
      Or merely with merely one?
      It’s not reasonable to do that, of course. Unless the probe runs out of power, anyway.

      1. > Heating/a stable temperature would improve reliability.

        There is this equation in chemistry called the Arrhenius equation, which when summarised in words would be that the “rate of reaction increases by a factor of about 2 or 3 for every 10°C (10K;18°F) rise in temperature”. Now if you flip it “every 10°C (10K;18°F) drop in temperature should lower the rate of reaction by a factor of about 2 or 3”. Now that can be interpenetrated as every 10°C (10K;18°F) drop in temperature should double the mean time between failure.

        You might think well that is a load of phooey, but it is used by silicon fabs before packaging to remove early failures in the standard bathtub curve (There are three periods to the bathtub curve: an infant mortality period with a decreasing failure rate followed by a normal life period with a low, relatively constant failure rate and concluding with a wear-out period that exhibits an increasing failure rate). By operating the parts at full voltage in high temperature ovens for 2 days to 2 weeks, depending on the operating temperature, they can weed out parts that would have failed in the six months to 3 year of operation. This operation does shorten the life of the parts.

        Anyhow, because of this, I would argue the opposite – that it is the cold temperature of space has helped make the probes function well for so long.

        1. “Anyhow, because of this, I would argue the opposite – that it is the cold temperature of space has helped make the probes function well for so long.”

          I’m neither an engineer nor a physician/chemist, but I think that the heating mechanism had a purpose to begin with.

          In electronics, a low temperature isn’t exactly the best per se, alao, from what I remember.
          There’s a range of operating temperature in which the components work as espected.

          For example, in the past, a so-called “quarz oven” was being used to keep crystal oscillator at at stable frequency. A stable temperature by heating was easier to accomplish than by cooling.

          Another factor is the material stability, maybe.
          I don’t know how the temperature inside the Voyager varies, but if there’s an ongoing heatup/cooldown happening, the components are under stress. On a microscopic level, traces may break or bend.
          It’s a bit like bending a piece of wire back and forth until it breaks.

          Also, on a very low temperature, a material like metal becomes brittle.
          So it’s not exactly good.

          Another related phenomenon is hot water that turns to ice quickly, maybe.
          Due to the water molecules moving quicker and more freely, they can better organize if the water starts to freeze. So a higher temperature can be helpful, too.

          Last but not least, space isn’t exactly cold. It has almost no atmosphere (interstellar gases excluded), so energy can merely being dissipated in form of infrared radiation/thermal energy.

          That in turn perhaps means that the Voyagers can’t be cooled properly, either. If there’s power, though, an electrically powered Peltier element could maybe assist at doing a thermal exchange.

          In same way, the heater can help to keep the inner temperature at a stable level, even if it’s above a lower, but unstable temperature.

          Again, I’m just a layman here. But I think that there are certain factors to consider. The instruments, for example, are meant to operate at a rated temperature. If it falls below, it might be out of its measuring range or malfunction in a similar fashion.

          Ok, another example. If we switch an electric lamp (say, LED bulb or incandescent bulb) on/off multiple times, it won’t last as long as if it was being left run all the time.
          Changes in its state will wear it out more than constant operation.

          The same is true for PCs and power supplies and fans in general.
          The on/off cycles are the worst thing that can happen to them.
          That’s why energy saving and efficiency-thinking are such a paradox thing. The end effect is worse than just leaving things in operation all the times.

          Spikes/peaks in a power-grid are the results of this, too.
          Millions of people in each city do switch off PCs in the evening and switch them on again in the morning.
          This causes unnecessary spikes in the power grid.

          Or, power-saving mechanisms in modern processors..
          The up/down clocking of the operating frequency of the cores make them age on a microscopic level.
          Changes in temperature may cause tiny breaks in the structures.

          That’s another reason as to why 70s was somewhat durable. Lower integration density. The structures were much bigger, so tiny damages didn’t have an effect on the functioning of the whole circuitry.

          1. Yep, quartz oscillator frequency varies by temperature, the oven contains the entire oscillator circuit and is self regulated to maintain a precise temperature (and thus precise temp). they’re heated because it’s much easier to use passive cooling to maintain the temp above ambient, than try and integrate active cooling. A good engineering solution to the problem. These were also mounted in ways to minimize vibration and shock, which can also cause frequency to drift.

          2. Actually, LEDs themselves can be switched bajillions of times (technical term :P). Their drive circuits may not, and switching quickly may be different from thermal cycles from switching slowly, but still. PWM and just wiring diodes up to alternating current are ways to make a LED switch far more times than manually turning it off when you’re not using it.

            Computers too; they’re much more in control over their own wear now. I *had* an old style chip, back when everyone overclocked by setting a high fixed voltage and seeing what speed you could get before it crashed under load. It wore chips out, using excess voltage and current that wasn’t necessary most of the time. And if you did it that way, you sacrificed everything to maximize all-core speed. After a certain amount of use you’d find you had to turn down the multiplier, cool it down a bit more, or add another fifty millivolts to keep stability. Now you can get most of the available performance just by upgrading the cooling and telling some software to figure out what it can get you per core and how to ramp up and down depending what you’re asking for and how many cores it needs for that.

        2. That being said, I’m grateful for your explanation here. It’s very fascinating. Thank you. 🙂👍

          It also remembers me of another thing. Reducing the operation voltage of an incandescent bulb just slightly can increase its lifetime noticeably.

          That’s why my father instinctively used to use a lower heating voltage for his electron tubes, I suppose.

          Using 6v to 4,5v rather than the official 6,3v does barely affect the characteristical curve of the tube, but it increases lifetime of it dramatically.

        3. Voltage causes electromigration in silicon, reducing its semiconductor characteristics. Increasing temperature is similar to adding voltage in terms of stress on the chemical bonds. That’s why hot high voltage testing weeds out silicon failures an order of magnitude earlier. So running at slow clocks for a longer time is better for the system. Lower clocks allow lower voltage, which saves V2 of power.
          Thermal cycling is the worst for crystals and layers of material, causing shearing stress and therefore cracks.
          Computers in general run better at lower temperature, lower resistance, saves power and allows operation at lower voltages.
          Overclocking CPUs drops temperature with liquid N2, raises voltage, clocks, and reduces life from years to hours mostly because of the massive power causing thermal transients.

          1. Current can wear things out by migration. If you have a high voltage at idle, then not that much current is going anywhere to wear things out. When the total load goes up but the voltage goes down, you may find your maximum clock is lower but the wear is less that way along with the heat production.

            Of course LLC was also just something that means you don’t have as much trouble with transient dips, because the higher initial voltage stored some charge that helps you thru the increased current demand when the load shoots up. But it fits neatly with the other stuff too.

            Voltage can still mean leakage or a sudden breakdown in a gate somewhere, and by increasing voltage you increase current. Temperature can worsen both of these, but compared to how it was when we overclocked by setting a fixed voltage and finding out whatever was the highest frequency we could run all the cores at simultaneously under full load at full heat soak, the more dynamic control can help quite a bit nowadays. Also (generic) silicon has *less* resistance at higher temperatures unlike typical metals; colder temperatures mean more voltage for the same current in anything where the dominant resistance curve is similar to a hunk of silicon.

            It used to be that some chips did and didn’t have a “cold bug”, but either way nobody’s setting things up for 24/7 use when they’re getting their LN2 from a dewar and pouring it by hand, and you can’t just chill the entire thing including the motherboard, at least not to LN2 temperatures. It’d be silly, but as easy as it is to get mini split heat pumps that can operate down to negative 25C or so (at least in heat mode) I wouldn’t be surprised if I saw a 24/7 subzero build using one eventually.

  4. I’m thinking that NASA better get this thing fixed. Remember V’ger? Almost destroyed the whole Federation. Now that was the Voyager 6, so maybe the baby Voyager 1 won’t be as bad, or maybe it is a test run of the V’gers before they go totally Tyrant on humanity. We cannot be too cautious.

    1. There’s actually a story behind it.
      JPL (?) was willing to build a real Voyager for Paramount (?), but the film team went with the dummy.
      Sounds wild, but that was the late 70s. 😁

        1. the plot, via Memory Alpha:

          > Voyager 6 emerged from the anomaly in what was believed to have been the far side of the galaxy, and fell into the gravitational field of a planet populated by living machines. These beings found Voyager 6 damaged by its travels, and the identifying plaque attached to the probe’s exterior had been burned, leaving only the letters “V”, “G”, “E”, and “R” legible; the inhabitants of the machine planet called the probe “V’ger”.

          > These entities found V’ger to be primitive, but of a kindred spirit. They discovered the probe’s simple, 20th century programming, “learn all that is learnable and return that knowledge to the creator,” and interpreted these instructions literally.

  5. No one speculating on what the problem is then?

    I’ll go first: the clocking of the incoming data from the instruments has gone squiffy. Could be a hardware fault. Unless there’s a redundant backup, game over.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.