After more than forty-six years all of us are likely to feel the wear of time, and Voyager 1 is no different. Following months of harrowing troubleshooting as the far-flung spacecraft stopped returning sensible data, NASA engineers now feel confident that they have tracked down the cause for the problem: a single defective memory chip. Why this particular chip failed is unknown, but possibilities range from wear and tear to an energetic particle hitting it and disrupting its operation.
We’ve covered the Voyager 1 troubleshooting saga so far, with the initial garbled responses attributed to a range of systems, but narrowed down to the Flight Data Subsystem (FDS), which prepares data for transmission by the telemetry modulation unit (TMU). Based on a recent ‘poke’ command that returned a memory dump engineers concluded that the approximately 3% of corrupted data fit with this one memory chip, opening the possibility of a workaround.
Recently NASA engineers have also been working on patching up the firmware in both Voyager spacecraft, against the background of the dwindling energy produced by the radioisotope generators that have kept both spacecraft powered and warm, even in the cold, dark depths of Deep Space far beyond the light of our Sun.
MT ram? :P
I wish the scientists all the wisdom and luck in finding a workaround! I guess that they crammed the ram full, when they designed it, so probably they will have to trade in some other functionality. Not an easy decision, I imagine. I don’t think there’s much that can be left out without rendering Voyager incapable.
I think the problem is the lack of a reserve probe here on earth.
If there had been a spare voyager to experiment with, several scenarios could be tested beforehand.
I mean, in theory there is/there are. Or there were. Voyager 2&1 weren’t the last of their kind back in the 1970s.
Back when the first Star Trek movie was being filmed in late 1970s, there was the proposal to get a real Voyager for filming. Minus the RTGs, maybe, of course. ;)
https://memory-alpha.fandom.com/wiki/Voyager_6
Would it not be cheaper to build a simulator?
Cheaper to clone a twin, especially if it doesn’t have to be space grade. You have semi acceptable parts, planning and assembly instructions and something to experiment on.
I think this was covered here in a previous article about Voyager, but there’s no practical way to simulate everything that’s happened to the real Voyager – think radiation, heat/cooling cycles, undetected micrometeroid strikes and so on.
Both the Voyagers have had a number of onboard failures as well which they just work around. They’re effectively both extremely bespoke objects at this point.
I’d also point out that given that they’ve been able to figure out the problem with a 47-year old spacecraft from 1 light-day away with ** 12 people **, I kinda don’t think they have a problem.
I think it’s not just age per se, but lack of proper power supply now (RTGs).
For a while already, the electronics and instruments are forced to keep working without the heating.
Secondly, the electronics are constantly working more and more torwards their electrical limits, already close to brownout perhaps, not sure.
The symptoms we see now aren’t too unusual to undervoltage.
Try using a solar calculator with insufficient lighting and it will produce wrong numbers.
No, it’s age, they’re years away from running into power problems with just operating the FDS and transmitters, and the heaters that were turned off were for science instruments.
This isn’t the first time both Voyagers have had problems with the system memory. V1 lost half its memory (the backup) and V2 lost a block due to permanent failure during the prime mission decades ago, and they’ve had plenty of periodic WTFs since, they’ve just been recoverable. There’s a paper from back in 2016 detailing the state of both spacecraft.
“No, it’s age [..]” Boy, that’s quite a stubborn point of view (imho). As if we laymen had any insights what’s actually going on.. *sigh* 🙄
Speaking of stubborness, personally, I don’t think that age alone is the culprit. Wear, maybe. But not “age”.
Space is a controlled environment, unlike earth. Except for space radiation, it’s a perfect place for preservation, like a vaccum chamber.
My Sharp MZ-80K computer on the desk here was built in 1978 and still works, without any maintenance. My FT-277 radio (Yaesu FT-101), as well. Or that Tono RTTY terminal.
My home is full of 1970s appliances which still work fine. all under destructive influence of the earth’s atmosphere. But of course, these weren’t running 24/7.
But some C64 computers may still be running for ages, which aren’t much younger than the Voyagers. They use 1970s tech, as well. Chips like 6502, Z80 or 8088 are children of the 70s.
“Boy, that’s quite a stubborn point of view (imho). As if we laymen had any insights what’s actually going on.”
Yes, me, a scientist in the cosmic ray field who’s actually met some of these guys could have no possible insights into what’s going on. Congratulations.
“Wear, maybe. But not “age”.”
There isn’t any difference to something that’s powered on 24/7. And “space is a controlled environment” is… an interesting phrasing. It isn’t. I mean, the combination of radiation, vacuum, and thermal cycling is just extreme stress.
seems you would have unused portions of the ram that were used by systems that are no longer used/not functioning/don’t have enough power. just remap your memory address into those areas.
V1 is already only operating with half its original RAM since one of the entire modules failed back in ’81, so it’s not quite that trivial. However, they did plan for partial memory failures (34 years ago!) so there is A Plan, just needs to be worked into what the working memory space is.
There are two articles I’ve recently found on which are covering the Voyager computer system(s).
First one is in German, of course. Because.. All good things are in German, after all. Just kidding (I have a poor sense of humor). ;)
Second one was being linked in the first one.
https://www.dev-crowd.com/2018/12/22/retro-engineering-am-beispiel-voyager-1-und-voyager-2-mission/
https://www.allaboutcircuits.com/news/voyager-mission-anniversary-computers-command-data-attitude-control
Like most NASA probes the designs are generally published and available, although most of them are under AIAA paywalls, I guess. The computer systems are covered in doi://10.2514/6.1987-501.
Some of the details in the article aren’t right – I think they misunderstood the engineer. Voyager *in total*, between all the computers, has about ~64 kB, not the CCS alone. All of the components are redundant, and the individual ones are:
CCS: 4096×18
AACS: 4096×18
FDS: 8192×16
times 2, for a total of 557,056 bits, or 68 kB total, although doing it in “bytes” is a bit wrong there as you can see.
I believe there is a third Voyager, the engineering test one. There are some photos of it on the WWW.
On the other hand, most of the scientific instruments have passed to the next dimension, so the memory requirements should be lesser. Can’t wait to see how they work around this one.
That came also to my mind.
In essence, Voyagers are merely listening to background noise anymore.
The most important part would be to keep the star tracker for Canopus working.
So the Voyagers can keep pointing their high gain antennas back to home.
No, they’re not! Both V1/V2 still have working particle/magnetic field detectors, and they’re sampling the low energy cosmic ray spectrum that can’t be observed at Earth as well as the transition out of the Sun’s magnetic field.
They’re very useful scientifically, but sadly the instruments will need to begin to shutdown soon in order to keep comms.
Okay. Though how does it contradict my statement?
I said they’re listening to background noise, which I consider including all sorts of radiation in space. It’s a radio receiver that’s essentially listening here. Other instruments like the cameras are long being disabled already.
What are you talking about? Its not a radio receiver. They’re specific particle detectors designed for this. This was always part of Voyager’s mission, just not quite the duration, obviously.
I’d count ‘charged particles’ as radio sources, too.
I’ve built myself a geiger counter using a geiger tube and some caps years ago.
The basic principle isn’t much different to a radio receiver (say, a coherer), except for a different type of “antenna” (feeler, sensor).
But ok, I suppose we’re seeing things with different eyes.
My apologies for my bad wording, also. I think I’ve should have been more precise.
“I’ve built myself a geiger counter using a geiger tube and some caps years ago.”
You… you do realize you’re basically saying the equivalent of “I built a radio detector by plugging an antenna into a radio receiver,” right? A Geiger tube is the detecting element. Everything else is support electronics.
But you’re just really stretching definitions to fit your original statement here. These aren’t geiger counters. They aren’t radio receivers. They’re multilayer detectors that can do particle spectroscopy splitting into both energy and particle mass. They measure energy deposition of the particles as they pass through the detector along with total energy.
Let me be clear: I build radio receivers *and* particle detectors for a living. They’re not *remotely* the same. I’m at the high-energy end of the particle spectrum and the Voyager stuff’s at the low-energy end, but Voyager gets plenary/summary talks at conferences anyway so I know quite a bit about the science involved.
They’re not just measuring background noise with some crappy receiver. They’re literally travelling through an area of magnetic/particle flux change and they’re measuring *all* of it.
The reason why the Voyager missions still get the attention from NASA isn’t just for the flex. They’re actively doing science with detectors that were *designed* to do this kind of science.
The shut-down experiments on V1/V2 could conceivably be working, there’s just no funding to operate them because there’s no science return. The original Voyager Interstellar Mission had 7 funded instruments – two more have been shutdown since as the probe’s gotten farther, and V1’s PLS (plasma science) instrument was shutdown because it malfunctioned during the prime mission and had limited science benefit.
As mentioned above it’s not quite *that* trivial because V1 already lost half its memory shortly after launch.
Shouldn’t have used Micron Technology RAM chips… ;)
CMOS at least.
https://news.ycombinator.com/item?id=39949555
Although I guess we’re finding out how durable core can be as well.
The FDS memory (which is where the issue is) is CMOS, yeah, because it’s less “total loss of spacecraft” risk. The CCS/AACS, which handle command/control/pointing/etc. both use a derivative of core (plated wire) to ensure rad-hard behavior. The FDS is what actually records data.
Hence why the spacecraft’s still pointing properly and receiving commands, just sending back apparent nonsense.
Hopefully they designed the “Intel inside” stickers to peel off at memory errors :D
Patching the firmware??
I’ve had things fail right in front of me, let alone over 15 billion miles away.
How does it still get a signal??
I can’t even get a mobile signal on the IOW!! 😂😂
> I can’t even get a mobile signal on the IOW!
That’s because they’ve only just accepted Teletext as a dangerous new fangled contrivance.
Was the memory implemented using some sort of error correcting code?
You would think after 47 years they would have invented another space craft with updated high tech communication. Longer range as well with video communication.
Yeah the only problem is that the planetary alignment for gravity assist needed for the Voyager missions takes significantly longer than 47 years. Also terrestrial electronics design is not geared towards multi-decade reliability.
also that gosh-darned inverse square law
” Also terrestrial electronics design is not geared towards multi-decade reliability.”
Not anymore, I think. Our technology from the 70s and early-mid 80s was, we had been there.
It was a combination of simplicity and elegance. The bigger structures were more tough, too. Also in terms of radiation. Let’s just think of an 6502 or an Cosmac 1802 (non-space hardened version, too!). Or heck, relays logic. Alas, that’s what many people don’t like to see. Because it can’t be what shouldn’t. It’s against their ideology.
Your thoughts aren’t wrong my friend, but the Voyagers also weren’t meant to leave solar system ASAP.
The primary mission was the exploration of the planets, the grand tour. That wouldn’t have been possible in same way if the Voyagers had been moving too fast at the time. Or in other words, the 70s tech wasn’t the hindrance for their traveling speeds. There’s another project (“Interstellar probe”) that aims to leave solar system at high speeds. Haven’t heard much about for a long time, though. https://en.wikipedia.org/wiki/Interstellar_Probe_(spacecraft)
The Grand Tour wasn’t the original mission either, it was only supposed to be Juptiter & Saturn.
But yes, they’ve done very well.
They have for another deep space mission:
New Horizons
https://science.nasa.gov/mission/new-horizons/
New Horizons is a NASA mission to study the dwarf planet Pluto, its moons, and other objects in the Kuiper Belt, a region of the solar system that extends from about 30 AU, near the orbit of Neptune, to about 50 AU from the Sun.
It was the first mission in NASA’s New Frontiers program, a medium-class, competitively selected, and principal investigator-led series of missions. (The program also includes Juno and OSIRIS-REx.)
New Horizons was the first spacecraft to encounter Pluto, a relic from the formation of the solar system. By the time it reached the Pluto system, the spacecraft had traveled farther away and for a longer time period (more than nine years) than any previous deep space spacecraft ever launched.
———-
Also, NASA is working on laser communications for space probes:
NASA’s Deep Space Optical Communications experiment beamed an ultra-high definition streaming video on Dec. 11 from a record-setting 19 million miles away (31 million kilometers, or about 80 times the Earth-Moon distance). The milestone is part of a NASA technology demonstration aimed at streaming very high-bandwidth video and other data from deep space…
I always think it’s funny in scifi when machines are considered immortal when it’s an absolutely heroic accomplishment (deserving of much praise) to get something like this to last fifty years, which would be too young to die for a person
It depends, I think. In space there’s no air, no corrosion. Our technology on earth, just like our buildings, is constantly being exposed to humidity, wind and temperature changes. All sorts of that accelerate an aging process. In space, it’s just radiation. Use lead plates for sensitive electronics and the problem is solved.
There is still that itty bitty problem that is called ‘temperature’, most electronics don’t like the extremes of space.
True, but what about temperature changes in space (inside Voyagers) vs on earth?
Material can also become brittle from extending/shrinking due to temperature.
Like a wire that’s being bent back and forth until it breaks (to use a very basic comparison).
All electronics have a shelf life. Everything goes bad eventually, even in a perfect climate-controlled environment and especially if it has power applied to it. Look up electromigration some time. Or perhaps tin whiskers. Yes, tin whiskers still occur in leaded solder, just more slowly. Even if you avoid cheap electrolytics with their 15 year lifespan, the plastics in your polymer caps will eventually become brittle and crack. Guess what took out a PLL loop circuit in this exact same Voyager 2 probe in 1978, making it damn hard to communicate with?
As for lead plates, no, that isn’t effective. For many scenarios in space even a shell of lead 10 feet thick wouldn’t be enough. Building a “radiation proof box” isn’t as easy as building something air or water tight. If it were that simple then there would be no need for chips like the IBM RAD6000.
The reality is that the only thing that can reduce radiation is *mass,* the one thing you want to avoid on a spacecraft, and the operative word is *reduce.* For every extra inch of material thickness with a density of some mass per cubic inch, you reduce radiation by some percentage. For many of the radiation sources that exist in the universe (eg jupiter’s radiation belts), there is no material that exists which can bring the radiation rate down to a reasonable level with a reasonable thickness. Neutrino experiments require as close to absolute silence as possible, and at a depth that dwarfs the sheilding of a simple sheet of lead they still require additional radiation detectors to identify events caused by cosmic radiation that gets past.
Thanks you for the explanation. There are a few things to consider, though.
Like for example, the use of Nuvistors, which were already being used in early days of space age.
They’re like miniature tubes, but with ver low heating required.
They’re no semiconductors and are radiation proof, just like real tubes.
Then secondly, the mass argument. It’s all about rocket starts and money, I think.
That might be worth keeping in mind, maybe. Because by the the time the Voyagers were under construction,
NASA’s budget was being reduced already. Apollo had been canceled around same time, too.
On other hand, in a fly by maneuver, the additional mass can be used for better acceleration.
The radiation shielding is most important for the computer system, also, I suppose. Like that CMOS RAM.
Other systems may not require such shielding if they’re being based on tube technology.
The cameras, for example, were using Vidicon technology or a technology derived from it.
A CCD camera would have been damaged quickly in same environment.
Nuvistors were a nice alternative for single transistors. They wouldn’t have helped with any of the failures that occurred to the Voyagers.
Can you imagine a 64kBit memory (such as the Voyagers used) made of Nuvistors? That’s at least 64000 Nuvistors.
This document gives a weight of 1.9 grams per Nuvistor. Assuming you could get the weight down to 1 gram, that’s still 64 kilograms for the Nuvistors alone.
https://frank.pocnet.net/other/RCA/RCA_Nuvistors_NIT140.pdf
“Nuvistors were a nice alternative for single transistors. They wouldn’t have helped with any of the failures that occurred to the Voyagers.”
Not in same design, but considering that early satellites had been using discreet TTL chips..
It would be an alternative technology, still.
A simpler, generational probe could be built using nuvistors. .
The goid thing about Nuvistors was that it incorporates the good parts of both tubes and transistors.
They’re often being more reliable than traditional tubes, too.
“Can you imagine a 64kBit memory (such as the Voyagers used) made of Nuvistors? That’s at least 64000 Nuvistors.”
Not in that size. But a 256 Byte memory could be built using it.
If the actual program was being stored in non-volatile core memory, it could last for decades or centuries.
Even if core memory was being too limited for daily use, it could be used to store a backup program for emergency use.
“This document gives a weight of 1.9 grams per Nuvistor. Assuming you could get the weight down to 1 gram, that’s still 64 kilograms for the Nuvistors alone.”
Thanks! 🙂👍
Just clad the thing with Mars colonising astronauts, I hear they are abolutely radiation-proof! :)
I knew it had to be a RAM issue.
“Voyager 1 Issue Tracked Down To Defective Memory Chip”
Well, considering it was launched in 1977 and built a few years before that for testing, I would say it wasn’t defective, just worn out due to high radiation levels and no maintenance after launch.
There’s a great documentary on the small remaining Voyager flight crew. Some have been on it for their entire careers. It’s Quieter in the Twilight. Recommended
The manufacturer will repair/replace the chip for free, but the unit has to be returned to the service center at owner’s expense.
And sending a service technician out there would be pretty expensive too.
Does anyone remember the alpha particle problem with 16k RAMs. Who knows what happened out there. Work around a failing chip sounds like an good approach. It’ll take time.