After many tense months, it seems that thanks to a gaggle of brilliant engineering talent and a lucky break the Voyager 1 spacecraft is once more back in action. Confirmation came on April 20th, when Voyager 1 transmitted its first data since it fell silent on November 14 2023. As previously suspected, the issue was a defective memory chip in the flight data system (FDS), which among other things is responsible for preparing the data it receives from other systems before it is transmitted back to Earth. As at this point in time Voyager 1 is at an approximate 24 billion kilometers distance, this made for a few tense days for those involved.
The firmware patch that got sent over on April 18th contained an initial test to validate the theory, moving the code responsible for the engineering data packaging to a new spot in the FDS memory. If the theory was correct, this should mean that this time the correct data should be sent back from Voyager. Twice a 22.5 hour trip and change through Deep Space and back later on April 20th the team was ecstatic to see what they had hoped for.
With this initial test successful, the team can now move on to moving the remaining code away from the faulty memory after which regular science operations should resume, and giving the plucky spacecraft a new lease on life at the still tender age of 46.
Awesome!
Once again, FORTRAN proves its superiority to all other programming languages!
In a million years when the aliens find Voyager FORTRAN may be all that’s left of the human race, ugh.
Except it’s actually implemented in C like Bash, Rust, Python, CLISP, Tcl, modern ports of Delphi/Turbo Pascal etc.
The Voyager computers’ software is written directly in assembly
It’s a mix of assembly, Fortran, and some portions that were coded in C. Unfortunately the software itself is internal to JPL and unavailable to a FOIA request since Caltech holds the copyright.
It’s not *that* hard to find hardware details on Voyager, although the public stuff often is a little muddled. Aaron Cummings has a talk from a conference available on YouTube about the computers of Voyager which clarifies a lot of it.
An important note that gets missed is that while we might think of uploading patches to a spacecraft to be a dangerous affair that only happens every once in a while, that’s not actually the way older spacecraft worked: their limited memory meant that new software was frequently uploaded, as often as every day-ish during the prime mission. They didn’t call them patches, they called them procedure decks – like scripts.
Nope. “Both the AACS and FDS use assembly language. The CCS uses assembly language and Voyager-unique
pseudo code (interpreter).” – from the excellent paper “Voyager Interstellar Mission: Challenges of Flying a Very Old Spacecraft on a Very Long Mission” by Sun Kang Matsumoto, a long time Voyager flight software engineer.
FORTRAN *was* used on the control and analysis software running on big iron back at JPL.
Yup, you’re right! I misread the original post and thought they were talking about all the Voyager software (including the ground stuff). Thanks for catching that.
I’m pretty sure the FDS/CCS/AACS are all custom JPL designs as well, just to make things worse for maintenance. There’s sadly precious little information on those guys out there.
F. Yeah.
Now that’s a high risk remote bug fix. One wrong move and you can’t just call somebody to go hit the reset button on the server.
It’s not high risk, actually. This is the FDS, not the CCS/AACS: it’s not responsible for commanding and interfacing. Instead it just handles science/engineering data packing. It’s been reprogrammed many times.
It’s also why it failed: the FDS used CMOS memory as opposed to core (ok plated wire but it’s core) which is what the CCS/AACS used.
Dang, you just can’t break that thing! Really amazing bit of work.
Voyager1 to Energizer Bunny : Hey kid, hold my beer.
*Obligatory Star Trek V’ger reference*
Meanwhile some unlucky alien in deep space has his home automation messed up when a firmware update from planet earth is accidentally flashed to his toaster, bricking it.
If it was a talking toaster, he may be very thankful for it.
Can I interest you in some toast?
It is now a microwave
A new lease until that next chip dies… this is just prolonging the inevitable: chips will die of old age and they are all about their expiration time… but I get it: while we can, we’ll squeeze every little bit of life of them :-)
The chips aren’t likely to be what kills the spacecraft, the lack of power is. They’ve got an internal goal of getting to 50 years (’27) – they start running into hard, unbeatable limits in the ’30s and beyond.
I wonder if this also means giving up on redundancy at some point.
Normally, multiple systems run in parallel, needing multiple times the power.
If power is very low, maybe using one of each systems only is a possibility?
Yeah, no, both Voyager crafts have had significant failures in redundant systems over the years: neither of them has fully redundant FDS resources, for instance (hell V1 no longer has a *single* fully functional FDS now). Plus once you get to the point of needing to shutdown one of the FDS/CCS/AACS you might as well just shut down the whole thing anyway.
Why so negative? You always tell what can’t be done, but don’t use your imagination to tell what can be done. 🙁
As far as I know, the ZX81 toy computer was being built from defective RAM chips, to lower price.
This was being done by disablin the defective bank (ie, not using some address pins).
So it wasn’t being unusual to install RAM chips that were partly being defect.
Not sure about the Voyagers, though. Someone should assume they had used parts for professional/industrial/military applications.
On the other hand, though, space probe makers also had used bog standard aluminium foil from the super market for insulation.. ;)
The supermarket alu foil is getting thinner and thinner and thinner.
It must have been pretty good stuff back then.
Here’s a link to the Voyager materials on the NASA Technical Reports Server:
https://ntrs.nasa.gov/collections/pubspace?title=voyager
meanwhile I kept reading FDS as Famicom Disk System
Well shit.
Thank god the project wasn’t agile ;)
Reading about Voyager’s long distance tech support, I wonder if an old man might be permitted to recount his modest story of LDTS, from 44 years ago.
In September 1980, British Leyland (I know, I know!) launched the Austin Metro to its dealers and fleet customers, aboard MS Vistafjord, a Norwegian America Lines ship. Vistafjord was fairly unique, in that its 400 staff could served 700 passengers their meals, at one sitting, and in one room.
The boat did five, 48-hour trips, between Liverpool and the Isle of Man, hosting some 3,500 guests.
To keep track of these, BL Marketing (“us”) used an Apple II+ “micro”, complete with twin floppies, a monitor and a printer ….. and a very hastily acquired voltage transformer – Vistafjord operating on 110V; the European II+ on 220/240V. AppleWorks was used as the database. For each trip, shortly after leaving Liverpool, we were able to provide the ship’s Hotel Captain with a complete, printed manifest of all BL guests and staff on board (names, homes addresses, NoK, and allocated cabin etc) – “A first”, I was told.
With two trips completed without incident, I skipped a weekend trip and began to relax. On the Sunday, I got a phone call, at home. “Hello, this is Holyhead Coast Guard, I’ve got a call for you, from Jayne, aboard the MS Vistafjord”, in the Irish Sea. “Help!”, says Jayne, “The computer isn’t working!”.
I ascertained that it was switched on, and AppleWorks would load, but that the database was corrupted. I booked a call back with Holyhead CG, for an hour later.
I was not an IT specialist; this was my hobby,, outside of my normal day job. I knew that AppleWorks, as a database, was basically a spreadsheet, and it needed pre-set character lengths for the data elements. I wondered if this was the root of the problem.
An hour later, I got “patched” back to Jayne and suggested she get to the database set-up page. And there it was! One of the “column” width settings had been corrupted and needed to be set back to “40”. Bingo! The data was readable, once more. Though extremely trivial today, for a few days the team thought I was some kind of magician.
I often wondered if, outside the military, this was an early example of shore to ship micro-computer technical support?