You’ve probably heard of Mars Pathfinder, the NASA lander and its Sojourner rover that took up residence on Mars in 1997. At first, all was well, but after a few days, the lander suffered a series of total system resets. A watchdog timer detected a problem and forced reboots. [Vivek Bhageria’s] recent post looks into what happened and the eventual solution.
The root cause was priority inversion. This happens when a lower-priority task holds a resource needed by a higher-priority task and prevents the higher-priority task from executing. In this case, the operating system was VxWorks, but this can happen on any kind of priority scheduling system.
The simplified version is that there were two tasks that each used the same mutex to avoid stepping on each other. The highest priority was a data distribution task. The lowest priority task dealt with meteorology data processing. There was also a medium-priority task that handled bus maintenance.
The failure would occur when the low-priority task had the mutex and lost its time slot before releasing it. The high-priority task would then find it couldn’t take the mutex. However, the low-priority task didn’t get control again because of the medium-priority tasks waiting to run.
In effect, the high-priority task was now waiting on something that couldn’t become available until the lowest-priority task was able to run. That’s priority inversion.
Sadly, engineers noticed this happening occasionally during testing but, as we are often prone to do, they chalked it up to glitches. As one of our old bosses used to say: “If you don’t know why it happened, then you don’t know that it won’t happen again.”
VxWorks, of course, understands that this might happen and, luckily, there was enough capability on the spacecraft to turn priority inheritance on for the mutex in question. This allows the low-priority task to get a boost because a higher-priority task is waiting on it.
You can read more about the details in the post. We also covered this a decade ago with our own analysis.
Tja, this should actually lead to an assertion or at least a ticket from the Integration check. Ideally one should have cereal ownership, no shared resources and communication happens though lockless queues.
I’m not sure how a box of cornflakes would help, but I’m pretty sure the cornflakes would eventually go bad and no longer serve their purpose.
Sorry. Couldn’t resist. The mental image of a box of cornflakes strapped to a Mars lander was just too much.
Serial.
Hehe, probably Cheereos, to ensure that someone owned the duty of jumping through all of the little hoops ;)
But yeah, Serial.
This is the kind of thing that irritates me:
So $280 Million of taxpayer funds, noticed the issues during testing, didn’t fix them then, then how many man hours to fix the problem they already knew about.. Yep sure seems like a great idea to notice “glitching” but send it anyway!
When your boss is breathing down your neck and pointing to his watch, asking you why you haven’t released your code yet, everyone else has…or he’ss get some one who can.
Yeah, I see how this happened. Another corrollary of Murphy’s Law.
Also: VxWorks. When I was at 3Com we used it in our network switch control processors (I think the NDA has expired by now, the company sure has).
Pathfinder was the first “cheap and fast” project: basically just cut down on a lot of testing and accept higher risk of failure because it’s literally so cheap you can send multiple.
And, y’know, it worked?
I could not agree with you more! There is NOTHING critical about these missions. Its just a sad case of pissing money away at the thought of curiosity. So if it was not ready NASA could have just pushed the launch window out.
Pushing a launch window out when you’re talking about Mars means a delay of 26 months. NASA did it for the Insight mission, launching in 2018 instead of 2016, but it added a couple of hundred million dollars to the costs.
“When someone describes themselves as a taxpayer, they’re about to be an a**hole.”
― Demetri Martin
I’m a taxpayer every damn day of my life
“It’s okay, it just has occasional glitches that screw it up.”
Yeah, your job is to iron those out before it goes to Mars lmao. But I suspect there is more to the story than that, NASA engineers want to do their job the right way… Unfortunately there are often forces beyond their control.
So all those NASA engineers had to do was write software without any bugs! Damn, they must be kicking themselves now for missing such an obvious solution.
/s
“I don’t always test my code. But when I do, I prefer to test it in production.” — The Most Interesting Engineer in the World
I would think such a conflict in programming would have become evident much sooner. The mission, like many by NASA, ran much longer than its conservative estimated longevity.
Can’t be worse than the Mars Climate Orbiter, which was famously lost due to one module outputting data in imperial units when the rest of the system expected metric.
Still, management decisions also doomed that mission, as NASA denied requests from mission controllers to adjust the probe’s trajectory when they found discrepancies between the transmitted data and what they actually observed.
It beats bricking itself trying to play Blur’s “Song 2” lol.
But yeah no I would have a serious sit down conversation with my software engineers and project managers and encourage them to update their CVs for the job market.
Is there a story behing the “Song 2” reference ?
Ota firmware flash
This was quite a letdown especially after reading the Margaret Hamilton tribute.
+1
Its quite easy to to enable priority inversion in vxworks, but it is not on by default, it is an option when you create the mutex. Even that only protects against two task competing, you can get the situation of 3. The important thing is to define your taks so that they both don’t try and access the same resource at the same time, such as uses messaging etc
“Sadly, engineers noticed this happening occasionally during testing but, as we are often prone to do, they chalked it up to glitches.”
Just another reason why NASA should have its funds cut. If your willing to piss away that much in tax dollars you have no reason to be doing what your doing.
Perhaps you could even invest those funds into teaching common people how to spell.
I have to say your reply is particularly amusing in the context of an AI industry that is, with the active support of the US government, spending billions on producing software that can’t reliably answer basic questions. I would say if anyone has no reason to be doing what they’re doing, it’s the AI industry.
As the Proof Of Concept for “Faster, Better, Cheaper” it would have been monumentally stupid to penalize JPL for doing it in half the time for a tenth the cost.
And they did fix it, and it lasted 76 sols beyond its 7 day planned mission.
Very good value.
Same team that tested Hubble?
“When someone describes themselves as a taxpayer, they’re about to be an a**hole.”
― Demetri Martin
“If your willing to piss away that much in tax dollars”
They didn’t. They saved tax dollars doing this. The #1 absolutely most important do not break must work thing for missions like this is 1. comms with earth and 2. science/operations updates.
Those worked. They worked on the ground, they worked in transit, they worked on Mars. They were confident the mission-critical parts would work. Which they did.
The issue that they ran into was diagnosable and fixable on Mars (they have engineering models which stay on Earth for this reason). And it was diagnosed. And fixed.
Fixing minor issues on Mars is actually cheaper in most instances, because the staffing for the project has already stepped down to much lower levels – and holding up a small number of people for a time is cheaper than a large number of people.
NASA has bar none the best ROI on tax money of any part of the government. If you want to save public funds, there’s a billion things you could trim before that. And you should turn around and give that money to NASA, they’ll multiply it in twenty years by giving us solid-state computing, or teflon, or turbopumps, or any other of a bajillion things they invented which we take for granted in modern life.
NASA’s budget is about equivalent to welfare spending in Minnesota. Just that one state.
“they’ll multiply it in twenty years by giving us solid-state computing, or teflon, or turbopumps, or any other of a bajillion things they invented”
The “direct” benefit of NASA is almost certainly their smallest benefit, because even if you ignore all of the science and all of the technology, the majority of that money is staying in the country to support the continued advancement of people who teach higher-level education.
As wonky as the lander was—it was more successful than better built Viking probes it that Pathfinder was lucky enough to land in lovely terrain.
The Viking landers both sat on ugly, rock strewn vistas…with only the “Face” on Mars getting attention.
Sometimes I wonder if Viking’s pictures of hostile terrain set Mars exploration back.
It was followed by MER rovers on Bickler pantograph bogies…UAB’s Wdowiak worked on them.
Pathfinder had the cute Sojourner rover, its solar panel on top disguising the fact that it had a radioactive element (Warm Electronics Box—-WEB).
I remember getting on the internet at the Springville Road Library…set up in the reference section now a dedicated computer room.
1997 also had two great comets.
When the lander lost power, the rover would try moving around it to reestablish comm.
“In November 1997, rover team leader Jake Matijevic suggested that Sojourner might still be waking from its programmed overnight sleep and maneuvering around the lander, waiting for further commands. He said this could continue until its solar cells could no longer provide sufficient power. That was an engineering assessment of what might be happening, not an observation”
Back when I was at Imagineering, a coworker had a family member at Caltech who said the reboot issue was indeed happening, they couldn’t figure it out (yet) but they needed to meet their launch window. And as a previous commenter pointed out, yes – just OTA update the craft on is way out.
If you readers not in consumer electronics knew how prevalent this is with so much stuff that needs to be in an air-sea container by June, floating to the US from China in time to be available in time for Christmas… its why just about everything with a radio or USB port requires a FW update when you first unbox it.