As the first part of a series, [George Emad] takes us through a few examples of the Linux operating system being used in spacecraft. These range from SpaceX’s Dragon capsule to everyone’s favorite Martian helicopter. An interesting aspect is that the freshest Linux kernel isn’t necessarily onboard, as stability is far more important than having the latest whizzbang features. This is why SpaceX uses Linux kernel 3.2 (with real-time patches) on the primary flight computers of both Dragon and its rockets (Falcon 9 and Starship).
SpaceX’s flight computers use the typical triple redundancy setup, with three independent dual-core processors running the exact same calculations and a different Linux instance on each of its cores, and the result being compared afterwards. If any result doesn’t match that of the others, it is dropped. This approach also allows SpaceX to use fairly off-the-shelf (OTS) x86 computing hardware, with the flight software written in C++.
NASA’s efforts are similar, with Ingenuity in particular heavily using OTS parts, along with NASA’s open source, C++-based F’ (F Prime) framework. The chopper also uses some version of the Linux kernel on a Snapdragon 801 SoC, which as we have seen over the past 72 flights works very well.
Which is not to say using Linux is a no-brainer when it comes to use in avionics and similar critical applications. There is a lot of code in the monolithic Linux kernel that requires you to customize it for a specific task, especially if it’s on a resource-constrained platform. Linux isn’t particularly good at hard real-time applications either, but using it does provide access to a wealth of software and documentation — something that needs to be weighed up against the project’s needs.
I’m always so bitterly disappointed that these machines don’t run exotic hardware and use esoteric languages.
I’m sure that tax payer is better off for it. Still I’d like to know the super safe and cool language NASA. Then usually it turns out they invariably use C++.
Like brainfuck!
… or Pidgin.
I worked on testing the power supplies used in the ISS Space Station. All the electronics was 20 year old technology. They tend to use very much tried and true tech. So real fancy chips was out. Also a lot of the chips had little lead plates glued on top to to try to mitigate cosmic radiation. Yes, it was very surprising to see such old tech being used. But it worked quite well.
The martian rovers use VxWorks, which is a real RTOS.
These are much smaller than conventional OSes for several reasons: so that they can be well understood, certified, etc.
Using a conventional OS would be unreliable.
SpaceX is risking a lot if they actually went with Linux.
It was discussed here:
https://hackaday.com/2023/07/17/discussing-the-finer-points-of-space-worthy-software/
Uh, you do realize that NASA certified the system to fly its astronauts and touch the ISS, right? Given their heightened paranoia over the program, I think they’ll do just fine without armchair prejudice.
Exactly. And realize that Falcon 9
– not only has the most consecutive launch count without failure of any rocket ever
– but the successfull consecutive launch count is higher than total launch count of any rocket including with failures and even when you count ALL Soyuz models as one
and by a wide margin.
Exactly, the days of open ended budgets for space travel have ended.
Every second you spend here, is a second you can spend there.
SpaceX got where it was by knowing where to save those seconds, and where to spend them.
SpaceX got to that point by realising it’s better to blow up a $100M USD rocket and learn whats wrong than to just spend $1B USD putting attention everywhere.
It’s a matter of iteractive learning.
Long as they’re no people on top of that “lesson”.
This results in relatively rapid hardware development. But the fallacy is if good first design principles aren’t used, you can introduce sporadic catastrophic bugs that won’t occur until much later. Not suitable for manned spaceflight.
LOL implying Windows is somehow safer??? Great joke!
What he suggested is that using small scale RTOS would be better choice. Windows is not RTOS and by no means can be called “small scale”. He also clearly stated that this is not a place for general purpose operating system which Windows is.
Embedded and IoT.
https://irontech-group.com/what-is-windows-iot-its-advantages-and-comparison-with-windows-embedded-and-windows-pro/
Correction, Windows is a garbage RTOS. It is used though. Regular desktops variants are not RT though.
Would be quite embarrassing to see a pop up menu asking if you want to update Libre-Office.
This is largely not there case any more, and was previously fine due to preferences of major engineering contractors, not NASA. When costs kept getting cut, it it went. NASA has been trying to use OSS for decades now, and they do test for reliability.
Software certification disappear with the c/c++ invasion in ~1991.
Another update?
FOIA for
2022 F-35C crash aboard Carl Vinson
software certification information
https://www.prosefights2.org/irp2023/usnavy1.htm
While there are lots of certifications that focus on you having good methodology in place, I don’t know how anybody can use linux in something carrying humans and still sleep at night. There is an absolute ton of code in there that came from all kinds of different places, with absolutely no way for you to audit it. You are relying on the cursory examination of whoever happened to be on the mailing list that week, possibly 20 years ago. It’s somewhat disingenuous to claim that your business has good practices and thus should be certified, when there is no human way you can be responsible for the quality of the entire kernel.
Something like vxworks? Yeah, you can reasonably expect a team of auditors to go through a codebase that small in a reasonable timeframe. It’s why a lot of RTOSs are small in the first place.
Spacex is living on borrowed time with that flight computer setup, abusing software for purposes it wasn’t designed. Hell, they’re using realtime patches that still haven’t landed for being unable to deliver, and the version of the patches and kernel they’re using are about a decade old.
LOL so what would you recommend?
Can’t be used:
Windows is closed source and far less stable, no potential to even audit.
MacOS? Many parts of that are closed source no potential to even audit.
Maybe??
Unix is a possibility, but it would be a lot to write all the drivers, update, and troubleshoot for hardware today….since the hardware it was run on isn’t in production anymore. Writing new code = more bugs.
What is used on literally everything? Linux. What has all major tech companies contributing to it? Linux. What is used on airplanes, military platforms hell even the majority of smartphones in use today? Linux (Android has 70.92% of the global smartphone marketshare.
What does NASA use? Linux.
You sound *very* ignorant.
Did the comment about using an RTOS like vxworks go over your head?
Does the idea that an RTOS literally *cant* do all the complex computations modern equipment needs go over yours?
Must be in the same place as your idea that they include a fully loaded kernel, and not strip out literally everything possible.
Really? Where I’m from the O/S doesn’t “do complex computations.”
Modern RT Linux has better tools and it’s easier to develop for in addition to the licensing issues with VXWorks, it’s just a better tool for the job. VXWorks is a pretty good RTOS, sure, but it’s expensive, proprietary, and hard to use. Meanwhile a NASA engineer can use their normal development workflow with Linux and build it with as many, or few, features as necessary.
You might want to actually try Linux.
I recommend baremetal. Stored on metallic punch cards.
Bit-bang atoms.
“the version of the patches and kernel they’re using are about a decade old.”
It’s a good thing code doesn’t rot.
I think the Canadarm runs QNX RTOS, And there was supposed to be som assistand drone running it too, flying around the station
A lot of NASA and ESA systems use the open source RTOS RTEMS.
Love RTEMS! It’s really nice to have an RTOS that runs in QEMU easily as well as on hardware.
“SpaceX’s flight computers use the typical triple redundancy setup, with three independent dual-core processors running the exact same calculations and a different Linux instance on each of its cores”
It would be interesting to see some details. As some pointed out regular Linux distro is not an option here so I guess it’s some modified (this particular has some RT patches), stripped down kernel. So is this some of the shelf kernel from some embedded Linux vendor? How is it certified? They must have some procedures to prove it’s stability and I don’t believe it’s “this one is 10 years old – should be stable enough” approach.
Anyone with embedded/real-time mission critical Linux experience here?
You tend to use a custom Embedded Linux build you create yourself with something like Yocto. This allows you to strip the OS to just the different pieces of the kernel you need. You’re certification generally is layered sets of tests. Start with software in the loop, then some integration testing, and then hardware in the loop testing. COTS hardware is key here, since it allows you to build test benches and automated testing really easily vs custom hardware and operating system. You can also do a tiered design to your system. Often, you have real-time controllers handling system critical tasks, but the Linux systems handling overarching tasks. An example would be the Linux system computing orbital dynamics while other, real time systems control the engines.
Peripherally relevant: a relative works as a principal engineer for one of the big name defense and space companies.They build satellites and other stuff (some of which, unfortunately, is designed to kill people). I asked him what they run on their laptops. “Linux, and a separate Windows laptop for administrative and corporate stuff” was the answer.
Well that’s nice (not the kill people stuff).
Linux is a toolbox, Windows is an embarrassment.
I dont understand why so many ‘good’ apps only run on Windows eg Fusion 360, Iblox Ucenter, DesignSpark PCB etc etc.
It’s simple, Microsoft sends people to broker every large contract very aggressively and always has. Never mind that the people they send are all marketing even if you’re told they are an engineer, your boss can’t tell, and that’s who signs the contract. They do this worldwide, to governments and corporations alike.
I think (and hope) that there is no “one and only” operating system for a SpaceX mission.
Spaceflight is a fairly complex affair with a lot of subsystems and functional levels.
For some levels “a Linux” will be OK, for others it will need an RTOS or even “bare metal” software without an OS.
Space software is much, much more than just the GUIs in the glass cockpit of SpaceX’s Crew Dragon promotional videos ;-)
Sure, my thinking is more on the lines of linux being the C2 (Command and Control), providing the tasking to the RTOS used on time critical stuff, say thrusters where precisely timing duration matters.
I should ask one of my two NASA friends
Ask them why Linux is preferred to bare metal …. please !!
Stuff like C2, as well as things that take more computation and coordination will probably run on the Linux computers, while the real-time MCUs will handle things like engine control or other control tasks.
Usage is jargon. Use use instead.
Strange they didn’t use ARM for power efficiency.
But then again maybe the x86’s were useful as heaters.
As far as I know, Ingenuity goes a bit further than what the article makes out. It uses an off the shelf SnapDragon Flight – https://docs.px4.io/v1.9.0/en/flight_controller/snapdragon_flight.html – https://www.modalai.com/pages/snapdragon-flight
IMO the obsession with Realtime OS’s is a bit overrated for martian robotics applications. Particularly in what is basically a massive distributed system. I don’t think Ingenuity has the real time patches, although I might be completely wrong. Curiosity and Perseverance both use VXWorks, which is realtime from the ground up though.
Whatever happened to QNX? I thought this is where it thrived?