At the dawn of the Space Race, when computers were something that took up whole rooms, satellites and probes had to rely on analog electronics to read from their various sensors and transmit the resulting data to the ground. But it wasn’t long before humanity’s space ambitions outgrew these early systems, which lead to vast advancements in space-bound digital computers in support of NASA’s Gemini and Apollo programs. Today, building a spacecraft without an onboard computer (or even multiple redundant computers) is unheard of. Even the smallest of CubeSats is likely running Linux on a multi-core system.
As such, software development has now become part an integral part of spacecraft design — from low-level code that’s responsible for firing off emergency systems to the 3D graphical touchscreen interfaces used by the crew to navigate the craft. But as you might expect, the stakes here are higher than any normal programming assignment. If your code locks up here on Earth, it’s an annoyance. If it locks up on a lunar lander seconds before it touches down on the surface, it could be the end of the mission.
To get a bit more insight into this fascinating corner of software development, we invited Jacob Killelea to host last week’s
Software for Satellites Hack Chat. Jacob is an engineer with a background in both aero and thermodynamics, control systems, and life support. He’s written code for spacecraft destined for the Moon, and perhaps most importantly, is an avid reader of Hackaday.
Reliability Above All Else
The conversation started about as you’d expect, with several people wanting to know what kind of languages, frameworks, and even operating systems are used in today’s spacecraft. Jacob says while there’s an incredible amount of variability out there depending on the hardware and what the software needs to do, much of it is familiar to folks like us. He says the language of choice tends to be C, and that while Linux is used, it’s generally only for higher level tasks that don’t need to happen in real-time. If not running on the bare metal, critical code is likely to be running on something like VxWorks. Although even here he cautions that the aerospace community prefers to stick with what works, so you may find the spacecraft you’ve been tasked to write code for is using an OS from the early 2000s.
Reliability is ultimately the name of the game when writing code for space applications, which brought the conversation towards fault tolerance and so-called “safe mode” operation. Given that faults can be triggered by external events outside of your control (such as cosmic rays), even the most carefully crafted and tested code may one day crash. In that event, there needs to be a secondary system that can take over and put the craft into a known-good state. Interestingly, these “safe-mode controllers” are often a dedicated module and not just a different operating mode of the main computer.
This provides true redundancy in the event of a complete computer failure, but isn’t without its own risk — Jacob recalled a mission he’d studied where a controller designed for a previous vehicle had been reused on one with a different physical layout. Everything was fine until the satellite eventually went into safe-mode, at which point the controller’s attempts to stabilize the craft actually caused it to tumble out of control. In the end the “safe-mode” ended up being anything but, and the vehicle was lost.
Test Like You Fly
Others in the chat wanted to know about what kind of simulations or testing can be done with spacecraft software here on the ground. Jacob says one of the most powerful tools is what’s called a FlatSat, which could be considered something akin to the breadboard version of the spacecraft’s final hardware. All of the craft’s electronic components are laid out on a workbench, with ample test-points that allow tapping into the power and communication busses. This gives engineers the necessary access to test different modules and simulate various failures in ways that would be difficult or perhaps even impossible if using a replica of the flight hardware.
But still, this only gets you so far. Jacob points out that it’s all but impossible to accurately simulate the space environment. There’s no way to create microgravity in the lab, so how will you know how it impacts your inertial measurement units (IMUs) in orbit? How can you be sure your optical star tracker won’t be confused as the craft rotates when your test rig is bolted to the table? The answer is, simply, that you can’t. All you can do is test for as many edge cases as you can think of, and make sure the system can fail as gracefully as possible.
That said, you can get an idea of how your electronics will respond to cosmic radiation without hitching a ride to space. Jacob says he was involved with a project where they tested their system against radiation induced faults by putting it in the cyclotron at Texas A&M University and hitting it with proton beams. This produced numerous fault conditions, some of which they were able to devise recovery procedures for. Of course there’s no way of predicting what will actually happen in space, and satellites in lower orbits are partially protected from radiation by the Earth’s magnetic field, but it’s still good to at least have an idea of what you’re up against.
On the Shoulders of Giants
In decades past, each spacecraft or mission was an entirely new venture — a real “to boldly go where no one has gone before” kind of thing. But today’s aerospace engineers have the benefit of shared spacecraft platforms and robust software frameworks that allow them to incorporate the hard-learned lessons of previous missions into their new craft.
For example, Jacob strongly recommended anyone interested in spacecraft software take a look at NASA’s Core Flight System (cFS). This Apache-licensed flight software framework can run on Linux and various real-time operating systems (RTOSs), and provides modules for various spaceflight-related tasks such as telemetry, process management, error reporting, command scheduling, etc. Even if your homebrew rover isn’t likely to get much farther than the back garden, it could probably benefit from some of the research and software development that NASA has already done.
Jacob also mentioned that more modern satellites, especially the larger and more capable variants that operate in deep space, have enough computing power that they can offer up virtualized environments for “software payloads” that are uploaded from the ground. So for example if you had some kind of research you wanted to conduct using the sensors on a given spacecraft, you could potentially upload your own code that would run in a protected environment where failures won’t endanger the craft’s vital systems.
This is especially helpful on spacecraft designed for exploration or scientific observations. It turns out the James Webb Space Telescope (JWST) uses a system like this so scientists that “visit” the observatory virtually can operate its instruments safely through programs written in, of all things, JavaScript.
Jacob was good enough to end the chat with a rundown of various systems and technologies at play in modern spacecraft, such as standardized data busses or telemetry protocols. Basically he took the time to answer the questions that most of us aren’t well versed enough to even ask, and the results were absolutely fascinating. Like they say, you don’t know what you don’t know.
We’d like to thank Jacob for giving us a taste of what it’s like to develop software for modern spacecraft. While we might not get the chance to run any of our own code in orbit, we can all certainly benefit from some of the lessons learned while operating in such a hostile environment.
The Hack Chat is a weekly online chat session hosted by leading experts from all corners of the hardware hacking universe. It’s a great way for hackers connect in a fun and informal way, but if you can’t make it live, these overview posts as well as the transcripts posted to Hackaday.io make sure you don’t miss out.
Thank you so much for this article! This is fascinating! Such an interesting shift of perspective from our everyday ‘tech’ software engineering (where I admit to ‘moving fast’ and ‘breaking things’). Thank you again – I will make sure to learn from this, and incorporate what I can into my everyday code writing.
I have always eschewed C++. For real critical stuff I would drop back to assembly (yes compiler bugs exist).
“He says the language of choice tends to be C, and that while Linux is used, it tends to be for higher level tasks that don’t need to happen in real-time.” 🤮
Please, please don’t do Unix/Linux. Too much demons and tasks. Also, case-sensitiveness and privilege levels are the least you want to tinker with in space. Use DOS. Or OS/2. Or any other, not perverted OS. Please. 🙏
Write everything in FORTH.
Forth ended with 8080 VM figForth?
But the VM software small module ideas persist?
no, that is not correct.
DOS in space? Good riddance!
http://www.bcdxc.org/stsplus_tracking_pgm.htm
https://www.orbiter-forum.com/threads/real-time-space-tracking.12975/#post-205225
Its something you may not like hearing but it is true.
How come [Jacob’s] name was not in square brackets?
The brackets are used on posts that are about an individual’s project, largely because many of them use handles that would make the sentence structure difficult to understand otherwise, such as [No true Scotsman].
On long form content that uses a person’s actual name, they are unnecessary.
Oh, and kudos to [Joe Kim] for the Title artwork!
I have often wondered why satellites don’t have a protective magnetic field. Or if they do, nobody mentions them.
Sure the earth’s field is huge – but earth is big, too. A protective field around a satellite could be much weaker. If, rather than eliminating cosmic rays, you settle for a 50% reduction, the field could be weaker still.
In spacecraft, power is a commodity and excess heat is challenging to get rid of so running a coil that powerful for so long would be very-resource consuming. Plus any magnet strong enough to deflect charged particles will likely wreak havoc with the instrumentation and would potentially disrupt MRAM, which we use for its reliability.
Sources? I don’t think that Alan is so wrong here. The principle is worth a thought. In ST:ENT (TV show) they had a charged hull, for example. Static may do already, not sure. Then V is needed, not A.
The principle is sound, but the sort of field required to deflect gamma radiation is impractical. I did look at some papers that talked about MRI-style electromagnets using superconductors but again that isn’t feasible yet and is more of a thought experiment. Another issue with the magnetic field is that the radiation would be funneled at the poles
Thank you for your research.
As an aside, anyone writing sci-fi with spaceships using “ion engines” is welcome to mention the radiation protection as a side effect of the propulsion system.
OMFG why has nobody thought of this before, you must be a _supergenius_.
Call NASA, today.
That or you’ve never passed a physics course in your life.
NASA? Super Geniuses in the US? They must be imports. 😂
Hey! Just wanted to chime in and say: It’s not impossible and a colleague of mine is actually researching this :)
The issue is usually that the required fields are too strong. Sure, the low energy particles can be easily deflected, but they don’t really pose a threat in the first place. It’s the stuff traveling at near c with loads of mega-electron volts that introduce bit-flips, or worse, latch-ups.
Then comes the issue that deflected particles would start orbiting around your spacecraft. Though my colleague is trying to figure out how to get rid of those, she calls it “regularly flushing the toilet eehhh the orbit”. It’s still in the simulation+concept phase but let’s see what happens!
Mind you that most satellites in LEO or MEO already have weak magnetic fields as they use magnetorquers. Those generate fields that align the satellite with earth’s magnetic field, mostly to desaturate the reaction wheels. They are incredibly weak and already use a lot of bus power, so anything stronger would need to be the primary payload to justify the power draw.
Cosmic rays are extremely high energy particles and, as such, a small, low energy magnetic field won’t deflect cosmic rays significantly before they’ve hit the spacecraft. You’d need a magnetic field on the order of 1 or more tesla to deal with heavier cosmic rays. (Earth gets away with 25 to 65 micro-tesla by having a huge magnetic field and an atmosphere equivalent to 10 meters of water shielding.)
It’s significantly lighter to use hardened, fault-resistant microchips than to put in the requisite megavolt electrostatic shielding or multi-tesla electromagnetic shielding.
You might want to check out my talkat the recent Embedded Linux Conference on YouTube: Keynote: Beyond World Domination: Open Source in Orbit and Beyond(https://youtube.com/watch?v=l_LbbvA0NiU&feature=share7). We use Linux, real time and bare metal operating systems but never DOS and never assembler. We use C and C++ and lots of people are interested in Rust.
Also, there are other several space-related ELC talks. You can see the list of all presentations, including ones focused on space, at: ELC Europe 2023 Presentations (https://elinux.org/ELC_Europe_2023_Presentations)
“We use Linux, real time and bare metal operating systems but never DOS and never assembler. We use C and C++ and lots of people are interested in Rust.”
What you don’t say. A Linux/FOSS conference that focuses on C++/Rust/Linux, but not DOS and Assembly. What a bummer. That’s as surprising as a Cristian conference that talks about the Bible and proves the Bible by quoting the Bible. Awrsome. 😃👍
It’s not clear to me what your point is. A general space conference might reasonably include DOS, but obviously that’s not what the focus of this conference. However, I doubt that you would heard much about DOS even in that case. It simply does not have the set of features that most modern spacecraft need (real time, robust support of modern versions of languages, efficient multi-thread, up-to-date network protocols, and a lot more).
As for assembly language, I’ve programmed in assembly on over a dozen processor *families* over five decades. I use it when appropriate. Though assembly code may be smaller or faster for small pieces of software, is simply not possible to sustain that level of quality for large projects for any practical amount of time.
@Alan
Magnetic fields only work in combo with something like an atmosphere: the field sends the particles on a detour, but they wouldn’t lose energy [1]. Their longer paths through some medium does the trick.
[1] Yeah, there’s Bremsstrahlung, but then you need magnetar-class fields, which also might do things to the on-board electronics :-)
A bit off topic, but this brings to mind a story told to me by a JPL engineer in the late 60s.
The story is about an early rocket failure involving a mechanical relay. That relay was well tested at standard atmospheric pressure, and at vacuum. But during launch, the contacts arced over at about 26,000 feet altitude. Nobody thought to test at intermediate pressures.
Space is hard.