Humans first visited the Moon in 1969. The last time we went was 1972, over 50 years ago. Back then, astronauts in the Apollo program made their journeys in spacecraft that relied on remarkably basic electronics that are totally unsophisticated compared to what you might find in an expensive blender or fridge these days. Core among them was the Apollo Guidance Computer, charged with keeping the craft on target as it travelled to its destination and back again.
Marc Verdiell, also known as CuriousMarc, is a bit of a dab hand at restoring old vintage electronics. Thus, when it came time to restore one of these rare and storied guidance computers, he was ready and willing to take on the task. Even better, he came to the 2023 Hackaday Supercon to tell us how it all went down!
Restoration
You might have heard whispers of this effort before, or seen some of our prior coverage. Indeed, the effort to resurrect this Apollo Guidance Computer (AGC) began in earnest back in late 2018. This particular example of the AGC was found in an electronics recycler in 1976, and the plan was to restore it prior to the 50th anniversary of the first Moon landing in 1969. Marc worked with Carl Claunch, Ken Shirriff, and Mike Stewart on the project, and documented much of it on his YouTube channel under the name [CuriousMarc].
As Marc explains, Apollo actually had four main computers. There was the Launch Vehicle Digital Computer (LVDC) which was charged with steering the Saturn V from launch, along with the Flight Control Computer which was actually an analog machine. There were then two identical Apollo Guidance Computers (AGC)—one was in the Apollo Command Module (Apollo CM), and one was in the Lunar Excursion Module (LEM). There was also the Abort Guidance System for handling any situation where things got out of hand. With the aid of old NASA diagrams, Marc shows us where the AGC lived in the Command Module and the LEM.
The Apollo program marked the first time humans had trusted the piloting a spacecraft or aircraft to a computer system. The AGC was designed by MIT, built by Raytheon, and it weighed 70 pounds and drew 50 watts. While it looks bulky and heavy by today’s standards, it was wildly compact and efficient by 1960s standards. The interface for the AGC was the DSKY—short for “Display and Keyboard.” It’s how the astronauts controlled and interacted with the AGC during the mission. The AGC was absolutely mission critical. In the words of NASA engineer Dan Lickly, ‘The AGC did everything.” It was responsible for orienting the spacecraft, controlling rocket burns and guiding the craft into orbit, and for handling the landing of the lunar module as well as re-entry into the Earth’s atmosphere.
Marc tells us how one example of the Apollo Guidance System ended up at an electronics scrapper in 1975, by order of one of NASA’s engineers. It soon ended up in the hands of hacker Jimmie Loocke, who got his hands on a full two tons of Apollo program hardware. Years later, one Mike Stewart reached out to Jimmie, hoping to look at a real AGC and take some measurements for the sake of working on a replica. When this was happening, Jimmie in turn asked if the device could be powered on, and eventually a team was assembled to try and make that happen. The first attempt happened in a Houston hotel room in October 2018, and the project took off from there.
The talk includes tons of glorious internal shots of the AGC, serial number RAY 14, which was built for an engineering test series. The AGC was assembled from stacks of neatly-arranged modules full of integrated circuits, all connected through an elegantly wire-wrapped backplane. In fact, this was one of the earliest computers that relied on integrated circuits. All of these were dual 3-input NOR gates or analog amplifier ICs. As non-flight hardware, most of the modules in this AGC weren’t potted, which had the benefit of making them easier to work on. However, two of the modules for the core memory did have potting which would make them harder to work on.
Repairing and restoring the AGC was aided by official schematics, so the team knew what they were working with. Modules were tested, and the core memory—essentially the RAM—was found bad. There was also no core rope memory—essentially the ROM for the AGC which stored the program. A rope memory emulator from Raytheon was used instead for this engineering test article. This was for ease of development, as it allowed changing the stored program more easily than rewiring a core rope memory device. However, the Raytheon emulator device was undocumented, which took some reverse engineering work to figure out. Still, powering up the device with no RAM or ROM was a positive experience—the AGC still tried to boot and there were some minor signs of life.
The rest of the talk outlines how the AGC was brought fully back to life. The team improvised an FPGA memory emulator, X-rayed failed components, and built replicas of things they couldn’t replace, all in service of the final goal. Sneaky hacks were used to get bits and pieces functional again against the ods.Even connectors had to be remade from scratch since the AGC used long-forgotten standards that are no longer in use. The fact that Marc works for connector supplier Samtec proved particularly useful in this case. Prepare to choke on your beverage when he explains how much that cost.
We won’t spoil the whole journey here, because Marc’s talk is worth watching from start to finish. There are plenty of twists and turns, and all along, you’re waiting for that ultimate delight—the joy every hacker knows when the beleagured machine finally roars into life. Never mind the wonderful heartwarming moments at the end.
It’s simply wonderful to see a dedicate team bring this storied hardware back to life—and even better to see them flying simulator missions with the real AGC hardware doing its thing. It’s a fantastic restoration effort and one that was documented in intimate detail, and it’s joyous that we get to see everything that went into making this happen.
English to English translation: “dab hand at” = “good at”
Now I dig you.
Now do “ods”
Kinda crazy to think that now probably every single sensor on a rocket has more embedded compute power built-in than the whole rocket did back in those days
Yes, but reliability is still missing in modern technology.
The AGC didn’t have much raw power, but a strict and reliable priority system. It also was robust enough to withstand radiation.
And then there’s difference in software.
Systems like AGC had hand-woven read-only memory with the application software being written by hand, too.
Or being debugged and tested step-by-step, at least.
That compares like a sophisticated and optimized DOS program (or C64 program) being written in assembly language vs a bloated Windows 10 application being written in Visual Studio 202x.
I am guessing you didn’t write much code back in the halcyon days of old. While the AGC does predate my intro to computers by a decade or more, I tell you 3 times that neither the hardware nor the software was as reliable as they are now. Hand tracing a few dozen lines of code of any type without any loops or calls might give you some confidence of correctness, but anything beyond that, well…
Not to mention, calling wire wrap technology and the semiconductors of the period reliable compared to what can be done today shows a fundamental misunderstanding of the technologies both now and almost 70 years ago. The only thing that could possibly be considered more reliable is back when transistors were the size of your thumb, or even the period on a sentence, they were less susceptible to radiation flipping bits randomly. Now in every other way those transistors were far inferior, when was the last time you saw the phrase “infant mortality” when talking about electronics? Or the phrase “burned in and tested” was a selling point? But radiation hard components are a thing. Silicon on sapphire was the choice soon after the AGC was designed and became the standard for space. Now we use silicon on insulator for similar uses now. It isn’t cheap, but the ICs in the AGC were probably a $1k a piece in today’s dollars. You can get a lot of rad hard transistors for that kind of cash.
Remember that it was Apollo 11 that actually landed on the Moon because the other 10 were test articles. Not all of which flew, but NASA tested everything they possibly could at every stage of the mission and they still ran into anomalies with Apollo 11.
The Apollo program was a success because of the people. It wasn’t because of the stellar reliability of the equipment of the day. But despite it. If you want to mythologize the program, that is your focus. Not just the men that flew. But also the brilliant and dedicated men and women who made it all actually work, despite being 2 decades ahead of where they should have been.
I’m not even sure why there’s an argument here.
The more complex a system becomes, the more vulnerable it is.
The modern processors are full of flaws and workarounds for those flaws.
And workarounds utilizing flaws in order to work and thus handle other flaws.
No one really understands anymore what’s really going on.
By comparison, the AGC program is so “small” that it can be dissasembled and analyzed by a small group of people, still.
Just like there are certain people who fully understand the inner workings of an 6502 and its machine language.
Sure, the many loops and the “spaghetti code” that come with it makes things more complicated.
But it’s nowhere as complex as, say, the Linux kernel.
Anyway, I’m puzzled why we even could disagree here.
I’m neither NASA fan boy, nor American. It’s just that I do aknowledge here that the AGC was an outstanding piece of engineering for its time.
Even by today’s standard, because it is a single-purpose device that is “clean” by design. It has just one job.
There’s no other software running in background that could suddenly interfere in a critical situation.
The USSR had similar devices, but maybe were rather focused on electro-mechanical solutions.
Which is equally impressive, they really make any clock maker really proud for their precision mechanics.
https://hackaday.com/2023/01/23/inside-globus-a-soviet-era-analog-space-computer/
If my life would be at stake, I would have more faith in these relics than in an recent Linux system
or PC hardware that runs an OS in its chipset (intel chipsets run a flavour of Minix).
If it really had to be an PC hardware, I’d rather depend on DOS and a 286 system (no V86). Something that’s predictable, simple, well documented.
The Russian DOS (Paragon/ PTS) was military grade, even. :)
Um, it was an embedded application and not a general application computer. Those are two very different areas of computers, with vastly different requirements. Because of the fixed function nature, the things that can go wrong can be constrained. But that is true of pretty much every modern use of computers. For every computer that you can run random applications on, there are dozens of embedded ones that do much, much simpler things. Often just implementing a finite state machine. Like reading a switch and setting some other device depending on the state. Such computers can sell for pennies in quantity.
There are plenty of modern, real time, embedded systems that would put the AGC to shame. And it should, one would hope that nearly a century of development, a lot of it because of the projects that yielded the AGC, would have that result. Take a pacemaker, which has a lot of similarities to the AGC, being life critical would be one of them. By the early 1990s, which emcompasses my direct experience, the processors were much more sophisticated than the AGC. Likely more computational resources, even. Certainly rad hard, too many microwave ovens and such. Ever hear of a pacemaker crashing?
The Apollo program in general, and the AGC in particular, were stunning examples of engineering and technology. But that wasn’t because of how great and reliable the tech was. It was in spite of it. I have a visceral dislike for the concept of there was some Golden Age, populated with heroes and we have all degenerated since then. Primarily because of the implied assumption that we have lost things that we can never recover. The fact of the matter is we can do better. Orders of magnitude better. Now we may choose not to for whatever reason, but it doesn’t mean we can’t.
This reminds me of the gearheads who swoon over the muscle cars of the 1950s and 1960s and feel the need to trash modern cars as unreliable trash. It just isn;’t true. Granted, those old cars are easy to work on, but you were doing that all the time. Assuming they were even assembled properly in the factory, given that quality just wasn’t an issue at the time, Deming having left for Japan in disgust years earlier because no one was interested in improving quality.
Just keep things in perspective.
The American manned space program from that era was a testimonial to American ingenuity and drive to achieve a great goal. Why distinctively American ? Well considering the ‘open source’ document dump of all manner of detailed engineering data available to anyone that wants it. No other nation state has yet put a human on the surface of the moon. Much less repeat it multiple times.
It’s also a sad commentary on the bureacratic bloat that NASA management has become. The same numbnuts that decided to launch Challenger, evidently were in charge of the recent launch of Starliner – knowing there was a helium leak, but yet chose to launch commit. Leading to the inevitable “oh duh, we have a helium leak in orbit, Elon please save us !” moment.
American put humans on the moon over half a century ago, yet can’t repeat the accomplishment in the here and now. Makes one wonder about the Egyptian pyramids – how were they built ? ET’s ? or human ingenuity (and the builders knowledge, much like reaching the moon was lost over the centuries).
I’m pretty sure that the Pharaohs didn’t have to worry about getting re-elected, nor about the people whining, “What a waste of money! It should be spent on social problems!” Nor environmental impact statements. (“We need to study the effects on English sparrows of all that copper in the water!”)
In short, the pee-pul got what they wanted.
Well, um, both USA and USSR still had German scientists back then.. 😁
Nothing more American than inhaling bits and pieces and personnel from across the world and branding it as quintessentially American.
I guess no one remembers William Proxmire. While fitting in many ways, it does mean you can’t understand what happened. Because it doesn’t make sense. But demagoguing reality for political advantage seems to work. And make life miserable for everyone.
The whole series on his channel ist totally fascinating. The amount of knowledge, skill and dedication displayed by Marc, Ken and Mike is absolutely staggering.
The most epic moment was when they fixed a (resin potted!) module with a broken signal transformer, that they couldn’t replace, by rewiring some stuff, simplifying the circuit, while keeping funktionality. Mike saying “Why didn’t NASA do it this way?” was hilarious XD
[Mike Stewart] probably knows more about the AGC and it’s software than anyone else these days. Even more than the designers who are still around, because they mostly worked on a specific component.
yeah, it’s uncanny how much he knows. Even which hardware was used in which mission, or testing setup…
Back in 2019, for the Apollo 11 anniversary, I did a project where I ported a compiler of my own devising to generate AGC code and then wrote a basic game in it (lunar lander. Of course). It was a really interesting experience. The AGC feels very modern in some ways, and very dated in others. Like, it has hardware multiply and divide, but also has memory-and-io-mapped registers (and the only way to do OR or XOR operations is to load a value into a register and use a logical-operation-on-io-port instruction). Plus, 1-s complement numbers, which were weird to deal with.
The writeup I did is here: http://cowlark.com/2019-07-20-cowgol-agc It’s even all on video, if you have nine spare hours and a high tolerance for boredom!
Another fascinating series is about the HP5061A Cesium Beam Frequency Standard. Quite the quantum physics experiment-in-a-box.