SpaceX has always been willing to break from aerospace tradition if they feel there’s a more pragmatic solution. Today this is most visible in their use of standard construction equipment like cranes in their Starship development facility. But the same focus on problem solving can also be found in their software parts we don’t see. Recently we got two different views behind the scenes. First, a four-part series about “software in space” published by StackOverflow blog, followed quickly by an Ask Me Anything (AMA) session on SpaceX Reddit.
Some of the StackOverflow series cover ground that has been previously discussed. Mostly in the first part dealing with their workhorse Falcon and Dragon vehicles, and some in the second part discussing Starlink whose beta program is reaching more and more people. Both confirmed that spaceflight software has to meet very stringent requirements and are mostly close to the metal bespoke C++ code. But we receive fascinating new information in part three, which focuses on code verification and testing. Here they leverage a lot of open source infrastructure more common to software startups than aerospace companies. The fourth and final component of this series covers software to support SpaceX hardware manufacturing, which had been rarely discussed before this point. (Unfortunately, there was nothing about how often SpaceX software developers copy and paste code from StackOverflow.)
The recent Reddit AMA likewise had some overlap with the SpaceX software AMA a year ago, but there were new information about SpaceX work within the past year. There was Crew Dragon’s transition from a test to an operational vehicle, and the aforementioned Starship development program. Our comments section had a lot of discussion about the practicality of touchscreen interfaces in real spacecraft, and here we learn SpaceX put a lot of study into building something functional and effective.
It also showed us that essentially every Sci-Fi Movie Interface was unrealistic and would be unreadable under extreme conditions.
In the course of this research, they learned a lot of pitfalls about fictional touch interfaces. Though to be fair, movie and television spacecraft UI are more concerned about looking cool than being useful.
If the standard AMA format is not to your liking, one of the contributors compiled all SpaceX answers alongside their related questions in a much more readable form here. And even though there’s an obvious recruiting side to these events, we’re happy to learn more about how SpaceX have continued to focus on getting the job done instead of rigidly conforming to aerospace tradition. An attitude that goes all the way back to the beginning of this company.
The film is presented without narration, but from the Dutch title cards and the fact that it’s Philips, we gather that this factory of gigantic proportions was somewhere in the Netherlands. In any case, it looks like something right out of [Fritz Lang]’s Metropolis and turned the rawest of materials into finished consumer products.
Much of the film focuses on the making of vacuum tubes; the sheer physicality of the job is what really stands out here. The upper body strength that the glassblowers had to have boggles the mind. Check out the chops — and the soon-to-be very unfashionable mustache — on the glassblower at the 12:00 mark. And it wasn’t just the gents who had mad skills — the fine motor control needed for the delicate assembly of the innards of the tubes, which seems to be mostly staffed by women, is just as impressive. We were also surprised by the amount these manual crafts were assisted by automated systems.
Especially interesting is the section where they build the luidspreker. Without narration or captions, it’s a little hard to tell what’s going on, but it appears that they used an enormous press to form chips of Bakelite into sleek covers for the speakers, which themselves are super-chunky affairs made from scratch in the factory. We’re also treated to assembly of the radios, packaging of finished products, and a group of dockworkers who clearly didn’t read the “Fragile” labels pasted on the boxes.
One can’t help but wonder if these people had the slightest inkling of what was about to sweep over them and the rest of the world. And if they did, would they even begin to comprehend how much the very products that they were making would contribute to both the slaughter of the coming war as well as to the sparing of so many lives? Likely not, but the film is still an interesting glimpse into the creation of an industry, one that relied very much on craftsmanship to get it started.
In whichever hemisphere you dwell, winter is the time of year when viruses come into their own. Cold weather forces people indoors, crowding them together in buildings and creating a perfect breeding ground for all sorts of viruses. Everything from the common cold to influenza spread quickly during the cold months, spreading misery and debilitation far and wide.
In addition to the usual cocktail of bugs making their annual appearance, this year a new virus appeared. Novel coronavirus 2019, or 2019-nCoV, cropped up first in the city of Wuhan in east-central China. From a family of viruses known to cause everything from the common cold to severe acute respiratory syndrome (SARS) in humans, 2019-nCoV tends toward the more virulent side of the spectrum, causing 600 deaths out of 28,000 infections reported so far, according to official numbers at the time of this writing.
(For scale: the influenzas hit tens of millions of people, resulting in around four million severe illnesses and 500,000 deaths per season, worldwide.)
With China’s unique position in the global economy, 2019-nCoV has the potential to seriously disrupt manufacturing. It may seem crass to worry about something as trivial as this when people are suffering, and of course our hearts go out to the people who are directly affected by this virus and its aftermath. But just like businesses have plans for contingencies such as this, so too should the hacking community know what impact something like 2019-nCoV will have on supply chains that we’ve come to depend on.
For those unfamiliar with the IME, it is a coprocessor on all Intel devices since around 2007 that allows access to the memory, hard drive, and network stack even when the computer is powered down. Intel claims it’s a feature, not a bug, but it’s also a source of secret, unaudited code that’s understandably a desirable target for any malicious user trying to gain access to a computer. The method that [netsec_burn] outlined for getting a computer with the IME disabled from the factory is as simple as buying a specific Dell laptop, intended for enterprise users, and selecting the option to disable the IME.
Of course Dell warns you that you may lose some system functionality if you purchase a computer with the IME disabled, but it seems that this won’t really effect users who aren’t involved in system administration. Also note that this doesn’t remove the management engine from the computer. For that, you’ll need one of only a handful of computers made before Intel made complete removal of the IME impossible. In the meantime, it’s good to see that at least one company has a computer available that allows for it to be disabled from the factory.
While the focus is on high-tech factories, the content of these moodily-lit videos is pretty diverse. Never before have we been so mesmerized by the folds of an air filter or the pressing of vinyl records. Our favorite might be GOKO BANE, which takes a bumpin’ look around the Goko Spring factory. It makes us want to throw on some rags and dance like they do down in Zion.
Once in a while they will play around with the video speed of the factory process for effect, and it works nicely. If there’s any downside, it’s that no one process is shown from start to finish. But that’s not the point, anyway.
Don’t have access to a factory? Us either. But if you can get stepper motors, it’s pretty easy to make music by driving them forward, or even backward.
A factory is a machine. It takes a fixed set of inputs – circuit boards, plastic enclosures, optimism – and produces a fixed set of outputs in the form of assembled products. Sometimes it is comprised of real machines (see any recent video of a Tesla assembly line) but more often it’s a mixture of mechanical machines and meaty humans working together. Regardless of the exact balance the factory machine is conceived of by a production engineer and goes through the same design, iteration, polish cycle that the rest of the product does (in this sense product development is somewhat fractal). Last year [Michael Ossmann] had a surprise production problem which is both a chilling tale of a nasty hardware bug and a great reminder of how fragile manufacturing can be. It’s a natural fit for this year’s theme of going to production.
The saga begins with [Michael] receiving an urgent message from the factory that an existing product which had been in production for years was failing at such a high rate that they had stopped the production line. There are few worse notes to get from a factory! The issue was apparently “failure to program” and Great Scott Gadgets immediately requested samples from their manufacturer to debug. What follows is a carefully described and very educational debug session from hell, involving reverse engineering ROMs, probing errant voltage rails, and large sample sizes. [Michael] doesn’t give us a sense for how long it took to isolate but given how minute the root cause was we’d bet that it was a long, long time.
The post stands alone as an exemplar for debugging nasty hardware glitches, but we’d like to call attention to the second root cause buried near the end of the post. What stopped the manufacturer wasn’t the hardware problem so much as a process issue which had been exposed. It turned out the bug had always been reproducible in about 3% of units but the factory had never mentioned it. Why? We’d suspect that [Michael]’s guess is correct. The operators who happened to perform the failing step had discovered a workaround years ago and transparently smoothed the failure over. Then there was a staff change and the new operator started flagging the failure instead of fixing it. Arguably this is what should have been happening the entire time, but in this one tiny corner of the process the manufacturing process had been slightly deviated from. For a little more color check out episode #440.2 of the Amp Hour to hear [Chris Gammell] talk about it with [Michael]. It’s a good reminder that a product is only as reliable as the process that builds it, and that process isn’t always as reliable as it seems.
Overseas factories can be sort of a mythical topic. News articles remind us that Flex (née Flextronics) employs nearly 200 thousand employees worldwide or that Foxconn is up to nearly a million. It must take an Apple-level of insider knowledge and capital to organize such a behemoth workforce, certainly something well past the level of cottage hardware manufacturing. And the manufacturing floor itself must be a temple to bead blasted aluminum and 20 axis robotic arms gleefully tossing products together. Right?
Well… the reality is a little different. The special sauce turns out to be people who are well trained for the task at hand and it doesn’t require a $1,000,000,000,000 market cap to get there.
[Adam leeb] was recently overseas to help out with the production ramp for one of his products and took a set of fantastic videos that walk us through an archetypical asian factory.
I’ve been to several factories and for me the weirdest part of the archetype is the soul crushing windowless conference room which is where every tour begins. Check out this one on the left. If you ever find yourself in a factory you will also find a room like this. It will have weird snacks and bottles of water and a shiny wood-esque table. It will be your home for many, many more hours than you ever dreamed. It’s actually possible there’s just one conference room in the universe and in the slice of spacetime where you visit it happens to be in your factory.
Ok, less metaphysics. It’s amazing to watch the myriad steps and people involved in taking one product from zero to retail-ready. [adam] gives us a well narrated overview of the steps to go from a single bare board to the fully assembled product. From The Conference Room he travels to The Floor and walks us through rows of operators performing their various tasks. If you’ve been reading for a while you will recognize the pick and place machines, the ovens, and the pogo pin test fixtures. But it’s a treat to go beyond that to see the physical product that houses the boards come together as well.
Check out [adam]’s videos after the break. The first deals with the assembly and test of his product, and the second covers the assembly of the circuit boards inside which is broadly referred to as SMT. Watching the second video you may notice the funny (and typical) contrast between the extremely automated SMT process and everything else.