This year marks the 30th anniversary of the Hubble Space Telescope. When you see all the great pictures today, it is hard to remember that when it first launched, it was nearly a failure, taking fuzzy pictures. The story of how that problem was fixed while the telescope was whizzing through space is a good one. But there’s another story: how did a $1.5 billion satellite get launched with defective optics? After all, we know space hardware gets tested and retested and, typically, little expense is spared to make sure once a satellite is in orbit, it will work well for a long time.
The problem was with a mirror. You might think mirrors are pretty simple, but it turns out there’s a lot to know about mirrors. For astronomy, you need a first surface mirror which is different from your bathroom mirror which almost certainly reflects off the back of the glass. In addition, the mirrors need a very precise curve to focus light.
Perkin Elmer — a name you don’t hear much anymore outside of the medical field, used to make many kinds of things including computers after they acquired Interdata. However, they also had a history working with optical systems, including for the KH-9 spy satellite. They were in charge of building and testing the Hubble’s mirror. It was off by about 2 micrometers. That doesn’t sound like much, but when you are focusing light — especially light from billions of miles away — it is a lot.
How Bad Was it Really?
Here are two pictures of the M100 galaxy taken by the Hubble. You can probably guess which one was before the correction and which was after.
To give you an idea of how sensitive the mirror is, they use special glass made with titanium that has a very low thermal expansion — nearly zero. The Hubble holds the mirror temperature at 70 F and the mirror only deviates about 1/800,000 of an inch in normal operation. I don’t know how to do the math, but I’ve heard that if the mirror were the diameter of the Earth, the highest deviation on it would be six inches. Amazing. Turns out the mirror isn’t as big as the Earth, but it is over 94 inches across — almost as big as the mirror at the Palomar Mount Wilson observatory.
A Tale of Two Mirrors
There were actually two mirror blanks, both made by Corning. One went to Perkin Elmer. The other went to Kodak who was making a backup mirror just in case. PE had bid $70 million while Kodak wanted $105 million to make the mirror. The cost of the Kodak mirror proposal was, in part, because they wanted to build two and use them to test each other. PE was only going to make one mirror and after NASA requested a backup mirror, they subbed that job back to Kodak who got the second blank for polishing.
I know $70 million sounds like a lot, but for this job, it really wasn’t. That means PE was scrambling to do too much without enough people and without enough time. We’ve all been on programs like that.
A technician shimmed an optical test device — a null corrector — using common washers. A piece of worn paint caused a laser that tracks the distance between the tester and the mirror to be off by just a slight amount. The official story is the tech “failed to report it,” but I would guess he was told not to report it in order to meet schedule. Wherever the blame lies, the error put the test equipment off by 1.3 mm. This led to the 2200 nanometer defect which caused the severe spherical aberration in the instrument.
According to the NASA investigation report:
The spacing of the field lens in the corrector was to have been done by laser measurements off the end of an invar bar. Instead of illuminating the end of the bar, however, the laser in fact was reflected from a worn spot on a black-anodized metal cap placed over the end of the bar to isolate its center (visible through a hole in the cap). The technician who performed the test noted an unexpected gap between the field lens and its supporting structure in the corrector and filled it in with an ordinary metal washer.
Conventional null detectors could not measure the required 10-nanometer flatness required for the mirror, so PE designed and built a special one — the one that was assembled incorrectly. Ironically, two standard null correctors reported the error, but since they were not as accurate as the custom one was supposed to be and — again, under budget and schedule pressure, they decided the certified tester was right when it said the mirror was perfect and didn’t investigate. In fact, the NASA investigation mentions that several engineers on the project had concerns about the mirror’s defects, but they were ignored.
And the Kodak mirror? It used a different test method and was actually reported to be slightly better than the PE mirror. But since PE was the prime contractor, it didn’t want to admit to Kodak having a better mirror, so those reports were not highlighted until the investigation after it was too late. Had the Hubble flown with the Kodak mirrors, it would have been fine from day one. If you are ever at the Smithsonian Air and Space Museum, it is sitting there as a reminder.
Aftermath
Some experiments could use the telescope the way it was, but others were impossible. So for three years, the telescope did the best it could do until a Shuttle mission brought it a new camera and some corrective lenses. Over time, astronauts added other instruments to the telescope, but they all had to account for the bad mirror. Even the original “eyeglasses” were eventually replaced with better ones.
Perkin Elmer didn’t look good in the investigation. They already had a strained relationship with NASA due to schedule and budget overruns. In the end, they agreed to pay back $15 million to avoid a lawsuit. Hughes, who had since bought the PE optical division, also agreed to pay $10 million. Apparently, NASA felt like Hughes found out about the problem during the acquisition about a year before the telescope’s launch and decided not to bring it up. Hughes, of course, denied that but noted that the $10 million was a “goodwill gesture.”
Lessons Learned
There are so many lessons to draw from this sad story. Make sure your test instruments are giving you the correct answers. Investigate problems even if they seem unlikely to be real. Don’t rely on single tests. But maybe the most important one is to design your tests correctly and heed their results even under pressure.
If you read the NASA report, one of the problems was that everyone assumed the mirrors would be close to correct. To totally test the mirror would require an even bigger mirror which would be cost-prohibitive — especially since you’d have to test that mirror somehow, too. No one wanted to pay for that test. But even simple tests would have shown the relatively gross error in the mirror that no one thought was possible. After all, the conventional refractive null detectors showed a problem, but either because no one wanted to hear that or the detectors were considered inferior, no one dug into the issue. So whoever designed the test plan failed to imagine that the mirror could be built that far off. But it was.
Perhaps my favorite lesson, though, is the one from what happened after all this. It would have been easy to give up once the flawed mirror became apparent. But, instead, engineers and scientists worked to find a way to fix it. How you handle success tells less about you than how you handle failures.
Of course, another lesson is to be careful with estimates and be transparent with customers. But those aren’t really technical issues.
History is full of cases where engineers knew something was wrong but either failed to act or were suppressed. The Brown’s Ferry nuclear plant comes to mind. In other cases, engineers just fail to pay attention to changes, like with the Hyatt Regency walkway collapse. My advice is don’t be one of those guys!
Another way to explain it the mirror was made perfect. Perfectly wrong. And so perfectly wrong that because they knew where the screw up was and exactly how far it was off that it was much, much easier to create the adaptive optics to fix the problem.
In a way it’s very impressive to screw up in such a way that you can precisely tell what the mistake was and the precise quantitative specifications it entails… all from the ground while your machine orbits outside the atmosphere above. All engineers make mistakes. It takes a truly excellent engineer to screw up this gracefully. It was an embarrassment for NASA at the time, but honestly NASA has had some unfair expectations. I’m stunned that they didn’t screw up way worse than this in their decades of work. I’m floored that nobody died on the moon, that the solemn presidential speech they planned for that event never had to be aired. The people who HAVE died in space are so fewer than anyone who truly knew the scale of dangers faced and the complexity of problems and logistics involved would have sanely expected. Nobody has the right to criticize their success/failure record in my opinion. Not considering the difficulty they deal with on a daily basis, and the constant, abject unknowns inherent in the field. To accomplish a fraction of what they have done is a miracle. To do it with errors so slight and far-between is transcendental.
Every mistake is a learning opportunity. This mistake was a work of art on its own, considering they were even able to diagnose it and fix it at all.
I may have a bit of a fanboy complex for the good people at NASA. I wish we’d once again fund and lean hard on public space exploration again. It’s sad that we’ve auctioned it off to private sector hacks. I said it, deal with it. Elon Musk will never be able to do what NASA did. I’m ready to eat crow, bring it on.
If I recall it was a mirror perfectly focused to image earth from a specific LEO altitude rather than infinite. Hubble was later revealed to be a buy from the in production Keyhole satellite line(primary US satellite for photo-recon) but pointed outwards rather than at Earth’s surface, probably some minor customizations for NASA but everyone apparently thought a telescope is a telescope until the focus issue became apparent. The NRO even offered NASA several more older surplus space telescopes from the Hubble era Keyhole series to use as additional Hubbles about 10 years ago.
But hen I RTFA.
I had that idea in mind, could the mistake be explained by the null corrector being calibrated for a mirror focused for earth observation rather than focused at infinity.
Do you have any resources to corroborate that ?
I came here to say this, except you have some extra Key(hole)words that make sense.
Then they made up a stupid story to cover it all up. Occam’s razor says “dave” is right!.
The Hooker telescope on Mt Wilson is 100″.
The Hale telescope at Palomar is 200″.
The Hubble at 94″ is almost the size of the Hooker but less than 1/2 the size of Palomar.
That factor of 2 better resolution with 4x the light gathering power is significant. Of course the Hale telescope is at the bottom of our atmosphere.
But a good article. Worth the read. Thanks, Al.
Whoops, I always get those confused.
And DO NOT FORGET that sometimes it costs life!
Not listening to engineers = Challenger.
I am not defending PE, and given NASA used the PE mirror and not the Kodak mirror, we really are not 100% sure how the Kodak mirror would have performed. The bigger thing to consider though is the PE engineers not being happy or questioning the mirror. To some extent it brings back memories of the space shuttle challenger explosion. I recall at the time being very upset that management did not listen to the engineers. Decades later, having experience on both sides of that aisle, I find the answer is never that simple and one dimensional. If you have a product, any kind of a product, and you have a few engineers working on aspects of it, you will never get them all to give you 100% assurity that things are good to go. Simply because nothing is ever 100% perfect. It is a hard line to walk between getting things as good as you can practically get them, and engineering paranoia. Add to that the competitive bidding, and time lines they were forced to adhere to and you have all the building blocks for a disaster. At least in the case of the Hubble it was correctable.
In my experience if an engineer tells you he has a gut-feel something is wrong, something is wrong. If an engineer tells you he’s not entirely sure it’s all right, it’s probably alright but caution is warranted.
And if an engineer assures you it’s 100% perfect, fire him and hire a lawyer.
And potentially duck and run, depending on what type of engineering it was.
Kodaker here; the Kodak mirror was ‘perfect’. I’ve seen (in fact, they share them with new engineers still even though we’ve been sold a half dozen times) the inferometric reports. There was a reason the Kodak bid was 30% more, and that was for the extra testing. PE didn’t do it, shaved their cost, won the ‘lowest bid’, and the country paid for it. They’d never done something that large or that competent and ended up needing to hire talent. Kodak made a lot of mirrors that had to work, as it was pointed out, and knew how to do it.
It’s still used as a lesson for newly hired MEs, Opticals, and Image employees. The tours themselves are worth it.
Our facility owns a PerkinElmer DSC+TGA that was purchased in 1999. They charged $1500 for each “seat’ of the analysis software, and enforced it with a parallel port dongle (which incidentally contains what looks exactly like a lithium coin cell, but isn’t, and if the ignorant replace it because there was an issue with the software/hardware config, its another multihundred dollar charge to get working again). The instrument works fine, but was EOLd by PE in 2012. Another group purchased their own PE TGA in 2015 (after ignoring cautions about our instrument issues), the newer software won’t support the old instrument, or its data files, and still uses a USB dongle to secure the software, and the instrument is virtually identical in physical shape and capabilities to the old one (except for minor modification to the sample hanging hook). And as of 2022, PE declared that instrument EOL also.
All of the problems with our instrument are not engineering flaws, but management decisions. Management decisions that place dollars to the company over the interests of their customers. I can rationalize charging by the seat for software – programmers, QA and support aren’t free. But I have an expectation, possibly unrealistic, that paying for that software support should mean that a working instrument can have upgraded software, since computer lifespans tend to be substantially shorter than instruments.
All I can say is, if you are buying equipment that costs in the tens of thousands of dollars (or more), ask careful questions about the history of past equipment support, talk to equipment owners who have had it more than a years, and think about how to get your data out if a management business decision decides that you are an orange to be squeezed dry repeatedly.
How you handle success tell less about you than how you handle failures.
Curious why both companies could not collaborate together to better cross check what each company had built?
People in manufacturing rarely have clear communication skills, even in places where the details really matter. I know this unfortunately from plenty of personal experience.
Bring into that reputation, gall, pride, and especially competing companies wanting money- and its amazing we even have anything that made it to space to begin with.
This is why you want your QC people well paid and taken deadly serious, sadly though, its often not that case anymore in America.
I’m working for a company in Germany with a long history in producing high quality stuff.
Only recently I discovered that one of our QC labs received a message from RnD questioning if a result that didn’t match their expectation was an measurement error of QC. Which they triple checked before.
This lead to the request from RnD to run the test a fouth time to verify the first three clear results again. A test in this case would bind 2 or 3 workers and some expensive machinery for at least a day.
This is a bad example on how to burn money and ressources only because one person cannot swallow his pride and accept that he was wrong with an idea.
And this is just one example of how things can go bad because of pride or greed that I encountered there.
Luckily there are also many instances which help to prevent individuals from causing too much trouble.
“History is full of cases where engineers knew something was wrong but either failed to act or were suppressed.”
The improper test technique by PE is obvious; and the complicity of some of the P-E engineers and technicians is obviously yet another link in a long chain of mistakes. But what history is full of is where managers knew that the product did not meet specs, yet ignored and/or actively suppressed this knowledge.
“Make sure your test instruments are giving you the correct answers.”
That is, test the test equipment. Test engineers and quality people seldom get any significant amount of face time with the design engineers. My policy has been to refuse to release the test driver code to the factory until a senior member of the design team reviews and approves the ATE stacks, the code, and the test spec.
This seems like the ultimate case of measure twice cut once. Also why is the Kodak mirror in the Smithsonian and not being prepared to be launched into space on another space telescope?
it’s been standing on edge in a 1G, variuable temperature open air enviroment with lots of foot traffic around it for 3 decades, it isn’t usable as it has gathered imperfections whcih would cost more to fix than make a new one.
Shame.
“Of course, another lesson is to be careful with estimates and be transparent with customers”
Unfortunately, doing a better job doesn’t always get you the contract. Perhaps customers should pay more attention to quality rather than apparent cost. Often the “cheaper” solution costs more in the long run.
And this applies in almost every industry out there. It seems that almost every industry works hard to commoditize every product and service within – to hell with actual value, to hell with doing the right thing. Customers themselves contribute to this, as most want the best price AND after things go south, the right to be “made whole” at no additional cost.
The result is that the bid process is simply a race to the bottom (lowest bid price) which then turns into “pin the tail on the donkey responsible for the shortfalls of this project”. Manufacturers, prime contractors and subcontractors blame each other, while the customer suffers.
I recently read the details of a lawsuit involving an 8-figure technology project for a high-profile venue. The players:
Company N – the client
Company I – a longstanding technology and solutions integrator
Company C – the manufacturer of the equipment
After a lengthy bidding process, Company N awarded the contract to Company I, who then subcontracted the design, products, and installation to Company C (not Cisco), the manufacturer of the equipment required for the job.
The project didn’t go well; after months installing and even more months of testing, the system never worked. In the end, the subcontractor failed. The installed system was independently reviewed by multiple respected 3rd party firms (at Company I’s cost), and all agreed that the system could not possibly meet the performance that was required by Co. N or promised by Co. C. There were known flaws with design tools and methods, and poor assumptions that could have been avoided if someone would have just asked a question or two. Until the lawyers got involved, none of these issues were made transparent to anyone upstream of Company C. But they proceeded to do the work anyway, hoping, I assume, that they’d get “close enough”.
In the documents for Company I’s lawsuit against Company C, one of the latter’s employees was quoted as saying (and I paraphrase) “We delivered a system as promised. We never promised it would actually work”.
This always happens when companies who don’t have any business being in the business try to act as middle men for other people.
The question is why didn’t N go directly to C for a quote? Probably because they did and were quoted a lot more for the real deal, or the company C refused to offer because they couldn’t do it. Company I stepped in and basically lied to both companies what they were about to do, deliberately obstructing the information exchange between N and C in the hopes that they wouldn’t notice.
It was impossible for them to be “transparent with customers”. It’s a mirror and a transparent mirror is a worthless mirror ;)
As I spend the last week correcting production aerospace parts from a production flaw, from other people who weren’t as careful- I am given a cutter by management that is not designed of the right material to last, nor the correct angles to cut the special material, and I know this because I validate on my own every aspect of what I am asked to do.
My bosses roll their eyes at me anyway. And several other issues- but its under a time frame, and I get 5 different people telling me what is “correct”, all conflicting. Despite the fact that there’s a blueprint for these.
Often, what I’ve found to be the issue in manufacturing is that the attitude of “good enough” is good enough to management, and flaws are addressed superficially, because either the engineer doesn’t consult the machinist making their parts, or management doesn’t listen to the engineer’s concerns.
Often, to boil it down- its usually someone who’s actually smart, whatever their role- being overseen by management that rolls their eyes when someone smarter than them contradicts them- especially when lots $ is involved.
As this story basically proves.
Don’t be the guy who shims your finished work- almost right isn’t right, it’s wrong.
Yep. Incompetent management shutting down or threatening people that actually care. Summa summarum.
You don’t want to end up in a position like the UK company that made their half of a pressure bulkhead fitting in a mirror image. In response the American company making the other half just mirror imaged their design rather than inform the Brits they’d screwed up. Reporting and investigating the error would’ve added a lot of delays and extra cost.
ISTR reading that in a response on Quora from one of the guys who was employed by the American company involved. Fortunately making the fitting backwards didn’t mess up anything else on whatever it was part of.
Good. Fast. Cheap.
Pick two.
>I don’t know how to do the math, but I’ve heard that if the mirror were the diameter of the Earth, the highest deviation on it would be six inches.
You could have just given the 1/800,000″ as 32 nanometers, and it would have been instantly comparable to the other numbers given in the article. This would have also put the 2200 nm error in better contrast.
“I don’t know how to do the math, but I’ve heard that if the mirror were the diameter of the Earth, the highest deviation on it would be six inches.”
You… you don’t know how to multiply something by a ratio? The scale factor is the ratio of the diameter of the Earth (12.7 Mm) to the diameter of the mirror (2.4 m). You multiply the size of the actual defect (1 inch/800000) by that, and get a bit over 6 inches.
If you understood that, congratulations. You could now pass as “competent” on this topic — for an 8th grader.
LOL, I worked with a chemical engineer who considered that changes in the third or 4th decimal place of a percentage in a spreadsheet should be indicative of a trend. The water softener in question was being fed water that varied in hardness by up to 5%, the test method varied by up to 0.5%, and the system was already at greater than 95% efficient (in the real world, you cannot force an ion exchange system to 100% efficiency, which made the competition’s claim of 99% efficiency a flat out lie). Yet management (engineers both) thought her suitable to be promoted.
I have to be a little pedantic and point out that they did not use “lenses” to correct Hubble. Hubble is a reflector telescope, not a refractor, so they had to use mirrors. Lenses would have brought with them all sorts of issues and probably not given satisfactory results. The correction instrument was called COSTAR and there is a nice Wikipedia article on it here:
https://en.wikipedia.org/wiki/Corrective_Optics_Space_Telescope_Axial_Replacement
“History is full of cases where engineers knew something was wrong but either failed to act or were suppressed.”
I will see your mirror and raise you one O-ring.
+1
Ok it was off by a 2 micrometer, but were they 0-1″ micrometer or 5-6″??
Here’s another example of something being so badly off we didn’t realise it:
Years ago, shooting rifle in a competition (match rife), we dropped back from 1000 to 1200 yards, and one person on the team was having trouble aiming. We compared his sights to teammates, no problem. We put it down to the wind and nerves, and carried on. It didn’t improve. His shots were really erratic. Then he missed completely. Eventually, we realised he’d mistakenly turned the vertical adjustment the wrong way (down instead of up). The dial on his sights was a *full rotation* out – that’s why they still looked correct when we compared to team-mates. His shots has been ricocheting off the ground into the target. We had been looking for a much smaller error in the sights.