Every engineer is going to have a bad day, but only an unlucky few will have a day so bad that it registers on a seismometer.
We’ve always had a morbid fascination with engineering mega-failures, few of which escape our attention. But we’d never heard of the Super-Kamiokande neutrino detector implosion until stumbling upon [Alexander the OK]’s video of the 2001 event. The first half of the video below describes neutrinos in some detail and the engineering problems related to detecting and studying a particle so elusive that it can pass through the entire planet without hitting anything. The Super-Kamiokande detector was built to solve that problem, courtesy of an enormous tank of ultrapure water buried 1,000 meters inside a mountain in Japan and lined with over 10,000 supersized photomultiplier tubes to detect the faint pulses of Chernkov radiation emitted on the rare occasion that a neutrino interacts with a water molecule.
Those enormous PM tubes would be the trigger for the sudden demise of the Super-K , which is covered in the second half of the video. During operations to refill the observatory after routine maintenance, technicians noticed a bang followed by a crescendo of noise from the thirteen-story-tall tank. They quickly powered down the system and took a look inside the tank to find almost every PM tube destroyed. The resulting investigation revealed that the tubes had failed in sequence following the sudden implosion of a single tube at the bottom of the tank. That implosion caused a shock wave to propagate through the water to surrounding tubes which exceeded their design limits, causing further implosions and further destruction. The cascading implosion took a full ten seconds to finish its wave of destruction, which destroyed $7 million worth of tubes.
The interesting part about this is the root cause analysis, which boils down to the fact that you shouldn’t stand on 50-cm photomultiplier tubes. Also at fault was the testing regimen for the tubes, which the project engineers anticipated could cause a cascading implosion. They tested this but were unable to cause a cascade failure, leading them to the conclusion it wasn’t likely to happen. But analysis of the destruction revealed a flaw in the testing, which should give pause to anyone who ever had to design a test like this before.
Luckily, nobody was killed or even hurt during the Super-K incident. The observatory was repaired with upgraded tubes and remains in service to this day, with an even bigger Hyper-Kamiokande detector in the works. We’ve covered neutrino observatories before, so check that out if you want more background on the science.
Insert “series of tubes” and “they should’ve used transistors” jokes here.
—
On a serious note, high-cost engineering failures like this need to be studied so they don’t happen again.
Someone should write a book perhaps.
https://www.amazon.com/gp/product/B0B3HS4C98/
Looks like they studied it very carefully (and they did testing to discover the exact same phenomenon beforehand, but they unfortunately did it in insufficiently deep water). The new detectors have some kind of acrylic screens inside to dampen implosions, I suspect a bit like the safety devices inside CRTs once they were mature enough.
The new hyper-kamiokande has far more depth and volume, hopefully their precautions were adequate and it works without a hitch
“The new detectors have some kind of acrylic screens inside”
Outside. It’s basically a “sleeve” around the PMT, with a fiber-reinforced bottom and acrylic top. Thankfully the amount of light received in SuperK is enough that the small amount of loss in the acrylic (few percent) wasn’t a big deal.
You can see a picture of what it looks like here:
https://cerncourier.com/a/superkamiokande-to-be-rebuilt-this-year/
“high-cost engineering failures like this need to be studied so they don’t happen again.”
They’ll always happen. People just make mistakes. They won’t make the same mistake again, but it’ll be a New and Interesting Mistake. Studying them is really important, but the problem is usually that the people who make the new mistakes are the ones who didn’t study the old ones. And of course a lot of times the fixes to the old mistakes cause new mistakes because there was too much focus on the old mistake.
Sigh.
On a more direct note as someone who encountered quite a few people from Super-K at conferences very shortly after this happened I can say that there were a lot of somber tones during those talks.
Henry Petrowski wrote a book about this, “To Engineer Is Human”, which describes the cycle of building something new, making it reliable, optimizing it, finding a failure as a result of optimization, and starting again, and ways in which engineers can design the optimization problem to reduce lethality on failure. He was largely writing about civil engineering, but the same thing holds, usually with reduced lethality, for most other engineering disciplines.
great book, I’ve read it a number of times.
Aren’t PM tubes supposed to use cesium? Isn’t cesium supposed to react vigorously with water?
The amount of photocathode material (which consists of various alkali elements since they need to be low work function and the alkalis love to give up electrons) is just amazingly tiny.
If it wasn’t so depressing when a PMT cracks (goodbye, a few thousand dollars) it’d be cool – the colored tinge on it from the photocathode just disappears almost instantly as soon as you lose vacuum.
Thankfully the most common failure points are where the leads stick through so the whole thing doesn’t usually go “boom” like what happened here. PMTs in general are surprisingly strong – I sprinted away in terror after someone’s hand slipped and an 8″ one fell to a concrete floor… only to watch it bounce, totally fine – but the lead penetrations (where you’ve got dissimilar materials touching) and neck bend are susceptible to cracks. We had one manufacturer not believe us that their design was failing often until we sent them back a slew of failed PMTs all with tiny cracks at the leads.
Agreed, the printing press should have been shelved until world hunger was solved.
Likewise, nuclear physics has never brought a single bit of good to the world, nor has it ever solved any problems.
We should cut all funding from physics research until we solve all the problems the physics research could help with.
This comment gives me flashbacks to a customer who, on being told that a bug was being investigated (not fast enough for them), asked that we fix it first and then investigate it afterwards
Maybe learn to how scientific development works. Understanding elementary particles is pretty helpful to a number of technologies and knock-on effects of related discoveries are complementary to other fields.
The amount of money regularly wasted propping up all kinds of the “too big to fail” unept monopolies truly boggles one’s mind.
TARP sucked out what $375 bils out of our economy – and we could have spent those $375 bils buying each US tax-paying citizen a house of his/her dream, every man, woman, newborn and elderly. We could have reintroduced public transportation nationwide and/or feed and home all the homeless, but, no, we gave them to the banksters who have plenty of their own money (mostly stashed tax-free offshore, of course).
Regardless, our Social Security is not broke, but it will be because all kinds of morons finance their pet peeve projects while nobody is looking.
Aside from that, NASA’s budget is what, 1/100th of the GNP (compare it with 4% GNP spent during the Apollo Program)? For that puny 1/100th GNP NASA regularly invents all kinds of civil technologies (this computer included, btw), whereas those $375 bils just disappeared in the banks’ black holes, nothing new came out, no patents, etc etc.
Talk about affordable housing!
$375 billion / 340 million people = ~ 1100 usd / person
While it’s a stupid thing to bail out “too big to fail” monopolies and investment in NASA or other science would great, your math does not add up.
$375 bil for $304 mil people is only around $1200 for every person. So, no dream houses.
I think your TARP calculations are off by many factors of 10… 375 billion is about $1000 per US resident, which is a rounding error on the price of houses these days.
Funny how it’s never the rivers of gold that are being redirected daily into the pockets of the already ultra-rich that are the target of these “stop wasting money!” cries.
There are entire massive segments of the modern day economy that are less productive/well aimed than science.
I could list so many, and im sure I’d step on someone’s toes for each of them.
I’m sure almost all of us participate in multiple of them.
Here’s a simple one: entertainment. In just a few hundred years, we’ve institutionalized mass entertainment, and we all pump a lot of money (and maybe more importantly time) into it. What could we all do for the world if we spend half the time we spend watching YouTube, TV shows, or browsing social media on something of lasting importance.
It’s easy to point fingers. It’s harder to make one’s self not pointable at.
Unlimately, action is what matters, not ranting online. The later is only good if it actually increases the former (and if I rant hoping that other people fix things…. I’m missing the point)
A cascade failure (shockwave in water)
Of cascade sensors (photomultipliers)
IN A CASCADE DETECTOR!? (radiation)
Um, I think the cascade in photomultipliers is the same as the one in a cascade detector. But good try.
yo dawg…
… I put some cascade failures in your cascade failures.
… to cascade the failures into your amazing cascading cascade failures failures.
There is a neutrino “cascade event”, but I’m not sure if that has any relation to the “cascade detector” name https://icecube.wisc.edu/news/research/2019/11/what-can-cascade-events-tell-us-about-neutrino-sources/
I’m not sure where the “cascade detector” bit came from, but the higher-energy neutrinos SuperK was looking at came from atmospheric cascades (in the field, they’re “air showers”) from cosmic rays.
It’s a little funny because the video mentions this since it needs to be deep underground to avoid muons from those air showers, but when those muons decay, they decay into electrons and a muon neutrino, and that neutrino basically doesn’t care how deep you are.
What SuperK won the Nobel Prize for was observing that there were significantly fewer of those neutrinos (“atmospheric neutrinos”, specifically muon neutrinos here) coming through the Earth or from the side than you’d expect. That, in combination with other experiments, led to the conclusion that the neutrinos created in one type of interaction (say, when a muon decays) don’t stay as that ‘type’ of interaction – that’s usually called “neutrino oscillation” as in “muon neutrinos oscillate into tau neutrinos,” although in truth it’s really “muon neutrinos aren’t actually real things that propagate – the things that propagate can interact in multiple ways,” which leads to the conclusion that there’s a mass difference between those propagating states.
The nitpickingly-correct version is then:
a cascade failure (many failures from one failure)
of cascade sensors (many electrons from one photon)
in a cascade detector (many neutrinos from one cosmic ray)
Nope, because the photomultiplier is detecting the light from the cascade of the particles, e.g. muons, cosmic rays, etc…
Funny that it’s scientific inquiry you’re wanting to sacrifice for saving the planet, and not private jets and megayachts.
I’m not seeing how a few million dollars worth of PM tubes is what is preventing us from saving the planet.
So should africans not be fed..?
You’re being far too charitable to the motives of the “save the planet” people. They want to save the planet from humans by destroying all humans.
I’m soooo glad people like you are so stupid. Wherever you’re from – how was the birthrate there 100 years ago?
Did rising living standards cause a reduction of birthrates? Does it still happen?
No mattet how hard you work to “save the planet”, it is guaranteed the planet will be taken out by an asteroid. If not the sun will go nova, etc. The only way to “save” anything is to figure out how to colonize the universe. That is why spending money to find out “if there is water on mars” is of utmost importance.
“The amount of money WE spend”
Always so eager to spend other peoples money and claim it as your own.
Unless you wrote your reply with a pencil on a piece of paper and mailed it to HAD, your have written it on a device that can only exist because of an incredible amount of fundamental research. Without that research, modern computers would not exist, cars would still have carburators and we’d all die younger because of pollution.
I really hope you were trolling.
It’s really worth some contemplation that we can look at celebrities, CEOs, doctors, lawyers, real-estate moguls, etc. living lives of absolutely obscene excess and think nothing of it. But a few million for an experiment to discover some fundamental detail of the workings of the universe, and it’s immediately “oh, what a waste, let’s save the planet before we spend another dime on science!!!”
Who’s more likely to figure out how to save the planet? The types of people I listed above, or science?
X-Rays, microwaves, penicillin, insulin, vulcanized rubber, Teflon, Velcro, safety glass, these are just some of the things that were discovered in part, or in full, totally accidentally by scientists working on other things.
Yes, let’s save the planet by stopping science. Elon Musk will save us… by buying another media platform to troll the world on. SMH.
Maybe I missed it watching the video, but didn’t see anything about how the tubes and electronics were isolated and connected, there’s alot of pressure at the bottom of the tank. Also, the system design allowed a tube to fail and still function with exposed flooded elements.
The electronics were all outside the detector tanks, they just routed cables to each of the PMTs. Lots, and lots, and lots of cables.
The tubes themselves were installed in modules of 12 20″ PMTs (plus two smaller PMTs looking outward) with a support frame tiled about the detector. The paper with design details isn’t freely accessible, I think, but this image link should (?) work?
https://ars.els-cdn.com/content/image/1-s2.0-S016890020300425X-gr11.jpg
Why?
Where’s the name calling, the political bias, sexual innuendo?
dumb question: why doesn’t the water just fill the vacuum with no fuss?
it does, but then it hits the other end of the vacuum with all the energy gained by the pressure difference
That question isn’t dumb. The problem is: imagine the water is su led into the tube from all sides and due to the tremendous water pressure filling the vacuum is accelerated to extreme speeds. All the water in the tank follows as filling the void doesn’t leave a hole anywhere else than at the surface. And then the tube is filled. Water at extremely high speeds collides with water at extremely high speed. All the water that followed this water is still moving, and much water even at low speed has the energy of a freight train. In fact the mathematical equation that explain how electrical grids deal with lightning strikes were created in the palace of Versailles when someone closed a tap and with that abruptly tried to stop 100m of flowing water in the pipes leading to that tap breaking them.
The problem is: all of the energy and momentum from these incidents doesn’t just disappear when the vacuum tube is filled. Your water now is loaded with the force a major explosion would have and vacuum-filled glass tubes.
HaD comments at its finest. Thanks for great explanation.
The video gets into this around 14:15. Post-accident simulations modeled the cascade and found that when a tube at the bottom of the tank imploded the tubes next to it experienced ~30.2 MPa = ~4380 psi = ~300 atm = ~37 times the tube’s rated 8 atm maximum design pressure. No wonder almost every tube below the water line imploded.
Actual tests afterwards showed more like 8x: this is partly why the early tests didn’t show anything. The dissipation at the air was enough it wouldn’t trigger anything. The simulation was pretty conservative.
I want an audio recording of that incident.
CERN employee for 10+ years here. I want to keep doing it, so you will understand I can’t give many details. It’s not the same place, but similar environment.
The video explains in good detail the technical problem, but for an insider the root of the problem is that building these things is lead by physicists that have NO IDEA how to build machines. They are the ones that get the funding though, so they rule.
People who dedicate their career to building machines (engineers) tend to be cautious and conservative. Their opinion is often dismissed and overpowered. And here you have the result.
The surprising thing is that such disasters don’t happen more often.