Nearly three years after the rapid unplanned disassembly of the Arecibo radio telescope, we finally have a culprit in the collapse: bad sockets.
In case you somehow missed it, back in 2020 we started getting ominous reports that the cables supporting the 900-ton instrument platform above the 300-meter primary reflector of what was at the time the world’s largest radio telescope were slowly coming undone. From the first sign of problems in August, when the first broken cable smashed a hole in the reflector, to the failure of a second cable in November, it surely seemed like Arecibo’s days were numbered, and that it would fall victim to all the other bad luck we seemed to be rapidly accruing in that fateful year. The inevitable finally happened on December 1, when over-stressed cables on support tower four finally gave way, sending the platform on a graceful swing into the side of the natural depression that cradled the reflector, damaging the telescope beyond all hope of repair.
The long run-up to the telescope’s final act had a silver lining in that it provided engineers and scientists with a chance to carefully observe the failure in real-time. So there was no real mystery as to what happened, at least from a big-picture perspective. But one always wants to know the fine-scale details of such failures, a task which fell to forensic investigation firm Thornton Tomasetti. They enlisted the help of the Columbia University Strength of Materials lab, which sent pieces of the failed cable to the Oak Ridge National Laboratory’s High Flux Isotope reactor for neutron imaging, which is like an X-ray study but uses streams of neutrons that interact with the material’s nuclei rather than their electrons.
The full report (PDF) reveals five proximate causes for the collapse, chief of which is “[T]he manual and inconsistent splay of the wires during cable socketing,” which we take to mean that the individual strands of the cables were not spread out correctly before the molten zinc “spelter socket” was molded around them. The resulting shear stress caused the zinc to slowly flow around the cable strands, letting them slip out of the surrounding steel socket and — well, you can watch the rest below for yourself.
As is usually the case with such failures, there are multiple causes, all of which are covered in the 300+ page report. But being able to pin the bulk of the failure on a single, easily understood — and easily addressed — defect is comforting, in a way. It’s cold comfort to astronomers and Arecibo staff, perhaps, but at least it’s a lesson that might prevent future failures of cable-supported structures.
[via New Atlas]
The choice to go with the lowest bidder always has consequences.
Potentially, but going with the second lowest would just drive up prices and not improve quality. There’s plenty of companies who offer nothing better but just a higher price.
The problem is not in selecting the lowest bidder, but selecting the lowest bidder without checking they’re currently competent.
It’s always easy to say that, but is using these sockets is that fraught, then maybe the engineering values for them need to be modified to reduce their maximum loads so that assembly issues are not a problem.
These sockets are and have been repeatedly tested by the US Navy, according to the retired materials engineer from the Navy. It is the only cable end (circa 1994) approved by the Navy.
This whole issue falls back to the level of construction inspection, regular continuing condition inspection, and maintenance provided for the system. Improper construction methods mixed with poor condition inspection and maintenance are a formula for a catastrophic failure.
In this case those consequences were a system that lasted almost 60 years, with multiple upgrades in functionality. Seems like the system worked pretty well.
The issue with the zinc sockets seems to have been one that wasn’t anticipated in the original design. It’s something inspection could have caught, and then fixed in some way, so I’d see this as more of a maintenance issue than a design or construction flaw.
Absolutely. Not only that, they had to wait 60 years for this type of technology to be used to understand what happened.
I’m thinking the same. Surly after 60 years and several hurricanes the warrantee on these things have expired!
Hi Seth i fully agree with your comments i as were these sockets fully made of zinc seems rather odd yes certainly lack of inspection would be an issue as well or poor inspection routine it certainly was a marvel when first presented to the world.
Do you really honestly think that thing was cheaply made? Jeez. High standards.
Not if the criteria are specific enough.
The problem is not the lowest bidder, it’s the lowest bidder that meets your badly-written spec.
If your process is done right the spec is correct and detailed and no-one gets paid until the job is done to the spec, and then there’s no problems.
Is this your experience from outsourcing work to lowest bidders?
It’s not something I’ve had to do, but the stories I’ve heard of going down this route is that you have to put in a lot of work monitoring the lowest bidder because they’re incentivized to cut corners, and if it does go wrong a water-tight spec is great but you still have a lot of hassle.
The people relating these stories tend to suggest preferring building an ongoing relationship built on trust – the promise of future work being a great incentive to get the job done correctly.
So, if you’ve done water-tight see specs & lowest bidder and made it work reliably, I’d love to know more – perhaps the size of project you’ve made this work for, as I imagine the dynamics change with project size. It’s always nice to hear the other side!
It’s easy to say a glib remark like that. Does anybody ever buy from the highest bidder? It’s about quality. Did they get the quality for the money they paid? And how sure are we that spending more money would yield a better outcome, or just spend more money? In the same way that the sockets weren’t the only cause, even if they were the primary, lack of money is not the only cause, even if it was an important part of it.
Well, we know why and now we can look into the details of the quality of the job, materials, standards and some heads will roll. Then publish the whole thing, including how much the investigation weighted ($) and recommandations so next time the problem will be entirely something else.
I don’t get why they used soldered connection instead of TIG welding the cable? From personal experience TIG welds tend to be much stronger than the base material.
Best way would be to avoid a cable ending at that place at all.
As far as I know (and that is not much, because, well, I only have that lot of boats I’ve seen and we have cables on boats that I never touched…) the best solution is to use a thimble and one/multiple ferrules or clamps to force fit the end of the cable. Splicing is better solution, but with thick cables it might be impossible to be manufactured.
“Problem” is, that this all needs space. Those sockets are way smaller, look totally elegant and (I think) the cable is easier to install that way.
https://en.wikipedia.org/wiki/Wire_rope has some nice pictures how it is done the better way. IMHO.
Interesting. Seems like that may be only suitable for smaller diameter. Clamping relies on friction, which I suspect requires close contact with the clamp and the rope strands being held. With the 3″ diameter cables at Arecibo I suspect that may not work so well.
As far as I know, lots of suspension bridges do their cable management that way.
Clamping is not really a problem, it just needs cable length. So again we have a size challenge.
Because welding kills the strength of the cable, which is not something you want to do. Basically, doing sockets is a good thing, because it puts the tensile loads into compression. The harder you pull on the cable, the tighter it seats into the socket.
You don’t tied knots into steel (or metal) cables because it causes too much bending which seriously reduces the strength of the cable. You also don’t do knots because a knot on it’s own reduces the strength rating by around 50%. Some are better, but I don’t know the numbers offhand, but if you’re rope can hold 10 tons and you tie a knot, suddenly it’s only good for 5 tons max.
So anyway, think of the sockets like taking a funnel and sticking a ten strand (really, it’s many more!) rope into the narrow end and then unravelling the rope so you have all the various strands and yarn fibers spread out in the bushy mass filling the vee of the funnel. Then pour in epoxy or glue and let it set. Now you have a seriously strong bond which is just as strong as the rope. You can now bolt, weld, screw, etc that connection to concrete, or some other material used to hold back the tension on the cable.
In this case, the zinc metal they used managed to creep under load, probably because some strands were touching each other and not supported on all sides by the zinc. Zinc is soft, but under compression it would only get stronger. And with the lower melting point when you poured it in, it wouldn’t affect the strength of the cable.
Very well put. I have spect the shelter on designs specifically when personnel safety is needed. When assembled correctly, they seemed at the time to be the best alternative
I’m just an amateur when it comes to welding, but my guess is that TIG-welding can make the strands lose tensile strength in the HAZ., I’m not conversant enough to know if you can apply post heat-treatment to mitigate any problems that occur when welding strands that have a diameter that are just fractions of an inch/cm.
The bonus to using molten zinc to secure the strands has the added bonus of being an anti-corrosive..
TIG welding heats the welded material beyond the point of contact (the “heat affected zone”), which would be hot enough to affect the internal structure of the steel rope in that region.
Steel isn’t just a random mix of iron and carbon. Its an arrangement of crystal grains in the metal that are formed by heat. Annealing destroys the grains and softens the steel, making it more flexible but weaker, while heat treating and quenching forms martensite, making the steel harder but brittle.
Either too brittle or too soft would create a transition zone inside the steel of the cable, where the wires would change from their designed state into a different state. That boundary is enough of a difference where it would eventually fail under the enormous tension.
Zinc has a much lower melting point than steel, and so pouring it around the wires never raises the temperature of the steel to the point where it could impact the structure of the metal. That’s one reason it’s used for sockets.
I agree. During construction they could’ve clamped the wires together arounds a post and then fill them solid with 7018 rods. Done this way it could probably survive even a direct hit from 155mm shell. If cash isn’t a problem maybe thermite welding could be used too (but it’s more popular on railway).
If Wikipedia is to be believed, TIG welding wasn’t exactly a mainstream thing (rather just catching on) when Arecibo was constructed, much less a decade earlier when it was being initially designed. It may not have even occurred to the engineers designing it that TIG welding was an option, and if it did occur to them, they may not have trusted it.
Welds tend to harden the metal, and a cable with a hardened bit is just a stress failure waiting to happen. Similar, but not the same, is that you never want to solder connectors to electric wires, but crimp/clamp then. Soldered connectors (crimped or not) fail before similar solely crimped connectors.
I’ve seen this firsthand in vehicles where welded parts vibrated and always seemed to break at the weld seam, despite metal being thicker there.
It’s a good example of why engineering is complicated, though. Simply adding strong things together doesn’t make a strong product. You need to account for how they interact and where wear, give, tolerances and all the other factors are acceptable and where they’re not. Nothing is rigid or static and everything will wear.
That’s an insightful comment, so I’m a little surprised to see one part of it. There are situations where crimped connectors make the most sense (high vibration such as cars, the need to disconnect and reconnect), and there are times when solder is clearly a better choice. So I’m surprised to see you wrote “always crimp, never solder”. Soldered wires take up much less space in an enclosure, don’t have the voltage drop of crimped connectors, and don’t require a different expensive tool for every size and type of connection.
It’s just like say – bolts and glue. Some things are best glued. Bolts are better in other applications. Crimped connectors fit some applications well, soldered wires fit others.
Anyone familiar with contractors in puerto rico, could’ve told you
it wasn’t a matter of “if” it would collapse, more a question of “when”.
Surprised it held out for so long.
I guess I’ll be That SRE:
It’s almost always a bad idea to frame the failure of a complex system as a single point of failure. Whenever a single point of failure of a complex system leads to a catastrophic failure, that’s a failure of the overall system.
You could argue it wasn’t a SPoF, given the previous breakages, but I’d say the failure to repair the previous breakages created a SPoF – it wasn’t necessary to let these failures overlap.
If you need to identify a leading “cause”, the most important one is almost always human-driven process, and I think that’s the case here: The structure stood up for decades, and if the response to the first breakages were more proactive, perhaps it would be standing now.
Translation: lousy work ethic and no pride in workmanship.
(which is what the prior poster was alluding to about the local labor pool)
And if you ask “Why?” again, insufficient oversight. If you keep asking “Why?” I bet you’ll get “insufficient funding”. I suspect the grant-writers saw it as old, low-value and expensive, and were willing to take the risks under-funding entailed. Worst-case, a catastrophic failure meant they could stop funding without pushing through the difficult decision to shut down an iconic project.
Continued underfunding hoping it would fail, and also not in a state so no congressperson to push for funding. The staff held it together with willpower until they couldn’t.
Racist much? I worked there for three years and I never met a more dedicated and hard working group as the locals working on site.
Most of them could easily make way more money working in the private sector, but stayed on to participate in the science being done and to keep the dream alive.
Good regular inspection and prompt repairs. If that was followed, perhaps the nation’s bridges and infrastructure would be in better condition. There seems to be an attitude among some managers to risk the consequences because of the internal redundancy of the system and the engineers are overly cautious.
Why, oh why, does every videoclip need blaring music in the “background”?
The creators are deaf?
I’d guess video creators have suggestions from Alphabet telling what sells more eyeballs, use loud music must be one of them. Like contractors selling military hardware with that punchy music track that makes me want to vomit.
What was the designed operating lifespan? This massive net of gossamer survived 60 years in the jungle, swept over by hurricanes, abused by low maintenance budget. I’d say it was correctly made. Nothing we currently build is going to have that track record; far from it.
Exactly put.
I’m sure there are some dams that have been around a long time.
Absolutely. Everything is a consumable eventually, this seems like a pretty good lifespan for something with tough environmental conditions. I guess then though you could say that there should have been a planned replacement program.
Did anyone have a look at the first cable that broke?
Or does it only count if it’s captured on video?
All of the analysis was on the original cable that broke.
All of the analysis was on the original cable that pulled out. The rest of them failed in the cable.
I spent 12 years working on heavy cranes, and they used both spelter sockets and wire rope “saddle ” grips. I never saw a spelter socket unserviceable, but wire grips had “5 dumb ways to die /kill someone”
Wire rope is not weldable. I wonder if Arecibo had any inspection or replacement schedule for these. Suspension bridges certainly would.
It was seen in an inspection, but the NSF had changed the management enough times that there was little understanding of the issue. Not that there was a budget to do anything about it if they had.
The folks that constructed it had long since been replaced as the managers.
Wouldn’t you expect there to be a system designed-in so that each minor cable could be hauled-out, whilst hauling-in its replacement?
It was. But the budget didn’t support spares for all of the cables and the agency dragged its feet on allowing emergency repairs, so by the time they got around to it it was too late.
I read that report over a year ago when it was first released.
Since I strangely didn’t see it in that report or anywhere else I could find, I emailed Tomasetti, the Florida Space Institute, the University of Central Florida, and the National Science Foundation to ask for the manufacturer’s name and country of origin for those defective sockets. Did I miss it?
No answer from any of them.
You can fabricate them on-site, but the problem cited is just one of the issues that need to be addressed by following instructions during the preparation of the wire and the socket. Socket and wire cleanliness, especially removal of oils and organics from manufacturing and the like, are also key.
That all comes from my memory of the trade manuals “Tiger Brand” Wire Rope used to issue to vendors and users
What for? If your home’s roof collapses after years of no maintenance, and after ignoring a leak or two, it hardly matters who the original contractor was.
China?
China is opening the world’s largest radio telescope up to international scientists
December 18, 2020
https://www.space.com/china-fast-radio-telescope-open-international-scientists
I remember reading all the booklets from Tiger-Brand Wire Rope (US Steel) MANY decades/years ago about the preparation and completion of such socket connection, and the failure mechanism is one I remember even NOW being covered in the zinc socket connection preparations and finishing.
The Tiger Brand ‘Manual’ is even available for download, still.
Is that where my 10mm went?! No wonder I can’t find.
You only had one 10mm?
so basically scientists build poop and tax payers must clean.
No, not really. Often the same quality as the highest ones.
There’s an uncomfortable number of uninformed comments here along the lines of “should have TIGged it”, “this is what you get by giving the job to the lowest bidder”, “it was bound to be overpriced, did they consider a local fabricator?” and so on. And there’s a decided shortage of people who’ve handled heavy cables.
I claim to be little better, but would direct the armchair commenters to https://www.youtube.com/watch?v=_4HrxFtgsG4 which, while it uses epoxy rather than zinc, illustrates the geometry of that type of joint and gives a good indication of the attention to detail which /should/ be applied.
The bottom line is that competent procurement will on occasion buy an extra cable and check its construction: Hell, all you have to do is cut one of those sockets in half with an angle grinder and check the peel strength of the strands. And competent maintenance has, for many years, had techniques for checking the health of lines/cables/hawsers.
So assuming it was built properly, which is likely since it was a cutting-edge cold-war research facility, the spotlight has to be on how it was treated subsequently.