How do you fix a shorted cable ? Not just any cable. An underground, 3-phase, 230kV, 800 amp per phase, 10 mile long one, carrying power from a power station to a distribution centre. It costs $13,000 per hour in downtime, counting 1989 money, and takes 8 months to fix. That’s almost $75 million. The Los Angeles Department of Water and Power did this fix about 26 years ago on the cable going from the Scattergood Steam Plant in El Segundo to a distribution center near Bundy and S.M. Blvd. [Jamie Zawinski] posted details on his blog in 2002. [Jamie] a.k.a [jwz] may be familiar to many as one of the founders of Netscape and Mozilla.
To begin with, you need Liquid Nitrogen. Lots of it. As in truckloads. The cable is 16 inch diameter co-axial, filled with 100,000 gallons of oil dielectric pressurised to 200 psi. You can’t drain out all the oil for lots of very good reasons – time and cost being on top of the list. That’s where the LN2 comes in. They dig holes on both sides (20-30 feet each way) of the fault, wrap the pipe with giant blankets filled with all kind of tubes and wires, feed LN2 through the tubes, and *freeze* the oil. With the frozen oil acting as a plug, the faulty section is cut open, drained, the bad stuff removed, replaced, welded back together, topped off, and the plugs are thawed. To make sure the frozen plugs don’t blow out, the oil pressure is reduced to 80 psi during the repair process. They can’t lower it any further, again due to several compelling reasons. The cable was laid in 1972 and was designed to have a MTBF of 60 years.
Finding out the location of the fault itself was quite a feat. It involved time-domain reflectometry (inconclusive), ultrasound, and radar (didn’t work) and then using an Impulse Generator-Tester (Thumper) which got them pretty close to the defective segment. What pinpointed the problem was a bunch of car batteries and some millivoltmeters. They hooked up car batteries to both ends, tapped the cable at several points and knowing the drops and resistance of the cable, got within a few feet of the fault. Finally, X-Ray equipment was brought in. Sure enough, they could see the cable shorting against the steel wall of the pipe. Cutting open, and closing it all up, required certified welders spending up to 8 hours on each section to avoid damage to the paper insulation. The welders placed their thumbs 3 inches away from the seams they were welding, and stopped when it got warm to touch, allowing it to cool off before starting again.
The failure was attributed to “TMB”, short for Thermal Mechanical Bending. TMB causes the cable to wiggle in place due to load surges. This eventually causes insulation failure due to abrasion against the pipe and separation of the many layers of paper tape. They repaired the short, put aluminum collars in most of the joints to hold the splices in place, and have added a load management scheme to reduce the current peaks. Apparently, the fix wasn’t good enough. According to this Wikipedia article, “the 315 megawatt capacity Scattergood Steam Plant (Unit 3) to West Los Angeles (Receiving Station K) 230 kV line is having to be replaced after only 45 years of operations, due to multiple failures within this rather long single-circuit, oil-filled, “pipe type” cable.”
You can read lots of other interesting bits about this repair job from [jwz]’s blog. Thanks to [J. Peterson] for sending in this tip, which was triggered by our recent post on “Why is there liquid nitrogen on the street corner?“. We also ran a post earlier today that discussed Time Domain Reflectometry which was mentioned earlier in this post.
Indeed a very interesting read.
I was just wondering if they used some kind of wave or signal reflection to detect that, just to read it a few seconds afterwards.
Altough it says it was inconclusive, was it because of the cable age? would have it worked in a newer one, or the fault itself made it impossible to work?
Shorts and breaks in communication cables can be found by the time it takes for an electrical signal to reflect off the problem. The same technique should work for a power cable.
If this is the link I followed from the comments section last week, I recall they used TDR to get close, and a “thumper” to get the final position.
It is the same link. Pretty amazing to think about, almost read like science fiction.
TDR works really well for transmission lines with perfect constant impedance. But much like a power cable makes for a crap antenna feedline a long power cable makes a hell of a mess of a TDR signal.
Fun fact when you use a thumper on a cable with a pinhole in the sheath and it suffers from water ingress, the pinhole becomes far more obvious,…. the thumper blows a massive hole in the insulation. :-)
Thank you. I wrote it.
You can eliminate the surges but evidently it is cheaper to replace cable.
The surge is a sudden change in load, not a voltage spike, so you really can’t prevent it short of putting a huge UPS in the line.
It is a classic transmission line and has propagation delay. I would think the most critical management is over voltage lest you damage the line. Momentary over current can be much longer in duration before it becomes critical.
If the short was between the conductor and the outer steel pipe, could they locate the fault just by scanning the voltage across the outer steel pipe, while some test current flows through the cable and leaks to the steel wall of the pipe?
I was thinking about this, and that could have resulted in false positives. The current will discharge from the pipe into the soil wherever there is damaged or failed coating (assuming it is coated). This discharge point could be anywhere along the pipe, and wouldn’t necessarily have to coincide with the location of the fault.
That being said, the fault could have blasted away the coating at this location, resulting in a very large coating defect that could have been identified using your suggested method (this is a variation on fairly standard pipeline integrity test methods, such as cathodic protection close-interval surveying and DC voltage gradient surveying).
Well MTBF is a statistic, if there are multiple of these cables installed around the world and the actual value is correct some power plant is going to be a happy camper not having to do a repair like this is a few decades!
Of course it this is based on the same shit estimates as harddisk manufacturers use for MTBF all others are likely to fail soon.
Even worse of this is the only cable “designed to have an MTBF of XX”, MTBF is wrong itsself, a statistic with N=1 is bullshit.
I wouldn’t go so far as to necessarily say it’s bullshit. You can use statistical modeling of the components in isolation to create an overall statistical model for the whole. What you would wind up with is an expected average MTBF. Now, it’s *applicability* to a sample size of one is arguable, but that doesn’t mean it doesn’t have some sort of validity (that is, it’s not just bullshit).
“The failure was attributed to “TMB”, short for Thermal Mechanical Bending. TMB causes the cable to wiggle in place due to load surges. This eventually causes insulation failure due to abrasion against the pipe and separation of the many layers of paper tape.”
Maybe that wasn’t counted into the specs eh. And thus not part of the MTBF calculation.
Same way the MTBF of a HD would not cover it being tossed around every day, or getting heated to 90C, or some such.
All around CA, there is quite a problem with water infiltration of power cables. The e-field and capacitance of the cable drive the infiltration, and a tiny bridge can lead to explosive failure of a cable:
https://www.youtube.com/watch?v=o2jV80zME-0
The thermal-mechanical-bending issue would be significantly mitigated if every house had a handful of lead-acid or sodium-acid batteries in the basement, with a microinverter, and a control system that participates in a region-wide planning algorithm. Power usage could be leveled over the hottest part of the day and the power lull at night.
how big of microinverter would you think you would need for your thought to be effective?
The new Tesla Powerwall is hardly a microinverter, and it only provides 2.2kw continuous load. Not that it’s a great example seeing as how all it really is is an expensive brick you can hang on your wall.
2.2kw is tiny, that would just cover a single window ac unit.
I think that that is the point alfiesauce is trying to make. A handful of lead-acid or sodium-acid batteries aren’t going to make much of a dent in the power usage spikes. It might help smooth it out just a tiny bit, which MIGHT hel in just the specific issue of thermal mechanical effects in the cables, but I think that’s questionable.
What is this “basement” you speak of?
Funny thing about power companies… a lot of them don’t care for folks to have their own on-site power generation systems, even if it means load balancing.
http://www.forbes.com/sites/jamestaylor/2014/10/17/electricity-prices-soaring-in-top-10-wind-power-states/
Also, as the wind farms here grow, these keep popping up: http://www.lawyersgunsmoneyblog.com/2014/05/idaho-anti-wind-energy-billboards
I guess that’s better than when I lived in WA: Regulators found that the electric utility had improperly raised rates more than allowed, yet turned around an approved a rate hike just a few months later.
On-site power generation systems are a real pain in the butt for power companies to manage. Their transmission lines are engineered with very specific specs in mind. When you add power generation in random spots to the mix, the voltage rise issues are very hard to deal with, especially since most power generation produces power at a very variable rate at any given moment.
Not to mention that down at my end, distribution, it can make for a very chancy safety scheme, with various power sources coming on and going off line. In farm territory, we’ve lost people, and burned up a lot of generators (improperly installed, natch) during storm repair, over the years, co-generation, if not done right, makes it even worse.
1,840,000 watts is a lot of power
Actually it is 3^.5 * 230,000 * 800 = 318 MW of power
1.21 gigaWatts? What was I thinking?!?
That’s a great story. Most people really have no idea how epic the scale of industrial processes can get. The energies involved are mind-blowing. This is a great glimpse into what it takes to make stuff like that happen and get it fixed.
In case of large electricity installations, sometimes more then just the mind is blown (up), usually in a very spectacular (and very expensive) manner ;-)
Step 1: Don’t build something that can’t be maintained/repaired in a manner less costly – or at less than a significant fraction of the installation cost – than it is to replace the entire thing.
In this case, power could be delivered with “smaller” parts in parallel that are much easier and less costly to replace and still maintain the spec load.
You can’t really make them much smaller. The issue regarding parallel cables isn’t actually the size of the conductor; it’s the size of the insulation. While you could use several smaller conductors, the insulation in each cable would have to be just as large and as expensive in order to withstand the voltage so the “smaller” cables wouldn’t be significantly smaller. You wouldn’t be saving money. You could lower the voltage, but that increases losses; it’s a tradeoff between cable losses and cable cost. About the only advantage I can think of having more than one cable is fault tolerance.
Being in the industry, I nearly discredited this article when I read the opening paragraph about using liquid nitrogen. My initial thought was, “Why didnt they just thump it?”. Glad I continued to read why the thumper was unsuccessful.
Personally I’ve never come across a situation where the thumper didn’t get you “close enough”, but I’ve also never worked in a setting quite like was described.
On a side note, most of these style of cables are going away due to the O&M costs (difficult to repair). Now the new standard seems to be injected cables where a substance with a similar consistency to sunscreen is injected into the cabled. It fills all of the voids and pushes out impurities. It’s really neat watching them pump this stuff in existing installs. All of the moisture it pushes out the opposite end is incredible. Not to mention the shear distance they can inject (miles).
Electric power generation on a large scale is a ponzi scheme, no better evidence than this that the future costs of this infrastructure disaster in waiting have any chance of getting paid……..we are all screwed……
Oh do explain.
We should never have left those safe caves.
Ah, those hunter gatherers and their 12 hour work weeks. I mean, who would wan’t that when you could work 60 hours a week? PFFFFFT!!!
That’s one hell of a Ponzi scheme given that we’ve built most of our current civilization on large scale power generation.
I do totally agree that chronically delayed infrastructure maintenance and upgrades are major issues. When these problems come up, our domestic, commercial and industrial power bills will go up to cover the cost. Which will be a pain in the backside and I am not looking forward to.
And before anyone mentions standalone systems for everyone, let me ask, what do the factories that make the silicon, the solar cells, the glass covers and metal frames run on? Or the fiberglass wind turbine blades, the gearboxes, the generators? What about the factories that make the wire, the circuit breakers, the capacitors, the transistors, the inductors in the inverters? Grid electricity (plus local power conditioning).
A ponzi scheme, defined as a scenario where the early adopters benefit more than the latter. The true costs of the infrastructure are not reflected in the current electricity rates, adjusted for inflation every successive generation will pay a higher rate than the previous.
It has a lot to do with how utility stocks are structured, they are somewhere between a bond and a regular stock. In short investors are pretty much guaranteed a return on their investment because of this “special” status.
And regarding the AC vs. DC debate, prior to grand electrification schemes, manufacturers had their own power plants. For at least 150 years, as a matter of fact. It was, a banker, by the way, who decided that AC was superior to DC (it is, in fact, in the short run), not “the market”. And who to best benefit from the massive capital allocations required for modern AC high distance power transmission? Yep, bankers….
A company is building a coking plant close to where I live, it will be self powered using a natural gas turbine, a cogeneration arrangement where the waste heat is part of the process. Not sure why it hasn’t always been done this way…..
are they producing their own natural gas as well? They are not self powered if they are not…
It’s a coking plant, they’ll be using the gas produced by the coking process. And I’m pretty sure that that’s how all coking/steel plants work.
Simply reading the title made my anus sweat. That is crazy power.
Extra high voltage transmission circuits at 345 kV and 3000 A are not uncommon these days.
Old but good. 500KV switch opened under load where the interrupter on one phase had problems. They shot the video to have proof that it was bad to get the bean counters to approve fixing it. https://www.youtube.com/watch?v=vb05j7KmPfc
That’s nice, dear. As my gran used to put it, “It makes my bum pucker”. Which I thought was quite crude up til now.
note to self: never joke around engineers. lighten up guys.
It’s what you would call a high pucker factor job.
Ive seen in person a worker repairing a 500amp 3 phase cable when our DC went down, and that was pretty interesting too. He started off with a tdr and traced the likely fault to a new lighting post put in a few weeks back by the council which had probably damaged the cable run during insertion, then before they called out the road gang to replace the entire run which ran through a few roads full of housing, he attached what I can only describe as a heart defibrulator for wires and shocked it.
He said sometimes they can blow away the short mitigating the need to do groundworks to repair, but apparently sometimes things go bang and really need repair after.
He put the energiser on and it charged with that whine, and had to do it a few times but managed to get it working. I’d guess it eroded the failed spot away with a hv pulse enough to get a plan into place to repair/replace the cable longer term.
We were on backup genny at that point, but the same cable carried a few streets of housing’s power so there was a few relieved looks around when it worked…
You can get freeze packs to do the same trick as the article in smaller scale on household plumbing, freeze the pipe upstream of the bad bit and work fast before the plugs melt.
Those cables would be a *@”!$ to weldsplice in situ too. The ones I saw at college cut away had 3 cores with a small gap between them with oiled paper and a cavity for insulating/cooling oil. Having blue glued a fair few things together with tig/mma and mig in my life, I cant imagine being able to manage that repair sucessfully, but thats why you have people who are welding deitys and paid insane amounts for those “special” jobs like this.
What ever made them think in 1972 that *paper* insulation in a power line of that capacity would last 60 years? Didn’t they have plastics of equal or better dielectric strength 43 years ago? Of course the mechanical strength would have been much greater than paper.
What kind of paper? Cotton fiber “rag” or wood? Or was it something else called paper but not actually a material used for things other than insulating high voltage cables?
Oil-impregnated paper insulation was the norm for high voltage cables for a very long time, and there are plenty of cables still around that use it, and have been operating fine for an extended period with it.
Exactly. Oil-impregnated paper is still the preferred insulator in most every transformer bushing out there.
Probably some kind of polymer would have degraded by now from long term contact with oil.
I reckon they knew a thing or two about what they were doing…
If you like this sort of thing, this article on the power crisis in Auckland NZ in the ’90s is fascinating reading: https://www.cs.auckland.ac.nz/~pgut001/misc/mercury.txt
There are some interesting similarities between the two articles:
From jwz’s:
“How do you weld a steel pipe with paper insulation inside? Slowly. They have special heliarc welding equipment and “certified operators” who take 8 hours to weld around one cross section of pipe. They are required to keep their hand on the pipe no more than 3 inches from the tip of the welder. If it is too hot for their hand they stop and let it cool.”
From the NZ piece:
“Closing up the pipe after repairs is a special task in itself because as the pipe is filled with oil and paper it has to be done with special equipment and takes 8 hours to weld one section of pipe. If the pipe is too hot to touch 10cm from the welding, they have to stop and let it cool before they can continue.”
Also:
From jwz’s:
“At both ends of the pipeline they have 6000 gallon tanks of Golden Bear lightly pressurized under a blanket of dry nitrogen. There are pumps at both ends. There is about 100K gallons in the entire pipeline, not including the 6K gal tanks. Every six hours they reverse the pumps so the oil oscillates back and forth in the pipe. The pumps only run at 3 gallons per minute but that is enough, over 6 hours, to get the oil in each 2000 foot segment to go at least a segment or two length in either direction. This eliminates hot spots in the copper conductors and spreads the heat out over several thousand feet. A little competitive pressure is always maintained between the pumps to get the 200 PSI.
They learned the hard way that you simply don’t reverse the pumps lest you get the Golden Bear equivalent of water hammer. The last hour of every 6 hour cycle is spent slowly reducing the oil velocity down to zero before you reverse it and then slowly ramp back up in the other direction.”
From the NZ piece:
“Both ends of an oil-filled cable typically have large holding tanks of oil, with pumps which are reversed every 6 hours so the oil oscillates back and forth in the cable, eliminating hot spots and spreading the heat over a large area. In the last hour of the cycle, things are slowly run down to get zero oil velocity, then everything is reversed and slowly run up again to move the oil in the opposite direction.”
Sounds suspiciously like plagiarism, in one direction or the other (confusingly, the article that appeared first seems to be the much more abbreviated one).
Like I said, interesting. There’s clearly a lot going on between the two, but you’re right, the NZ account is earlier. The again, I Googled “so the oil oscillates back and forth in the” which is found verbatim in both pieces and found this from 1989(!):
http://www.mit.edu/afs.new/net/user/tytso/archive/high-power
Man, everybody likes some good cable porn on the Internets, I guess!
Which is, of course, the exact piece reposted by jwz, just from a different source. I skipped right over the email header in his repost.
It wasn’t necessarily malicious, the author states at the end:
“This writeup originally started as a page of notes covering an afternoon power
cut. By the time it had grown into the current lengthy saga, I’d lost track of
who had contributed what, and when (and even the dates were a bit hazy, since
it was only expected to last a week I used references like “Wednesday” to
specify a time). Suffice to say that lots of people have helped in bringing
you this information.”
So it could be that somebody just forwarded him McMahon’s piece and it all got mixed together.
Hi, if you have other links like that (about other stories but with similar information quality), please share them ! :)
(The NZ account you shared is earlier, BTW)
Never mind…the post date is later on jwz’s, but the piece was written earlier, see above. This is why I didn’t jump straight into alleging plagarism, didn’t want to falsely impugn anyone.
something to be said for overhead power lines. no sure of the current on UK 400kv lines. but much cheaper to install and maintain.
there must have been some compelling reason to go to the trouble and expense of building it.
This line runs through the middle of Los Angeles. It would be impractical to build large overhead lines, since the city was there first.
Did anyone notice how the ground return works?
>” The ground return is the Santa Monica Bay. Down at the Scattergood Steam Plant and up in Santa Monica they have a giant copper anchors out in the bay.”
That’s cool as heck.
The greatest thing about power generation is that they are not actually producing any power. They are only creating a large deficit of electrons in a controlled manner. Here in north America the earth widely used as the return path to complete the circuit on a utility scale.
If I understood an instructor of mine correctly, in isreal you have not 3 but 4 lines that come to your house, two hots, a neutral, and a ground. All 4 conductors are complete all the way back to the power plant. They do this because the earth there is such a poor conductor there they cannot use it as a return path for neutral or count on it as a ground to ground your house properly.
They do this in the UK too, its called Protective Multiple Earthing (PME), they bond neutral to ground provided by the sheath carrying the current into the property and any phase->neutral faults immediately become phase->gnd faults back at the station and trigger the protection systems.
Because of the risk of the ground conductor being broken, you can only use PME in certain classes of permanent installs in the UK, its banned for temporary structures, petrol stations, mobile home feeds etc.
I have a friend that make similar repairs 20 years ago, when the problem was a short circuit between two lines, he sort circuit a good line and with a Wien bridge calculate the place to make the hole in the ground, the amazing thing is that the accuracy was 1-2 meters in 5 KM cable.
Funny, in EMS, ‘TMB’ is often quoted as a reason for failure, but it stands for ‘Too Many Birthdays’.
:)
Fluid filled high voltage cables have been used for over 80 years with some circuits installed in the 1930’s still operating successfully today without any deterioration.
The fluid contained within a fluid filled cable has to be kept at a positive pressure under all conditions of loading and ambient temperature change.Hydro One
Fluid filled cables rely on the presence of pressurised cable fluid to work efficiently – if the circuit suffers an oil leak the cable network is compromised.
See – How To Find A Single Drip Of Oil In Approximately 12km Of LPOF PILC 115KV Direct Buried Cable
http://www.cablejoints.co.uk/blog/article/how-to-find-a-single-drip-of-oil-in-approximately-12km-of-lpof-pilc-115kv-direct-buried-cable