Followers of alternative energy technology will remember how earlier in the year a battery container at Tesla’s Megapack Australian battery grid storage plant caught fire. Lithium ion batteries are not the easiest to extinguish once aflame, but fortunately the fire was contained to only two of the many battery containers on the site.
The regulator Energy Safe Victoria have completed their investigation into the incident, and concluded that it was caused by a coolant leak in a container which caused an electrical component failure that led to the fire. It seems that the container was in a service mode at the time so its protection systems weren’t active, and that also its alarm system was not being monitored. They have required that cooling systems should henceforth be pressure tested and inspected for leaks, and that alarm procedures should be changed for the site.
When a new technology such as large-scale battery storage is brought on-line, it is inevitable that their teething troubles will include catastrophic failures such as this one. The key comes in how those involved handle them, and for that we must give Tesla and the site’s operators credit for their co-operation with the regulators. The site’s modular design and the work of the firefighters in cooling the surrounding packs ensured that a far worse outcome was averted. Given these new procedures, it’s hoped that future installations will be safer still.
You can read our original coverage of the fire here, if you’re interested in more information.
[Main image source: CFA]
Dave Jones of EEVblog did a video on this, rightly pointed out that’s there’s absolutely no reason for these banks of cells not to be more spaced out and airgapped. Had a battery fire like this n my city a couple years back and my neighborhood was blanketed in the magic blue (toxic) smoke for a day, all to save a little space, these banks are expensive, best spring for a little extra land to protect investment going up in smoke
Very true up to a point, but the point at which making it larger is too expensive to build, maintain, and creates a less efficient store makes this a balancing act. I would have expected more space, and for the cranes on site to be the dockyard inverted U style mobile cranes that straddle their load to make a larger site easier to manage and scaling up the site hopefully cheaper but I didn’t have to spend time trying to figure all the logistics and efficiencies out…
As this fire was well contained, no real harm was done to the facility at large and the situation that caused it sounds like something which should be preventable or at least further mitagatable now its proven to be an issue I think they probably got the design just about right…
Given the consequences of a fire, it does seem that not packing the modules quite so densely might be prudent. There’s plenty of room, it’s not like this is an urban environment where land is astronomically costly.
I suggest that the cost of the land may be the cheapest part of increasing the spacing. Lengthening the interconnects may be more expensive, especially if they need cooling as well.
Overhead gantry cranes (their correct name) are very expensive too. In this use case, once the installation was complete it would sit doing nothing for most of it’s life, but would still need regular inspection and maintenance. You would not want to try to use it in an emergency only to find it had rusted up!
My experience of large industrial projects is that they are exhaustively modelled and even simulated before ground is broken. Even then they don’t always catch things, and there is always the problem of the operator not following the instructions!
Yes the overhead crane is expensive, but then so are the regular crane used around the site – all heavy industrial type things are either ancient and kinda broken or rather expensive because they are expected to survive a long hard working life – and the regular crane have very limited reach, take time to reposition etc – which all helps define the maximum practical site area, and I would think makes scaling up operations much harder.
I do agree though lots of good reasons no doubt to pick the methods they have used, and costs of making it bigger just in the interconnects just to push the units a little further apart would be huge – plus the larger the interconnects, coolant flow routes etc the more chance of failures, and lower efficiency of the system.
Maybe once this incident is fully analysed and the setup run for a while the decision will be made that more spaced out units is cost effective enough and makes working on the units easier etc. But seems like they got the design about right, and it is effectively a learning exercise for future similar installs – till it has been done you are only making educated guesses on how economic/safe/etc to run it is…
Dave Jones, is to old school in his thinking. With water cooling there is no need to have them spaced out.
I mean water-cooling is to keep them cool, not to avoid a fire spreading between banks in case of a malfunction.
So spacing them out is the better method here to avoid spreading of the fire, no?
Given that lithium and water don’t get along, is that the best option for cooling??
What? It’s not elemental lithium in these cells. Sodium reacts more violently with water than lithium but you’ll notice the sea, which is full of sodium, is actually not on fire.
Problems, problems, and even a lawsuit.
https://www.vice.com/en/article/epneq4/a-tesla-big-battery-is-getting-sued-over-power-grid-failures-in-australia
And that lawsuit wasn’t against Tesla, but the operator/owner of a battery that Tesla built.
And the lawsuit wasn’t about any hardware problem or inability, it was about the owner/operator failing to support the grid when asked.
The lawsuit has to play out, but word on the street is that the owners were making profits on the energy market by purchasing energy at low rates and putting it into the battery, then selling it at high rates during peak times, and being unwilling to provide energy into the grid at low rates when asked to prop up the grid.
There’s also some noise about ambiguous language in the contract (hence – the lawsuit) about how often and how much they would prop up the grid. My take on that issue is that it’s a disconnect between engineering terms and legal/contract terms, but again it’s a matter for lawyers under a court to decide.
Overall, the mentioned lawsuit has nothing to do with Tesla and does not have any bearing on anything Tesla built.
while true it technically has nothing to do with Tesla, in realty it has a lot to do with who/how/why they sell their products like this big battery.
ie is the battery –
1) something to be used to arbitrage the market ie by while cheap, sell why expensive.
2) is it a backup for some other types of supplies, irrespective of the current price.
In a perfect market – which electricity certainly isn’t – you could almost say they are the main thing ie when there is a power shortage prices will be very high and they will sell.
But in the real world, no so much. They might be sitting there with a full battery while there are blackouts happening, simply waiting for the price to go higher. etc etc etc. In fact, as many USA companies seem to do now, the best way for power companies seems to be a shortage of supplies, so prices are driven up..
Not so good for the consumer though..
So the legislative model for all these big batteries is going to be interesting..
I agree that this is an interesting take on electricity arbitrage.
The lawsuit is about breach of contract, so putting that (and Tesla) aside for a moment, it would appear that there’s a market for battery installations that make money on energy arbitrage.
I think that’s good for the consumer. The battery installation would be another potential seller when prices are high, which would naturally drive down prices.
I also think that’s good for the economy and the environment: there’s an incentive to build more batteries, and more batteries would drive peaker plants off the grid.
With proper management, ie not getting rid of *all* peaker plants to ensure grid safety, this would be overall a good thing for everyone.
“I think that’s good for the consumer. The battery installation would be another potential seller when prices are high, which would naturally drive down prices.”
Ehh. What?
We dont want cheap electricity, we want stable electricity.
The best thing for the USA would be to have an actual Federal Energy Regulator with legal powers. This would have prevented Texas and others from operating in a total free-for-all. The US seems to be the only developed country which not only doesn’t have a proper national grid, but also doesn’t have a national regulator. A properly regulated and managed national grid should be treated as a national security issue. Look at what has happened in the UK because a stupid short sighted government decide it was too expensive to maintain a Strategic Natural Gas reserve!
@sid1950,
I have doubts about Federal control of the national grid.
Considering how ineffective the government can be in solving problems, I see Federal control as a big “single point of failure” waiting to happen, especially regarding terrorist attacks.
“We dont want cheap electricity, we want stable electricity.”
The energy market is *explicitly designed* to manipulate generators and consumers into whatever is needed to keep the grid stable. It’s the *primary mechanism* by which stability is achieved. When demand outstrips supply, the price goes up. And vice versa, even so far as going negative when the imbalance is really bad.
Arbitrage isn’t a problem here, it’s literally the solution.
This particular lawsuit is about an arrangement where the battery owner offered an extra insurance service for the grid in the case of emergencies, got paid for it, and then didn’t provide that service when required. That’s independent and unrelated to the market mechanism.
That sounds about right, the economic viability of the installations may even depend on them playing the system like that. They always seemed like a fraud to me, at best a giant UPS with very little actual capacity to match the entire load for the time periods that are realistically required.
‘… it was caused by a coolant link …’ maybe ‘leak’ ?
Thanks! Fixed.
Service mode disabling self-protections and alarms sounds crazy. You have a battery bank that you know there is something wrong with (that’s why it is in service mode) – and then you leave alarms off for who knows how long? I can see disabling capacity and perhaps low voltage alerts, but no point disabling temperature and over-whatever-alerts.
Or in service mode because it’s not yet operational
It does sound strange. All equipment I worked with once stopped and in service mode would still trigger alarms and shutdown signals.
So, you know the battery/whatever has a problem, and you’ve taken it offline, and the repair won’t happen for an extended period of time. But you love the alarm sounding off continuously, masking any new alarms that may crop up elsewhere?
I don’t agree that we should accept, even if it is true in a few cases, that it’s inevitable that large scale new technologies teething problems might be catastrophic. Learning from past mistakes and problems might be harder than doing the let’s see if this works approach, but with our global reliance on technology in almost every aspect of life, we need to hold the technologists and the operations teams and bean counters to higher standards. In this case they were lucky it happened away from major a population centre and was contained.
“Learning from past mistakes and problems might be harder than doing the let’s see if this works approach…”
That sounds good on paper but the big problem is that in these NEW technologies, we don’t know what we don’t know. Since there are no “past mistakes” in new ventures, it’s often only in hindsight that one can look at problems that happened in other (related or not) fields and say “Oh, you should have thought that X,Y, and Z could have happened because something vaguely similar happened at some time in the past some place in the world. (It’s kind of like looking at Nostradamus’ works AFTER something happens and see if you can make it fit his writings. No one has ever used his predictions to see anything before it happens, they just say goofy things like “Oh, he predicted two mountains on fire – he ‘obviously’ meant the twin towers on 9/11”.)
I would imagine that the designers and engineers on this project would be a bit insulted at someone implying they didn’t do their homework and were just doing a “hey – let’s try this and see what happens” approach. Most of the people on this project have as some part of their job, making sure the installation is as secure, robust, and stable as possible; they’re not just throwing darts at a board to see if it’s the best solution. :-)
And, you actually address part of this but perhaps without realizing it. The actual reason this installation WAS “away from a major population [center]” was specifically because the parties knew that not every possibility could be anticipated and that building the first of *anything* is done out in the middle of nowhere to reduce certain risks. Ie: it wasn’t “luck”. :-)
Well, that’s my perspective, anyway. :-)
“That sounds good on paper but the big problem is that in these NEW technologies, we don’t know what we don’t know”
Excellent!
>Reports that say that something hasn’t happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don’t know we don’t know. And if one looks throughout the history of our country and other free countries, it is the latter category that tends to be the difficult ones.
– Donald Rumsfeld
Exactly except no matter where it happens, there are animals, and organisms that are vital to the overall planet’s biosphere. Death by a thousand cuts rings true again and agin and again. How do you think the mercury got into the fish?
Beta version, might kill people – Tesla
how do they not have monitoring equipment to prevent this. You can even buy a reservoir for your custom pc cooling loop that detects pressure drops. why features such as this would not be stock on systems this volatile and mission critical is well beyond reasoning. oh wait… it might cost 5 bucks more.
>”When a new technology such as large-scale battery storage is brought on-line, it is inevitable that their teething troubles will include catastrophic failures such as this one”
I guess that I am getting slow and stupid in my old age. Why is this ‘inevitable’?
People aren’t perfect.
Right, but “the people” didn’t democratically decide to install the batteries, it was decided by industry leaders who need them to power their businesses.
It’s not “inevitable” insofar as the batteries are legally compelled to be installed right there and in that fashion. It’s not “inevitable” to see these failures if the money to build them was allowed to break even with or exceed the return on investment. It’s not “inevitable” to always build in ways that are profitable. It’s just that labor of love isn’t respected in the business world.
There are always ways to build better, but few ways to build profitably. History has shown that the focusing on the latter is prone to failure more often.
The assertion was that catastrophic failures of new technology are inevitable, not that new technology itself or installations of new technology are inevitable.
The point is anything new and designed/run by people will be flawed in some fashion. So failures are 100% certain.
And it will be, no matter how much a labour of love it was in production there will be one thing here that out right breaks it, that little niggle there that makes it harder to use. Everyone always overlooks something, and often its only by building it you can possibly find the ‘oh how stupid’ 20-20 hindsight thing you feel you really should have spotted, but nobody did at the time.
Or the age old favourite operator error – often made because the design and documentation is such a user can easily make one tiny but crucial error and the system doesn’t have checks to catch it – look at the old mechanical signal boxes on railways, all the interlocks and the like on those aren’t there because its cheap, or required for function, they are there because without it accidents are much more likely.
The more effort that goes in to the design and construction phases is generally going to tend to better results, but even that isn’t certain till you create the engineering test cases and prototypes to prove it really works exactly as expected in the real world, the first of anything has more teething troubles because you haven’t really used it.
Like some things I’ve ended up having to fix have screws that are set soo deep in high aspect ratio holes its nearly impossible to actually get them in properly or reach them with a screwdriver – but there isn’t anything ‘wrong’ with that design, the fastener holds the bits closed just fine, its just a real pain to deal with. You read through a service manual for some things and find the only way to change a common consumable is to dismantle the whole thing basically down to the frame – again nothing much wrong with the design, but damn inconvenient. Those sort of examples are still a failure, in a case like this grid storage system it means any downtime becomes so long you need far more of the modules to make up for how many will be stuck being fixed for days that should have taken seconds or have service outages…