Black Starts: How The Grid Gets Restarted

Gripped as we are at the time of this writing by a historic heatwave, it’s hard for those of us in the western United States to picture a time when cold and ice reigned across the land. But really, it was only about four months back that another bit of freakish weather was visited across most of the country, including places ill-equipped to deal with the consequences. The now-fabled “February Freeze” left millions, mostly in Texas, scrabbling about in the dark and cold as a series of cascading engineering failures took apart their electrical grid, piece by piece, county by county.

The event has been much discussed and dissected, as an event with such far-reaching impact should be. Like much discussion these days, precious little of it is either informed or civil, and that’s not good news for those seeking to understand what happened and how to prevent it from happening again, or at least to mitigate the effects somewhat. Part of that is understandable, given the life-disrupting and often life-threatening situations the disaster forced people to suddenly face. It’s also difficult for people to discuss an event so widespread in its scope and impact — there’s just too much for anyone to wrap their head around.

To make the present discussion a little easier, we’ll be focusing on one aspect of the February grid crash that’s often bandied about but rarely explained: that the Texas grid was mere minutes away from collapsing completely, and that it would have taken weeks or months to restore had it been able to slip away. Is that really possible? Can the power grid just “go away” completely and suddenly? The answer, sadly, is yes, but thankfully a lot of thought has been put into not only preventing it from happening but also how to restart everything if it does happen, by performing what’s known as a “Black Start.”

All That Overhead

By some measures, the planet’s electrical grid is the largest and most complex machine we’ve ever built. It seems a fair claim; in the USA and Canada alone, the transmission grid has over 120,000 miles (190,000 km) of lines that stretch across the entire continent. And that’s just the system to move bulk electricity from where it’s produced to substations located near population centers; add in the millions of miles of cable that form the distribution system that connects individual customers, and you begin to see the sheer size of the system. Then consider that this system has over 10,000 generation plants, each of which has to be synchronized with all the other plants regardless of demand, and the complexity involved starts to reveal itself.

That’s one big machine. Every power plant and transmission line in the conterminous United States. Everything in this map has to work in unison, regardless of load or local conditions. Source: US Energy Information Agency

With all the equipment in place to make the power grid work over such a wide area, it’s hard to recall that this was not always the case. The North American grid of today grew bit by bit, starting mainly in the population centers of the east and midwest, electrifying first the cities and later spreading out into the rural areas. Building on existing systems allowed power companies to not only leverage the hard-won knowledge of what works and what doesn’t work when stringing up wires and connecting customers, but also to provide the power needed for their new generation facilities. The simple fact is that it takes power to make power, and that’s the heart of the black start problem.

Power generation is simple in theory, and we all learned the basics at one time or another — turn potential energy into kinetic energy to spin a magnet inside a big coil of wire. But the details are where the complexity lies. For example, in a coal power plant, milling the raw coal to the proper size to be used as fuel in the boilers takes power, as do the conveyors that feed the boilers, the actuators that control the valves, the sensors and control systems that regulate the speed of the turbines, and the switchgear that connects the generators to the grid. It takes power to make power, and a power plant can use a significant fraction of its own power. When a plant can turn out 700 megawatts or more, the overhead load needed to run — or restart — a plant can be huge.

Aside from the equipment needed to fuel and control the power plant, there’s another piece of the black start puzzle that may come into play: excitation current. Most power plants use self-excited generators, meaning a small amount of the current they produce is used to power the field coils of the generator, creating the powerful magnetic field needed to generate electricity. Once a self-excited generator spins to a stop, there’s no current available to excite the field coils. For short outages, that’s generally not a problem, as the residual magnetism of the iron in the generator’s rotor will be enough to start a weak flow of current, which will then excite the field coils and allow the generator to come up to full capacity.

Extended outages, however, may cause a rotor’s magnetic field to weaken enough that it will need a little help getting going. Black start procedures need to account for this eventuality by providing a means to “flash” the field windings with external power. The process for smaller generators is very similar, and it’s worth keeping in mind for anyone who stores a generator without actually taking it out and using it occasionally. Just keep in mind that for a power plant, it’s going to take much more than a hand drill to flash the windings.

Generators? What Generators?

Assuming an idled power plant is still connected to the transmission system, and assuming other plants within the region are still operating, the black start process is pretty simple — just take the power from the grid using switchgear and transformers the plant has for just this purpose. But in the case of a regional catastrophe — like the Texas winter storm, where condensate at natural gas wellheads and in the lines supplying power plants froze solid, pinching off fuel to dozens of operators — plants over a wide area may all go down together, making it impossible to import the power needed to restart. This condition is called “islanding”, and this is where the power plant’s black start procedure comes into play.

Utilities are understandably reluctant to share details publicly, but black starts of islanded plants are generally a cascade of operations where successively larger emergency generators are brought online, until enough power is available to black start the main generator. The process usually starts with diesel generators, which produce enough power to keep the power plant’s lights and control systems on. This will allow operators to start a larger generator, perhaps a gas turbine cogenerator, which then provides enough power to run the pumps, valves, feeds, and switchgear of one of the main generators in the plant. Once one of those is turning, the rest of the generators in the plant can start, and service can be restored.

All the procedures for black starting an islanded power plant are carefully documented, and the plans are supposed to be practiced at regular intervals to make sure everything works. Unfortunately, when crunch time came in Texas last February, and multiple power plants were islanded, the black start process was anything but smooth. One report said that nine out of thirteen generators designated as primary black start units were found to be not working, while six of the fifteen generators designated to back up the primary black start generators were also down. The problems with generators ranged from freeze damage to fuel issues, including the inability of trucks to deliver fuel oil on icy roads.

At the risk of oversimplifying a complex and far-flung series of events, for want of a set of tire chains, Texas came astonishingly close to losing their power not just for a couple of days, but for weeks or possibly months. It didn’t happen in this case, but only just barely and by several strokes of luck. We’ve no doubt that a lot of engineering skill and ingenuity went into getting the reluctant black start generators back online, too, so hats off to everyone who worked hard to avert the catastrophe. Hopefully, this will serve as a wake-up call, and that some thought will be put into how to better engineer the whole black start system, not only in Texas but at every power plant in the world.

96 thoughts on “Black Starts: How The Grid Gets Restarted

  1. I’ve never seen an article that mentioned and showed exciting a generator to get it to generate again…..

    I’ve done it once myself and taught an NPC how a few weeks ago from FB marketplace

  2. Wow, I knew it had been bad. Didn’t know it was actually that bad though, I’d assumed the on site backups or at least enough of them to make a difference would work and the state wide blackout was mostly because the powerstations were not exporting anything till they got back up to a true stable state….

    Your emergency restart fuel really should already be on site, in large enough quantities you won’t need a restock to restart a pair of primary generator, needing to truck it in seems like a major flaw.. But then JIT seems to be the rule of thumb for everything now, lets not have any stocks on site everything should be delivered just in time, as that can never go horrible wrong…

    That freeze damaged things isn’t much of a surprise really, nowhere puts in the effort to rugidise their infrastructure for events that just don’t happen there (or at least happen there historically something like 1 day in every 100 years) – its not worth it. Though with how wild the climate has become historical normal really should be thrown out and everything overbuilt to handle all the weather can throw at you.

    1. Fuel has finite lifetime, so you have to rotate it somehow—can be done, of course, but I bet there were places that decided that the risk-to-benefit was low enough that they didn’t do it.
      During Katrina, the phone network survived the power outages, but only by about a week—the backup generators kicked in, but they could not be resupplied with fuel because of flooding. This will be a problem for every major utility disruption, because everyone makes this standard assumption that backup fuel will be available.
      In general, most of our systems are designed to have zero or one backup mechanisms—but that means that two or more system failures will take them out. There’s a nice illustration of that: a swiss-cheese paradigm, where we’re protected from disaster by an armor made of moving slices of swiss cheese. Every once in a time the holes align, and we’re vulnerable.

      1. That explanation stacks up for surprise winter storms but February in Texas was anything but. Operators had a week’s notice to get their ducks in a row, but instead chose to sit on their asses. As a Texan I’m not sure whether I’m more upset by the complete incompetence, the lack of any sort of accountability or justice for those responsible, or my own impotence to enact any sort of change.

        I’ve settled for just assuming the grid is going to continue to decay as profits are prioritized over essential services, trying to establish an effective enough solar & battery backup system that I just won’t have to care.

        1. I worked in major electric utilities (G,T&D) for 15 years as an engineer and economist. The simple fact of the matter is that your assertions are 100% correct, even when applied to cooperatives. Public energy policy designed by lawyers, CPA’s and ignorant politicians have led to exclusively focusing on renewables over RELIABLE generation sources, adequate transmission capacity and interconnections between different grids, and incentivizing construction of micro scale distributed generation in the areas that have the highest level of demand; these bad policies have created a greater problem than previously existed. Combine this with over reliance on internet technology to run SCADA with, and backdoors into all the back office IT systems, and the system is even less stable than when manual cutovers and load transfers occurred. Automation is great…until it’s not.

          Moreover, I’ve witnessed corporate policies that Focus exclusively on stupid stuff such as diversity hires, not hiring the best employees, and promoting morons to fill political quotas. Those PC policies have les us to the current state: broods of politically focused, woke retards running the utilities. 90% The young engineers and analysts that I have hired or managed knew nothing about the job, system dynamics, or how to solve a problem as a systems engineer; but man, did they know how to be political and perfidious to advance their careers. They are WOKE, while the ratepayers go BROKE.

          At the last utility I worked at, which is in a state that directly abuts Texas, we had the Board of Directors tell the former CEO and wimpy VP of transmission that we were going to be fired if we ever mentioned being a dual peaking utility to them again. I didn’t care and explicitly stated that failure to plan for large adverse weather and winter temperature excursions outside of normal, would lead to a -100% hit to margins at some point within the next 10 years, meaning that there would be 0% profit that year, or even going into the red by a minimum factor of 100%. Margins for 2021 actually went -500% due to a polar vortex (a “new normal” that is a result of jet stream changes driven by changes in the Pacific’s northern current and cold temperatures not experienced in 50-100 years. It’s all driven by the cycles of the Sun). Over $100,000,000 in spot had to be capitalized and collected over 9 months. People’s electric energy bills went up 300% or more.

          I’m an award winning published researcher and forecasting expert. Working with people who had only worked in one utility that never had any kind of problems due to consistent weather for 30 years, And people that were more concerned about their jobs over doing the right thing (P.E. Engineers have a MORAL and LEGAL obligation to do the right thing, even if it costs them their job!), meant that The political class were the ones promoted into management positions, instead of the most competent. This Stands true in the army and rest of the military in many cases, and it stands true in business: the vast majority of leadership will possess a level of incompetence that is inversely related to the level of sycophantic ass-kissing each leader exhibits. The most competent potential leadership either votes with their feet or are fired. I repeatedly and unashamedly spoke up and was fired, so be it. At the end of the day I have my integrity. I’ve only seen one company, the G&T statewide in NC, have effective and skilled leadership that resulted in MAJOR improvements and resiliency. They were trusted advisers to their member owners. Every other co-op I’ve worked for, and IOU’s I’ve consulted for, were run by totally incompetent boobs.

          Just remember, a backup system is only as good as the deliverability of fuel to that system. Batteries that have adequate capacity need a system capable of charging them quickly to be useful.

    2. El Paso ran into this in the early 2000’s. Since my folks live there, they gave me the blow-by-blow. In essence, the local electrical generation facilities were rated down to a prolonged temperature in the mid 30’s fahrenheit. When a cold snap hit, it froze the fuel inlet valves and a lot of other equipment. It also caused a pressure drop in the gas supply to the point that the online plants couldn’t operate and they too dropped off.

      At the end of the day, they got restarted and El Paso, which has a deal with Arizona to get power from a nuclear plant, didn’t go dark, but they were dependent on a single transmission line; paging Mr. Murphy.

      So El Paso sunk several billion into upgrading their power infrastructure to -15F or -20F, something like that anyway, and putting in a redundant transmission lines to external sources.

      So while the rest of Texas was minutes away from catastrophe in this latest incident, El Paso was like; “What was that noise?”

  3. The main reason larger thermic power plants can’t black start is that those turbines will be jammed at low temperatures and can’t be turned. It needs to be preheated until it lengthend enough to turn freely. Thus it is needed to have at least some working feed water pump(s), working boilers etc etc.

  4. “The North American grid of today grew bit by bit, starting mainly in the population centers of the east and midwest,”

    Ahh, the good old days before we had those westerners out in Enron land to destabilize our network. At least Texas isn’t connected well enough to leach much.

    1. The modern 3-phase grid started in California with Mill Creek #1 in 1893. It’s still operating.

      Until 1934, the exciter power was provided by DC generators, each with its own Pelton wheel for power. A few more moving parts, but a very straightforward black start sequence.

      1. Water powered exciters are a pain from a grid stability perspective since they cannot react as fast as modern static exciters.

        I worked in a plant that still had a rotating exciter and the field current was adjusted coarsely by speeding up the DC generator and it was fine turned with a huge oil cooled variable resistor. It had to go after regulatory changes from the 2003 blackout.

  5. Some systems are lucky enough to have hydroelectric generators on their system. These can be used for black start by manually opening the wicket gates that control the water flow to the turbines. That spins up the generator and from the residual magnetism in the rotor, the voltage slowly builds up. The output of the generator is used power up other harder to start stations.

        1. A mix of wind and solar with battery backup would keep you safe. The Scandinavian countries have lots of wind, and they get similar conditions to the freeze in Texas. Mostly they keep going. The Scottish generators are building pumped storage and large battery centres to backup their solar and wind. There plan is to have enough storage to keep the wind turbines running when demand is low to charge up the storage.

          A more interesting happening is that several city and towns authorities in the UK are buying into battery storage to give them an emergency backup, and to iron out the ups and downs in their demand. It turns out that buy swapping all their old sodium vapour street lamps for LEDs, they are saving lots on their bills. Since the infrastructure for the power distribution is still there, they are partnering with utilities to install kerbside charging units for EVs! A friend who lives in a London suburb says that over the last 2 years, all 12 street lamps in his street have acquired charging stations. The chargers use the spare capacity built into the street lamp infrastructure, the utility and the local authority gain a new revenue stream, and residents with no off-street parking have somewhere to charge their EV! Wins all round.

          In South Australia, a combination of a very large Tesla battery farm, and lots of household generation and storage, means they now have enough backup to run the whole state for a couple of hours if ALL of the generation goes down. The future is now. Texas just needs to get with the programme.

          1. The scandinavian countries also have massive hydroelectric reserves, and nuclear power. They can drop all their wind and solar power off the grid and not notice it.

            > the spare capacity built into the street lamp infrastructure

            HPS and LED street lamps get similar luminous efficacy, but the LED units are about half the power because of heat management concerns. For each HPS bulb you basically need two LED bulbs. The savings they get, or think they’re getting, are based on the longer lifespan of LEDs which is supposed to be twice that of HPS – if you believe the sales pitch.

            If the street lamp infrastructure is old, the “spare capacity” probably dates back all the way to ancient incandescent and low pressure sodium lamps, besides, it’s not being used at daytime anyhow.

      1. For a large power plant the excitation power is pretty minor compared to other loads. The plant I work at needs about 40 Megawatts for the heat transport pumps and boiler feedwater systems but only a few Mw for the exciter.

    1. I worked at a (rare!) black start capable Hydroelectric plant. It was part of a network of plants and was the only one capable of black start.

      The reason it was capable of black starting was that it had a mechanical turbine governor (Flyballs) and manual valves for the penstocks.

      If the residual magnetism in the rotor was not enough to get some output you could manually flash the field with a large jump starter pack. You would literally watch for the lights in the control room to come on to know that you had successfully flashed the rotor.

        1. There were procedures in place to do it but there were no regular tests other than checking that the portable equipment was ready to do it.

          Other than having no lights and having to flash the field it would not be much different than a normal startup for this old plant.

          The only time this would have been used would be for a severe loss of grid and this area would need to be islanded from the bigger sections of the grid. Even in the 2003 blackout it did not need to be black started.

          Working on really old hydroelectric plants was an interesting summer job. I work at a plant now that has standby generators capable of more output than some of the hydro stations.

  6. In my experience, if there’s a near miss (in any system) the money WON’T be spent to rectify those problems found, lessons are not learned, and you’ll bumble along until the proper disaster happens. Seen time and time again in public infrastructure (in the UK as well).

  7. Buy some plows and salt trucks. There is no excuse. NM does it even in our southeastern area. Arkansas does, Oklahoma does, but somehow TX has an abysmal amount of snow removal equipment even where it snows yearly. The sympathy is simply not there from your neighbors who you take for granted.

    1. The thing is that we just shut down and hang out with our families for the ONE week a year that there is snow on the ground. This storm in texas was a once in 30yr storm according to the old timers, might see it once or twice in your lifetime.

      People have to learn to be self sufficient again. Natural gas never shut off, so we heated our house with it using a fireplace while the power was down. If not, I would’ve broken out the propane catalytic heaters. Many around here had natural gas generators. BBQ grills and camp stoves were pressed into service for making food. My neighbor heated their house with wood in their fireplace. It’s time to wake up and take some responsibility for yourself and your family, no one owes you anything!

        1. — So the consumers have to spend money to be prepared for the outage
          Yes!

          — so that utility providers can be unprepared and save a a few bucks?
          Not exactly

          It very well may be that preparing for this particular eventuality is something that a utility provider SHOULD be expected to do. But in a state with an island grid, disconnected because they are so against regulation you would be a fool to expect this. Why do you think they don’t want to be regulated? Its so they don’t have to spend money on stuff like that.

          But even if the utilities were preparing for everything one would reasonably expect them to prepare for… expect the unexpected! Shit happens. If you don’t prepare it’s you freezing in your own home as the pipes burst not some utility exec. They probably have enough money to pay 1000 people to stand in their houses rubbing their hands together to keep it warm until the power is back anyway.

          Take care of yourself.

        2. I can imagine a movie where this is the last quote of a guy who is freezing to death at his home because he was buying a new smartphone every year but had no alternative heat source for his place ;)

        3. How about a Business Case Analysis comparing the two costs? I either: A) Spend X dollars to have a backup for when I lose electricity. B) Spend Y dollars every year paying the utility to tack another 9 onto the reliability.

          I’m honestly not sure which is cheaper.

          1. I’m sure which will become more expensive though.

            Everyone buys as much backup as they need, or the utility company adding more and more redundancy everywhere it’s not needed because they get to justify higher rates that way.

      1. Really? A once per 30 year event is also a once per lifetime event? You Texans really need to work on your life expectancy! It’s must be like the early industrial revolution over there still.

        Get some exercise, get your shots, eat some vegetables and demand all those oil companies dispose of their waste responsibly. You ought to be able to expect to see 2 of these in a lifetime at least, quite possibly 3!

      2. Propane heaters or even a rarely used natural gas furnace do not really count as emergency preparation unless you also have a CO alarm and a way to power it.

        Otherwise it might just be another way to die. Although… probably a more pleasant way to go than freezing. But still!

  8. Had a really inexpensive generator used for powering the beer fridge at motor racing events. Since it had an engine, it got meddled with, though not for more power. The fridge and all our phones could be charged with AC between 110V and 240V, so decided to drop the voltage (and frequency) a bit to see if fuel could be saved – were not using anything like fill power.

    Vaguely from memory, before dropping the voltage/frequency it lasted about 8 hours on a tank. After it was up to 11. The beer was still cold, but it wouldn’t self excite without reaching full speed. So to “flash” it (a term never used) we poked the governor with a stick to make it rev up a bit and the lights come on.

    Can’t remember the voltage/frequency, it was whatever still worked but nowhere near 110V. Maybe 180 or so.

    Running a little slower made it sound a bit more pleasant too. Though being near the track at Le Mans drowned it out anyhow.

  9. This is another fairly positive aspect of next-gen nuclear power. A plant designed to automatically SCRAM and passively manage decay heat ought to also be able to be restarted with minimal external resources it seems to me.

      1. Yeah, but that wasn’t a next-gen design. The problem with PWRs and BWRs is that they can’t handle their decay heat without active cooling.

        We’ve known that that’s a problem AND known how to fix it for over 50 years.

      2. I just love it when the anti-nuke crowd uses an event regarding a plant that was built on the shore, in an active fault area where tsunamis are known to be a thing as their example.

        Just don’t build them in stupid places.

          1. Remember that TEPCO built the plant too low down on the shore to save on plumbing costs, violating their own approved safety designs, and then paying the officials to approve of it anyways.

    1. That’s not the case, unfortunately. The difference between the passive cooling systems and the systems required to get usable power out of it are substantial, on the order of megawatts. With any steam plant, you cannot turn the turbine at rated speed without condenser vacuum, and you can’t achieve a stable vacuum without pumped cooling water. You also need condensate and feed pumps to maintain reactor/stream generator water level.

    2. Almost all, steam cycle plants (coal, natual gas, nuclear) require a large amount of power to restart. The Reactor Coolant Pumps for Nuclear Plants, Feed Water Pumps to maintain steam drum water levels, and Circulating Water pumps for moving water through the condenser are massive. One 600 MW Nuclear plant needs about 30 MW for these pumps.

  10. Every year here at my company in MT, our dispatchers go through black start training (as was mentioned in article) and simulate bringing the systems back on line. Hope we never see it… But with more ‘un-reliable’ energy in the system (and more every year) the likely hood is greater for brown, then possibly black outs…. Really need more reliable coal, gas, water, nuclear energy available as energy consumption grows… But politically, its not in ‘vogue’ … Especially now.

    1. no, what we actually really need is a functioning planet/climate. What you fail to mention is whilst it is unreliable it is almost pollution clean and that being the crucial point. There is little point having the lights on, on a dead planet now is there.

      1. The planet will be just fine. It has seen greater calamities. What so many climatemongers fail to acknowledge is that this whole “climate change” meme is about maintaining civilization, not the planet.

        1. Or nuclear, though thanks to the 1979 atomic fart in Pennsylvania, that is probably off the table until people start realizing their options are nuclear power, devastating global warming, or a world population of 500M in grass huts.

  11. “…scrabbling about in the dark and cold…”
    When the power is out and there isn’t much to do, I’m sure a game or two of Scrabble might be a fun way to pass the time. Otherwise, I might be _scrambling_ about trying to find a way to stay warm! ;^)

    1. Although I suppose the activity of scrabblling – scratching or groping around with one’s fingers to find, collect, or hold on to something – is also applicable. :^)

      Take your pick!

  12. The near failure of the Texas power grid can’t be fixed if the primary issue: Climate breakdown isn’t considered. It’s inevitable that solutions will be wasting money addressing the wrong questions and failing to anticipate changes which are obvious to the vast majority of climate scientists.

    And yet this post, in a forum that prides itself on getting to grips with technical issues, completely fails to even mention the word ‘climate’. In turn, because it fails to consider the problem from the viewpoint of climate change, fails to consider that Texas shouldn’t be looking for ways to make fossil fuel plants more resilient in the long-term, when it should be recognising that the critical issue is to decarbonise as fast as possible.

    For example, the major issue being addressed here is how to restart large fossil fuel power plants, but this kind of problem is significantly reduced when electricity generation is distributed amongst a far larger number of renewable sources.

    And again, the issue of why:

    ” condensate at natural gas wellheads and in the lines supplying power plants froze solid, pinching off fuel to dozens of operators”

    Is not addressed, because at the heart of it is that climate destabilisation means Texas will have to face the same problems increasingly frequently. It will have to do the same for renewable sources so that they’re more resilient to weather extremes as wind turbines and solar farms are in other climates. That is power grid engineers in Texas need to consider more radical solutions and fast.

    One of the obvious solutions is of course to massively increase the amount of renewable energy generation and support domestic renewables. Its distributed nature is inherently more reliable: during the blackouts Texas homes with solar power were able to keep at least some of the lights on when their neighbors could not, despite the snow.

    This won’t mean that potential blackout issues won’t occur in the future, for example, the UK power failure on August 9, 2019 due to a gas turbine outage, which in turn temporarily brought down an off-shore wind farm resulted in an inquiry that sought to better manage similar outages, but importantly, while maintaining the continuing decarbonisation of the power grid itself.

    https://www.sciencemediacentre.org/expert-reaction-to-ofgem-report-on-the-power-outage-of-9-august-2019/

    1. Well spoken. If power outage ever strikes my home, our small solar generator will at least keep our lights on and the telephone system running. And on good days it will allow for the coffee machine and the heat pump to be operated.

      1. Be aware that the average solar inverter shuts down once 50/60hz mains has gone. The average solar inverter does not run without mains available. Certain specific ones can run like that, but most don’t.
        This is to prevent backfeeding of the mains if power is deliberately shut off.

        1. and I have to wonder which shortsighted authority mandated that little feature?! I would have thought with the addition of a few dollars worth of isolation breaker and a standalone mode, most of solar enabled homes in Texas would have had some of their heating running and lights on at least during the daylight hours. Pure idiocy IMHO.

          1. Without a reasonably large bank of batteries, you can’t run a solar setup in “island mode” anyways – it would brown out repeatedly with passing clouds and break the equipment.

            Most of the home solar systems up to date were built simply as subsidy mills, intended for selling the power to the grid to collect ITC/PTC and Net Metering, plus any state/county level incentive that requires you to sell the power. When the grid goes down, the meter doesn’t run, so there’s no need to keep the panels on.

          2. It’s a safety thing. If a grid-connected solar or wind facility energized a powerline that’s supposed to be isolated, people working on the line could die. I’m a former wind power plant operator: it’s not something that people in the industry disagree with at all.

        2. really depends on the design…most with batteries have a “backup power” outlet, which will give line voltage at the correct frequency even when mains is not present. This is usually wired to fridges and freezers or for powering the furnace control and pumps.
          Still, if I owned a house with a solar setup, I’d definitely invest into an approved isolator setup, so the whole house can be switched to island power…

          1. I’ve been wondering if its worth upping the battery capacity and having a limited DC only wiring run through the house from out little solar set up – get more out of the batteries if you don’t waste lots of energy making it pretend to be 240vac to just transform it down, probably to less than the 12vdc the batteries would likely be – why go expensive when 12v lead acid is durable and suitable…

            Currently do have an automatic cutoff of some sort – so we have battery power in theory if the grid power ever goes down its automatically in island mode – but that was all fitted by the energy company to meet the rules never actually needed it (actually don’t know exactly how it functions) . And beyond the test when it was fitted its not be tested, probably should test it every now and then just to be sure it still works (but even if it doesn’t there is still power available from the battery directly, and in the UK power outages are nearly unthinkable for a very long time now, and look likely to remain so for a while still with how much redundancy the grid has and the ever growing interconnects to Europe and the French Nukes etc, so its not like it should be a super high priority)

    2. In 1989 there was a total gas curtailment.
      The natural gas plants stayed on by burning fuel oil as was a requirement under the PUCT rules for a regulated utility.
      Utilities included the on site fuel storage in their rate base.
      No regulation, no TV.

    3. While you’re right to a point, the the implementation of renewables wouldn’t have saved Texas from this incident. The Texas grid is comprised of 20-30% wind power and due to severe icing on the blades, were forced to pull turbine out of commission throughout the week. Solar panels are awesome but have the inherent weakness of being covered with snow and the inefficiently of collecting diffused light in shitty weather. Often the arrays up north could go days with ice and snow build up before it melts enough to allow sunlight to hit the cells and it warms up enough to shed the rest of the ice.

      I’m all for renewables, I got my degree in renewable energy and have worked in the industry for years. However, in terms of the national grid, it provides many issues for grid reliability and engineering. I agree distributed forms of energy are needed and renewables pair nicely with that, but we can’t rely on them alone. Alternatives such as nuclear, biomass, natural gas, and even coal have to be considered for grid security and stability. Sure some of the more harmful ones could be scaled back, but there will be situations when a 100MW plant that only has a footprint of a couple thousand square feet will make more sense than a solar array that takes up hundreds of acres for the same capacity of production.

      It’s a complex problem with no right answer. And I agree that climate is definitely an underlying issue that contributes to the problem. However, it’s a PART of the problem, and we can’t throw away all components of our grid within 5 years and expect things to continue smoothly In our day to day lives. It took the better part of a Century for us to build what we have now.

      1. renewables need either backup or storage, otherwise even 40% active power is pushing your luck. Both are ridiculously expensive and come with engineering challenges.

          1. Liquid metal batteries suffer from the same “black start” issues that the article describes, because the electrolyte has to be heated before use.

            Liquid metal batteries might work for grid storage, although it would probably depend how much maintenance is required for large installations.

        1. South Australia (population 2M) is at 60% RE average with 100% excursions on quiet days. They have a tiny amount of storage and that is plenty sufficient to stabilize the grid.

      2. My next door neighbor has solar panels on his roof. Due to the gap between the roof and the panels, there was an inch or so of snow on them that would not melt. I’m sure those weren’t the only solar panels to get covered up.

        A big problem with the freeze was that the outside temperature did not ever go above freezing for over three days. In my 40 years of living in the area, there was only one other big snow event here in San Antonio, in 1985. But I don’t think the temperature stayed below freezing for even 24 hours in a row. That was the main problem, that nothing was able to thaw, except by sunlight.

          1. Solar power is easier to implement as decentralized systems. The grid is maintained to establish political power over the peple. Soon, your social credit score will determine how much power you get to use. The average sized house in the usa is huge. If you reduce the size by 500 square feet you just paid for an independent power system. I have been off grid since 2015 and have never regretted the decision.

  13. Great article. I’ve long been curious what the process of restoring power would look like in the event of a coronal mass ejection that took power generation offline essentially nationwide. How would the country go about black starting thousands of islanded power stations with the grid totally collapsed? Damage to high voltage transformers might isolate some systems creating additional complications. I could also see issues with matching phase/frequency and the potential for brownouts to trip new generators as they’re brought on-line due to the load demands with no other power stations assisting.

    1. I believe that since the 1989 solar storm that blacked out the province of Quebec and was felt down the northeastern seaboard that many people if nothing most utilities have installed equipment on their transformers to manage the large zero sequence currents (ie DC current) that solar stormsccreate on transmission lines

      1. I believe many of the long distance transmission lines are moving towards DC as well, which means the frequency can be shifted, and eliminates the risk of being out of phase when connected.

    2. but wouldn’t there be auto phase matching setups in most large substations? Ie installations could be allowed to island operate and when the phases alligned as in in-sync, breakers thrown? After all that circuitry is in every grid connected inverter out there so not rocket material… granted the currents and voltages are huge but there are methods to deal with that like mechanical withdrawal of transformer core or something like that.

  14. wow! i had no idea induction motors could be used as generators! thanks

    this inspired me to look into how induction motors are “asynchronous” — they have to lag (or lead, for generation) the electrical field — if they were exactly synchronous somehow then the rotor’s field would collapse. and now i finally know why my dehumidifier has a beat note! it’s got the 60Hz tone from the mains power, and a 59Hz tone from the motor lagging that rotating field! 1Hz beat note.

    i’ve known for yeaars that the motor was somehow lagging compared to synchronous i just never had a clue how that was possible.

  15. I can’t speak for others, but our black start units are designed to start completely independent of any external power source other than a capable battery bank. All critical components are dependent only on DC power. It is not a complicated process but startup should be initiated immediately upon loss of connection with the external grid. After the unit is started and operating in Isochronous (island) mode, the area and load capacity of the island is expanded by larger units starting in droop mode as they come online (being synchronized to the black start unit’s frequency). When connection to the external grid is available again, the actual transition to establishing this connection becomes the next hurdle. This sometimes a difficult process to include in annual testing since it requires temporarily connecting a generator running in Isoch mode to the grid. This is where I start scratching my head.

  16. If you mean the steam turbines – I would guess that would be because of thermal expansion. They should still be able to turn when cold – but they’re going to heat up quickly when fed steam. The rotor will heat up more quickly than the stator, and if the temperature differential becomes too much then there will be interference which could begin a very expensive internal lathing operation.

    ==

    We run a couple of GE 7EA gas turbines that drive generators (~80MW units).
    Not quite the same animal, but with a similarish aspect.

    The requirement on these turbines is to get them ‘on ratchet’ within ~15 minutes of rotation stopping. The ratchet is a hydraulic actuator that that turns it around slowly with a ratcheting mechanism – a few degrees at a time.
    If this time window is achieved, the turbine can be restarted at any time.

    If not rotated, the rotor doesn’t cool evenly and it can bow. Trying to rotate the unit with the rotor bowed can cause damage to it – with compressor blades on the rotor interfering with blades on the stator. There could also be excessive imbalance trying to run with the rotor bowed – causing the unit to trip itself off on vibration.

    The solution to a bowed rotor is to wait for it to cool down. By memory, the default recommendation for these is more than 12 hours.

    Actually turning the turbine to get it going is the biggest starting load – a gas turbine needs to pull in and compress a lot of air: the starting motor for these takes up to a few MW by itself.

    A gas turbine is probably the best case (minimal auxiliaries) for being able to black start a generator that is larger than you’ll typically see a diesel engine driving (visualize the engine in a train locomotive – might get 3MW out of those).

    Cogeneration units may have a gas turbine up front – but that doesn’t necessarily make them good candidates for a black start. The HRSG that is downstream of the gas turbine will be destroyed by the exhaust gases if there is no water provided to cool it. The most flexibility for doing a black start is to install a diverter stack – which allows dumping exhaust gas outside instead of forcing it through the HRSG.

    1. Steam turbines are even more problematic, especially the big ones easily reach 100t of rotating mass. You need steam to heat them and running the boilers takes single digits of MW, not to mention that your coal pile which is the size of multi-store condo will last you only a few days, railroads will not work without power (signalling equipment and switches).
      It gets even more exciting if it’s a big nuke plant…using the kinetic energy of the still running turbine didn’t turn out to be such a great idea (reactor 4, Chernobyl)…

  17. Another thing that Black Start plans have to account for is generator hydrogen. Most large generators (like over 100 MW) use pressurized hydrogen gas as a cooling medium. This gas is kept safely inside the generator using a oil seal at the generator’s shaft where it connects to the turbine at one end and typically a generator exciter on the other. Normally this oil seal is maintained by a pump with multiple back-ups. If a large plant loses it’s off-site power (such as in a large scale black out), DC pumps with a battery back-up kick in.

    These DC oil pumps supply the hydrogen seal oil and bearing seal oil in a blackout. As you can imagine its pretty important to have bearing oil while these large turbine/generators coast down and as another poster mentioned the turbines have to be maintained on a very slow speed turning gear afterwards to keep the turbine from bowing. Since the batteries that keep the DC pumps running have a limited life span, eventually the plant operators have to purge the generator hydrogen out with CO2, then the CO2 with air.

    Most plants have a decent amount of hydrogen on site to reverse this process and re-pressurize the generator but a lack of stored hydrogen has on kept units offline. It is also a fairly slow process since care must be taken to not have both hydrogen and oxygen in the generator at the same time.

    Getting station auxiliary power restored is the first step in most black start plans for this reason

  18. One item which didn’t seem to get a mention is what happened as the frequency dropped. I’m not sure whether it is the case in the US/Texas, but in the UK, the Electricity System Operator (ESO) (Nee National Grid) impose a requirement on the distribution network operators (DNOs) to effect low frequency demand disconnection (LFDD).

    This sets nine frequency set points of decreasing frequency at which the DNO must disconnect a given percentage of the total demand at that time. These bands are decided ahead of time by the DNO, and are communicated to the ESO via regulatory submission.

    It is implemented by the use of bespoke LFDD relays, generally shedding blocks of demand at 33kV/66kV. By shedding demand in this manner aims to reduce the demand to balance the depleted generation.

    It has operated twice in recent years, 2009 and 2019, both times due to a frequency excursion due to simultaneous loss of two large sets (from memory in 2009 a large coal set in Scotland, and a nuclear set in Suffolk, the 2019 event was large CCGT set in Bedfordshire, and a large offshore wind farm) this alone shows how the generation mix has changed in 10 years.

    But in both cases LFDD saved the day, preventing black start.

  19. Flywheels, one at each power station, large enough to get it up to speed again.
    I think it is a relatively cheap way to store energy.

    Would it be feasible to restart a generator?

    1. There had been research as to whether the generator / unpowered turbine could be used in a defunct steam set to act as a rotary reactive compensator and a contribution to network inertia. I don’t think it has advanced that far up to this point. Interesting to know if anyone has experience of this type of thing.

      1. Certainly leaving a synchronous generator unloaded and over or under excited has been used for years to control reactive power on the grid. But I haven’t heard about using it solely for inertia. One of the problems with wind and especially solar has been the lack of inertia which helps system stability.

  20. There are ppl that actually lacked this knowledge and finally learned it (or just finally heard it), here?

    I do not wish to diss ppl…. but all of us working to expand the knowledge of ourselves and other kinda now need a name for this kind of “failure to inform ourselves”.

  21. The real reason the grid is unreliable in Texas is because the political leaders (governor, lt. governor and attorney general) want it that way. Texas has hundreds of world class engineering companies that routinely design generation plants that work reliably in places like Northern Alaska.
    This won’t be fixed until 3 heads roll. Their supporters make billions of dollars speculating on spot pricing during “emergencies” and after the Texas freeze ERCOT actually apologized on a conference call to one of the speculators because their profit windfall ($ billions to be burdened onto the ratepayers who actually had *outages” during the freeze) hadn’t been approved quickly enough.
    The latest plan by the governor is to make renewable energy providers pay for fossil fuel sources when the sun doesn’t shine or the wind doesn’t blow.

    “What is here is not a serious or prudent plan for improving the grid,” Daniel Cohan, an associate professor of civil and environmental engineering at Rice University, said in an interview Tuesday. “It’s more of a political job favoring some [energy] sources over others. For Texans to have a more reliable power supply, we need clearer thinking that makes the best of all the sources we have.”

    Imagine if those billions went to hardening the grid instead of to fuel profiteering. In today’s world political psychobabble gets campaign contributions and ignoring deficiencies that cost hundreds of Texas lives has absolutely no downside.
    It’s why the emergency special legislative session didn’t include energy issues but did include why it is mandatory to limit the number for 5 million Harris county residents to 1 ballot dropbox.
    Nero is fiddling while Rome burns. By the way, a few weeks ago we were told to reduce our thermostats to avoid another brownout by ERCOT. (It was about 80 degrees outside). Presumably they were speculating for price spikes on the spot market again.
    Texas needs to nationalize the power grid and start a real investigation on corruption and the power grid failures and follow the money.

  22. Yes and no. The technology exists and is called autosync. It works a bit differently than inverters. Inverters use autosync with a phase locked loop to tune into the power frequency, and they use phase angle/voltage levels to control power flow out of the inverter. This is transistor controlled and very fast due to the control system in the inverter.

    For substations with autosync enabled, phase angle difference is measured, and a relay will close a breaker if the angles are close enough automatically (~20 degrees). For manual resyncing, the operator relies on synchrocheck (logic that won’t close a large angle difference). Otherwise, you can have extreme power flow changes on closing the breaker (not good).

    It’s atypical to use synchronizing relays at every station. Normally, points are chosen for synchronization (either automatic or manual) to plan islands that are mostly balanced (generation and load). Otherwise, you could have a situation where two areas sync but are unstable. This would lead to another cascade on instability/generators tripping. Your goal as the operator for blackstart is to build your own islands from your own blackstart units, reconnect to the interconnection, and/or synch your islands back together. Or you can start your system off of the interconnection (if it’s there). It’s normally some combination of the two.

  23. I live on a med/smallish islanded grid serving maybe 2.5 million people over a relatively large area. The capacitive charging current required to bring up the transmission lines on it’s own is 2 x GE Frame 5s which are good for around 25MW each, before you even look at auxilliary power for any other generation.

Leave a Reply to Greg ACancel reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.