The 2003 Northeast Blackout And The Harsh Lessons Of Grid Failures

The grid failure in 2003 which reverted much of the eastern US and Canada back to a pre-electrification era may be rather memorable, yet it was not the first time that a national, or even international power grid failed. Nor is it likely that it will be the last. In August of 2023 we mark the 20th anniversary of this blackout which left many people without electricity for up to three days, while costing dozens of  people their lives. This raises the question of what lessons we learned from this event since then.

Although damage to transmission lines and related infrastructure is a big cause of power outages – especially in countries where overhead wiring is the norm – the most serious blackouts involve the large-scale desynchronization of the grid, to the point where generators shutdown to protect themselves. Bringing the grid back from such a complete blackout can take hours to days, as sections of the grid are reconnected after a cascade scenario as seen with the 2003 blackout, or the rather similar 1965 blackout which affected nearly the same region.

With how much more modern society relies today on constant access to electrical power than it did twenty, let alone fifty-eight years ago, exactly how afraid should we be of another, possibly worse blackout?

Continue reading “The 2003 Northeast Blackout And The Harsh Lessons Of Grid Failures”

Cascade Failures, Computer Problems, And Ohms Law: Understanding The Northeast Blackout Of 2003

We’ve all experienced power outages of some kind, be it a breaker tripping at an inconvenient time to a storm causing a lack of separation between a tree and a power line. The impact is generally localized and rarely is there a loss of life, though it can happen. But in the video below the break, [Grady] of Practical Engineering breaks down the Northeast Blackout of 2003, the largest power failure ever experienced in North America. Power was out for days in some cases, and almost 100 deaths were attributed to the loss of electricity.

[Grady] goes into a good amount of detail regarding the monitoring systems, software simulation, and contingency planning that goes into operating a large scale power grid. The video explains how inductive loads cause reactance and how the effect exacerbated an already complex problem. Don’t know what inductive loads and reactance are? That’s okay, the video explains it quite well, and it gives an excellent basis for understanding AC electronics and even RF electronic theories surrounding inductance, capacitance, and reactance.

So, what caused the actual outage? The complex cascade failure is explained step by step, and the video is certainly worth the watch, even if you’re already familiar with the event.

It would be irresponsible to bring up the 2003 outage without talking about the Texas ERCOT outages just one year ago– an article whose comments section nearly caused a blackout at the Hackaday Data Center!

Continue reading “Cascade Failures, Computer Problems, And Ohms Law: Understanding The Northeast Blackout Of 2003”

What’s The Deal With Rolling Blackouts In California’s Power Grid?

A heat wave spreading across a large portion of the west coast of the United States is not surprising for this time of year, but the frequency and severity of these heat waves have been getting worse in recent years as the side effects from climate change become more obvious. In response to this, the grid operators in California have instituted limited rolling blackouts as electricity demand ramps up.

This isn’t California’s first run-in with elective blackouts, either. The electrical grid in California is particularly prone to issues like this, both from engineering issues and from other less obvious problems as well.

Continue reading “What’s The Deal With Rolling Blackouts In California’s Power Grid?”

Hackaday Links Column Banner

Hackaday Links: October 13, 2019

Trouble in the Golden State this week, as parts of California were subjected to planned blackouts. Intended to prevent a repeat of last year’s deadly wildfires, which were tied in part to defective electrical distribution equipment, the blackouts could plunge millions in the counties surrounding Sacramento into the dark for days. Schools have canceled classes, the few stores that are open are taking cash only, and hospitals are running on generators. It seems a drastic move for PG&E, the utility that promptly went into bankruptcy after being blamed for last year’s fires, but it has the support of the governor, so the plan is likely to continue as long as the winds do. One group is not likely to complain, though;  California amateur radio operators must be enjoying a greatly decreased noise floor in the blackout areas, thanks to the loss of millions of switch-mode power supplies and their RF noise.

Good news, bad news for Fusion 360 users. Autodesk, the company behind the popular and remarkably capable CAD/CAM/CAE package, has announced changes to its licensing scheme, which went into effect this week. Users no longer have to pay for the “Ultimate” license tier to get goodies like 5-axis machining and generative design tools, as all capabilities are now included in the single paid version of Fusion 360. That’s good because plenty of users were unwilling to bump their $310 annual “Standard” license fee up to $1535 to get those features, but it’s bad because now the annual rate goes to $495. In a nice nod to the current userbase, those currently on the Standard license, as well as early adopters, will get to keep the $310 annual rate as long as they renew, and The $495 pricing tier went into effect in November of 2018, while anyone still on the $310 annual price was grandfathered in (and will remain to be). At that time there was still a $1535 tier called Ultimate, whose price will now be going away but the features remain in the $495 tier which is now the only pricing option for Fusion 360. Ultimate users will see a $1040 price drop. As for the current base of freeloaders like yours truly, fear not: Fusion 360 is still free for personal, non-commercial use. No generative design or tech support for us, though. (Editor’s Note: This paragraph was updated on 10/14/2019 to clarify the tier changes after Autodesk reached out to Hackaday via email.)

You might have had a bad day at the bench, but was it as bad as Román’s? He tipped us off to his nightmare of running into defective Wemos D1 boards – a lot of them. The 50 boards were to satisfy an order of data loggers for a customer, but all the boards seemed caught in an endless reboot loop when plugged into a USB port for programming. He changed PCs, changed cables, but nothing worked to stop the cycle except for one thing: touching the metal case of the module. His write up goes through all the dead-ends he went down to fix the problem, which ended up being a capacitor between the antenna and ground. Was it supposed to be there? Who knows, because once that cap was removed, the boards worked fine. Hats off to Román for troubleshooting this and sharing the results with us.

Ever since giving up their “Don’t be evil” schtick, Google seems to have really embraced the alternative. Now they’re in trouble for targeting the homeless in their quest for facial recognition data. The “volunteer research studies” consisted of playing what Google contractors were trained to describe as a “mini-game” on a modified smartphone, which captured video of the player’s face. Participants were compensated with $5 Starbucks gift cards but were not told that video was being captured, and if asked, contractors were allegedly trained to lie about that. Contractors were also allegedly trained to seek out people with dark skin, ostensibly to improve facial recognition algorithms that notoriously have a hard time with darker complexions. To be fair, the homeless were not exclusively targeted; college students were also given gift cards in exchange for their facial data.

For most of us, 3D-printing is a hobby, or at least in service of other hobbies. Few of us make a living at it, but professionals who do are often a great source of tips and tricks. One such pro is industrial designer Eric Strebel, who recently posted a video of his 3D-printing pro-tips. A lot of it is concerned with post-processing prints, like using a cake decorator’s spatula to pry prints off the bed, or the use of card scrapers and dental chisels to clean up prints. But the money tip from this video is the rolling cart he made for his Ultimaker. With the printer on top and storage below, it’s a great way to free up some bench space.

And finally, have you ever wondered how we hackers will rebuild society once the apocalypse hits and mutant zombie biker gangs roam the Earth? If so, then you need to check out Collapse OS, the operating system for an uncertain future. Designed to be as self-contained as possible, Collapse OS is intended to run on “field expedient” computers, cobbled together from whatever e-waste can be scrounged, as long as it includes a Z80 microprocessor. The OS has been tested on an RC2014 and a Sega Master System so far, but keep an eye out for TRS-80s, Kaypros, and the odd TI-84 graphing calculator as you pick through the remains of civilization.

The South American Power Outage That Plunged 48 Million Into Blackout

A massive power outage in South America last month left most of Argentina, Uruguay, and Paraguay in the dark and may also have impacted small portions of Chile and Brazil. It’s estimated that 48 million people were affected and as of this writing there has still been no official explanation of how a blackout of this magnitude occurred.

While blackouts of some form or another are virtually guaranteed on any power grid, whether it’s from weather events, accidental damage to power lines and equipment, lightning, or equipment malfunctioning, every grid will eventually see small outages from time to time. The scope of this one, however, was much larger than it should have been, but isn’t completely out of the realm of possibility for systems that are this complex.

Initial reports on June 17th cite vague, nondescript possible causes but seem to focus on transmission lines connecting population centers with the hydroelectric power plant at Yacyretá Dam on the border of Argentina and Paraguay, as well as some ongoing issues with the power grid itself. Problems with the transmission line system caused this power generation facility to become separated from the rest of the grid, which seems to have cascaded to a massive power failure. One positive note was that the power was restored in less than a day, suggesting at least that the cause of the blackout was not physical damage to the grid. (Presumably major physical damage would take longer to repair.) Officials also downplayed the possibility of cyber attack, which is in line with the short length of time that the blackout lasted as well, although not completely out of the realm of possibility.

This incident is exceptionally interesting from a technical point-of-view as well. Once we rule out physical damage and cyber attack, what remains is a complete failure of the grid’s largely automatic protective system. This automation can be a force for good, where grid outages can be restored quickly in most cases, but it can also be a weakness when the automation is poorly understood, implemented, or maintained. A closer look at some protective devices and strategies is warranted, and will give us greater insight into this problem and grid issues in general. Join me after the break for a look at some of the grid equipment that is involved in this system.

Continue reading “The South American Power Outage That Plunged 48 Million Into Blackout”

When The Grid Goes Dark

If you lived through the Y2K fiasco, you might remember a lot of hype with almost zero real-world ramifications in the end. As the calendar year flipped from 1999 to 2000 many forecast disastrous software bugs in machines controlling our banking and infrastructure. While this potential disaster didn’t quite live up to its expectations there was another major infrastructure problem, resulting in many blackouts in North America, that reared its head shortly after the new millennium began. While it may have seemed like Y2K was finally coming to fruition based on the amount of chaos that was caused, the actual cause of these blackouts was simply institutional problems with the power grid itself.

Continue reading “When The Grid Goes Dark”

LED Ice Cubes Prevent Alcohol Induced Blackouts

cube

On November 23rd last year, [Dhairya] attended a little shindig at MIT. Three drinks into the night, he blacked out and woke up in the hospital the next day. It was an alcohol-induced blackout, and like all parties at MIT, there’s an ingenious solution to [Dhairya]’s problem.

[Dhairya] came up with an alcohol-aware ice cube made of a coin cell battery, an ATtiny microcontroller, and an IR transceiver are molded into an edible gelatin ice cube. The microcontroller counts the number of sips per drink, and after one glass of adult beverage changes the color of the flashing LED from green to yellow. After two drinks the LED changes from yellow to red, signaling [Dhairya] to slow down.

If [Dhairya] feels the night is too young and keeps on drinking, the IR transmitter signals to his cell phone to send a text to a friend telling them to go take [Dhairya] home.

Less than three weeks after waking up in the hospital, [Dhairya] tested out his glowing ice cubes at another party. Everything performed wonderfully, even if he admits his creation is a little crude. A neat piece of work, and we can’t wait to see an update to this project.

Continue reading “LED Ice Cubes Prevent Alcohol Induced Blackouts”