We’ve all experienced power outages of some kind, be it a breaker tripping at an inconvenient time to a storm causing a lack of separation between a tree and a power line. The impact is generally localized and rarely is there a loss of life, though it can happen. But in the video below the break, [Grady] of Practical Engineering breaks down the Northeast Blackout of 2003, the largest power failure ever experienced in North America. Power was out for days in some cases, and almost 100 deaths were attributed to the loss of electricity.
[Grady] goes into a good amount of detail regarding the monitoring systems, software simulation, and contingency planning that goes into operating a large scale power grid. The video explains how inductive loads cause reactance and how the effect exacerbated an already complex problem. Don’t know what inductive loads and reactance are? That’s okay, the video explains it quite well, and it gives an excellent basis for understanding AC electronics and even RF electronic theories surrounding inductance, capacitance, and reactance.
So, what caused the actual outage? The complex cascade failure is explained step by step, and the video is certainly worth the watch, even if you’re already familiar with the event.
It would be irresponsible to bring up the 2003 outage without talking about the Texas ERCOT outages just one year ago– an article whose comments section nearly caused a blackout at the Hackaday Data Center!
Continue reading “Cascade Failures, Computer Problems, And Ohms Law: Understanding The Northeast Blackout Of 2003”
The best type of power outage is no power outage, but they will inevitably happen. When they do, a hacker with a house full of stuff and a head full of ideas is often the person of the hour. Or the day, or perhaps the week, should the outage last long past the fun little adventure phase and become a nuisance or even an outright emergency.
Such was the position that [FFcossag] found himself in at the beginning of January, when a freak storm knocked out power to his community on a remote island in the middle of the Baltic Sea. [FFcossag] documented his attempts to survive the eight-day outage in vlog form, and although each entry is fairly long, there’s a lot to be learned from his ordeal. His main asset was a wood cook stove in the basement of the house, which served as his heat source. He used a car radiator and a small water pump to get some heat upstairs – a battery bank provided the power for that, at least for a while. The system evolved over the outage and became surprisingly good at keeping the upstairs warm.
The power eventually came back on, but to add insult to injury, almost as soon as it did, the ground-source heat pump in the house went on the fritz. A little sleuthing revealed an open power resistor in the heat pump control panel, but without a replacement on hand, [FFcossag] improvised. Parts from a 30-year-old TV transmitter were close at hand, including a nice handful of power resistors. A small parallel network gave the correct value and the heat pump came back online.
All in all, it was a long, cold week for [FFcossag], but he probably fared better than his neighbors. Want to be as prepared for your next outage? Check out [Jenny]’s comprehensive guide.
Continue reading “Adventures In Power Outage Hacking”
You no doubt heard about the Amazon S3 outage that happened earlier this week. It was reported far and wide by media outlets who normally don’t delve into details of the technology supporting our connected world. It is an interesting thing to think that most people have heard about The Cloud but never AWS and certainly not S3.
We didn’t report on the outage, but we ate up the details of the aftermath. It’s an excellent look under the hood. We say kudos to Amazon for adding to the growing trend of companies sharing the gory details surrounding events like this so that we can all understand what caused this and how they plan to avoid it in the future.
Turns out the S3 team was working on a problem with some part of the billing system and to do so, needed to take a few servers down. An incorrect command used when taking those machines down ended up affecting a larger block than expected. So they went out like a light switch — but turning that switch back on wasn’t nearly as easy.
The servers that went down run various commands in the S3 API. With the explosive growth of the Simple Storage Service, this “reboot” hadn’t been tried in several years and took far longer than expected. Compounding this was a backlog of tasks that built up while they were bringing the API servers back online. Working through that backlog took time as well. The process was like waiting for a bathtub to fill up with water. It must have been an agonizing process for those involved, but certainly not as bad as the folks who had to restore GitLab service a few weeks back.
A post about Operation Chokehold popped up on (fake) Steve Jobs’ blog this morning. It seems some folks are just plain tired of AT&T giving excuses about their network. The straw that broke the camel’s back came when AT&T floated the idea of instituting bandwidth limitations for data accounts. Now, someone hatched the idea of organizing enough users to bring the whole network down by maxing their bandwidth at the same time.
We’re not quite sure what to think about this. Our friend Google told us that there’s plenty of press already out there regarding Operation Chokehold so it’s not beyond comprehension that this could have an effect on the network. On the other hand, AT&T already knows about it and we’d wager they’re working on a plan to mitigate any outages that might occur.
As for the effectiveness of the message? We’d have more sympathy for AT&T if they didn’t have exclusivity contracts for their smart phones (most notably the iPhone). And if you’re selling an “Unlimited Plan” it should be just that. What do you think?
Yesterday’s Gmail service outage is a hot topic on just about every news site right now. For so many of us that have always taken the reliability of Gmail for granted it was a real shock to lose all of the functionality of the web based system. Now that we’ve learned our lesson, here’s a couple of tips to help you out the next time there’s an outage.
Continue reading “Gmail Without The Cloud: Tips For Next Time”