The South American Power Outage That Plunged 48 Million Into Blackout

A massive power outage in South America last month left most of Argentina, Uruguay, and Paraguay in the dark and may also have impacted small portions of Chile and Brazil. It’s estimated that 48 million people were affected and as of this writing there has still been no official explanation of how a blackout of this magnitude occurred.

While blackouts of some form or another are virtually guaranteed on any power grid, whether it’s from weather events, accidental damage to power lines and equipment, lightning, or equipment malfunctioning, every grid will eventually see small outages from time to time. The scope of this one, however, was much larger than it should have been, but isn’t completely out of the realm of possibility for systems that are this complex.

Initial reports on June 17th cite vague, nondescript possible causes but seem to focus on transmission lines connecting population centers with the hydroelectric power plant at Yacyretá Dam on the border of Argentina and Paraguay, as well as some ongoing issues with the power grid itself. Problems with the transmission line system caused this power generation facility to become separated from the rest of the grid, which seems to have cascaded to a massive power failure. One positive note was that the power was restored in less than a day, suggesting at least that the cause of the blackout was not physical damage to the grid. (Presumably major physical damage would take longer to repair.) Officials also downplayed the possibility of cyber attack, which is in line with the short length of time that the blackout lasted as well, although not completely out of the realm of possibility.

This incident is exceptionally interesting from a technical point-of-view as well. Once we rule out physical damage and cyber attack, what remains is a complete failure of the grid’s largely automatic protective system. This automation can be a force for good, where grid outages can be restored quickly in most cases, but it can also be a weakness when the automation is poorly understood, implemented, or maintained. A closer look at some protective devices and strategies is warranted, and will give us greater insight into this problem and grid issues in general. Join me after the break for a look at some of the grid equipment that is involved in this system.

Continue reading “The South American Power Outage That Plunged 48 Million Into Blackout”

Inside The Mysterious Global Navigation Outage You Probably Didn’t Notice

The entire world has come to depend on satellite navigation systems in the forty or so years since the first Global Positioning System satellites took to orbit. Modern economies have been built on the presumption that people and assets can be located to within a meter or better anywhere on, above, or even slightly under the surface of the planet. For years, GPS was the only way to do that, but billions have been sunk into fielding other global navigation systems, achieving a measure of independence from GPS and to putting in place some badly needed redundancy in case of outages, like that suffered by the European Union’s Galileo system recently.

The problem with Galileo, the high-accuracy public access location system that’s optimized for higher latitudes, seems to be resolved as of this writing. The EU has been tight-lipped about the outage, however, leaving investigation into its root cause to a few clever hackers armed with SDRs and comprehensive knowledge of exactly how a constellation of satellites can use the principles of both general and special relativity to point you to your nearest Starbucks.

Continue reading “Inside The Mysterious Global Navigation Outage You Probably Didn’t Notice”

The Saga Of 32-Bit Linux: Why Going 64-Bit Raises Concerns Over Multilib

The story of Linux so far, as short as it may be in the grand scheme of things, is one of constant forward momentum. There’s always another feature to implement, an optimization to make, and of course, another device to support. With developer’s eyes always on the horizon ahead of them, it should come as no surprise to find that support for older hardware or protocols occasionally falls to the wayside. When maintaining antiquated code monopolizes developer time, or even directly conflicts with new code, a difficult decision needs to be made.

Of course, some decisions are easier to make than others. Back in 2012 when Linus Torvalds officially ended kernel support for legacy 386 processors, he famously closed the commit message with “Good riddance.” Maintaining support for such old hardware had been complicating things behind the scenes for years while offering very little practical benefit, so removing all that legacy code was like taking a weight off the developer’s shoulders.

The rationale was the same a few years ago when distributions like Arch Linux decided to drop support for 32-bit hardware entirely. Maintainers had noticed the drop-off in downloads for the 32-bit versions of their distributions and decided it didn’t make sense to keep producing them. In an era where even budget smartphones are shipping with 64-bit processors, many Linux distributions have at this point decided 32-bit CPUs weren’t worth their time.

Given this trend, you’d think Ubuntu announcing last month that they’d no longer be providing 32-bit versions of packages in their repository would hardly be newsworthy. But as it turns out, the threat of ending 32-bit packages caused the sort of uproar that we don’t traditionally see in the Linux community. But why?

Continue reading “The Saga Of 32-Bit Linux: Why Going 64-Bit Raises Concerns Over Multilib”

Electric Cars Sound Off, Starting July 1st

By and large, automakers have spent much of the last century trying to make cars quieter and more comfortable. Noise from vehicles can be disruptive and just generally annoying, so it makes sense to minimise it where possible.

However, the noise from the average motor vehicle can serve a useful purpose. A running engine acts as an auditory warning to those nearby. This is particularly useful to help people avoid walking in front of moving vehicles, and is especially important for the visually impaired.

Electric vehicles, with their near-silent powertrains, have put this in jeopardy. Thus, from July 1st, 2019, the European Union will enforce regulations on the installation of noise-making devices on new electric and hybrid vehicles. They are referred to as the “Acoustic Vehicle Alert System”, and it’s been a hot area of development for some time now. Continue reading “Electric Cars Sound Off, Starting July 1st”

Automate The Freight: Autonomous Delivery Hits The Mainstream

It should come as no surprise that we here at Hackaday are big boosters of autonomous systems like self-driving vehicles. That’s not to say we’re without a healthy degree of skepticism, and indeed, the whole point of the “Automate the Freight” series is that economic forces will create powerful incentives for companies to build out automated delivery systems before they can afford to capitalize on demand for self-driving passenger vehicles. There’s a path to the glorious day when you can (safely) nap on the way to work, but that path will be paved by shipping and logistics companies with far deeper pockets than the average commuter.

So it was with some interest that we saw a flurry of announcements in the popular press recently regarding automated deliveries. Each by itself wouldn’t be worthy of much attention; companies are always maneuvering to be seen as ahead of the curve on coming trends, and often show off glitzy, over-produced videos and well-crafted press releases as a low-effort way to position themselves as well as to test markets. But seeing three announcements at one time was unusual, and may point to a general feeling by manufacturers that automated deliveries are just around the corner. Plus, each story highlighted advancements in areas specifically covered by “Automate the Freight” articles, so it seemed like a perfect time to review them and perhaps toot our own horn a bit.

Continue reading “Automate The Freight: Autonomous Delivery Hits The Mainstream”

Takata’s Deadly Airbags: An Engineering Omnishambles

Engineers are, for the time being, only human. This applies even more so to executives, and all the other people that make up a modern organisation. Naturally, mistakes are made. Some are minor, while others are less so. It’s common knowledge that problems are best dealt with swift and early, and yet so often they are ignored in the hopes that they’ll go away.

You might have heard the name Takata in the news over the last few years. If that name doesn’t ring a bell you’ve likely heard that there was a major recall of airbag-equipped vehicles lately. The story behind it is one of a single decision leading to multiple deaths, scores of injuries, a $1 billion fine, and the collapse of a formerly massive automotive supplier.

Continue reading “Takata’s Deadly Airbags: An Engineering Omnishambles”

GPS And ADS-B Problems Cause Cancelled Flights

Something strange has been going on in the friendly skies over the last day or so. Flights are being canceled. Aircraft are grounded. Passengers are understandably upset. The core of the issue is GPS and ADS-B systems. The ADS-B system depends on GPS data to function properly, but over this weekend a problem with the quality of the GPS data has disrupted normal ADS-B features on some planes, leading to the cancellations.

What is ADS-B and Why Is It Having Trouble?

Automatic Dependent Surveillance-Broadcast (ADS-B) is a communication system used in aircraft worldwide. Planes transmit location, speed, flight number, and other information on 1090 MHz. This data is picked up by ground stations and eventually displayed on air traffic controller screens. Aircraft also receive this data from each other as part of the Traffic Collision Avoidance System (TCAS).

ADS-B isn’t a complex or encrypted signal. In fact, anyone with a cheap RTL-SDR can receive the signal. Aviation buffs know how cool it is to see a map of all the aircraft flying above your house. Plenty of hackers have worked on these systems, and we’ve covered that here on Hackaday. In the USA, the FAA will effectively require all aircraft to carry ADS-B transponders by January 1st, 2020. So as you can imagine, most aircraft already have the systems installed.

The ADS-B system in a plane needs to get position data before it can transmit. These days, that data comes from a global satellite navigation system. In the USA, that means GPS. GPS is currently having some problems though. This is where Receiver autonomous integrity monitoring (RAIM) comes in. Safety-critical GPS systems (those in planes and ships) cross-check their current position. If GPS is sending degraded or incorrect data, it is sent to the FAA who displays it on their website. The non-precision approach current outage map is showing degraded service all over the US Eastern seaboard, as well as the North. The cause of this signal degradation is currently unknown.

What Hardware is Affected?

GPS isn’t down though — you can walk outside with your cell phone to verify that. However, it is degraded. How a plane’s GPS system reacts to that depends on the software built into the GPS receiver. If the system fails, the pilots will have to rely on older systems like VOR to navigate. But ADS-B will have even more problems. An aircraft ADS-B system needs position data to operate.  If you can’t transmit your position information, air traffic controllers need to rely on old fashioned radar to determine position. All of this adds up to a safety of flight problem, which means grounding the aircraft.

Digging through canceled flight lists, one can glean which aircraft are having issues. From the early reports, it seems like Bombardier CRJ 700 and 900 have problems. Folks on Airliners.net are speculating that any aircraft with Rockwell Collins flight management systems are having problems.

This is not a small issue, there are hundreds or thousands of canceled flights. The FAA set up a teleconference to assess the issue. Since then, the FAA has issued a blanket waiver to all affected flights. They can fly, but only up to 28,000 feet.

This is a developing story, and we’ll be keeping an eye on it. Seeing how the industry handles major problems is always educational, and there will be much to learn in the coming days.