internet outages

2 Articles

How One Uncaught Rust Exception Took Out Cloudflare

November 20, 2025 by 2 Comments

On November 18 of 2025 a large part of the Internet suddenly cried out and went silent, as Cloudflare’s infrastructure suffered the software equivalent of a cardiac arrest. After much panicked debugging and troubleshooting, engineers were able to coax things back to life again, setting the stage for the subsequent investigation. The results of said investigation show how a mangled input file caused an exception to be thrown in the Rust-based FL2 proxy which went uncaught, throwing up an HTTP 5xx error and thus for the proxy to stop proxying customer traffic. Customers who were on the old FL proxy did not see this error.

The input file in question was the features file that is generated dynamically depending on the customer’s settings related to e.g. bot traffic. A change here resulted in said feature file to contain duplicate rows, increasing the number of typical features from about 60 to over 200, which is a problem since the proxy pre-allocates memory to contain this feature data.

Continue reading “How One Uncaught Rust Exception Took Out Cloudflare” →

A History Of Internet Outages

July 27, 2024 by 31 Comments

We heard a story that after the recent hurricane, a man noted that while the house was sweltering hot because the power was still out, his kids were more anxious for the internet to come back online. The Internet is practically a basic necessity for most people, but as you may have noticed with the recent CrowdStrike debacle, the Internet isn’t always reliable. Granted, the problem in that case wasn’t the Internet per se, but a problem with many critical hosts that provide services. [Thomas Germain] from the BBC took the opportunity to recall some of the more bizarre reasons we’ve had massive Internet outages in the past.

While teens after a hurricane might miss social media, global outages can be serious business. With 8.5 million computers dead, 911 services went down, medical surgeries were canceled, and — of course — around 46,000 flights were canceled in a single day. We have short memories for these outages, but as [Thomas] points out, this was far from the first massive outage, and many of them have very strange backstories.

Continue reading “A History Of Internet Outages” →