The Day Hackaday’s Theme Was Broken

Today at about 10:00 AM Pacific time, Hackaday’s site host had an outage. All websites on the WordPress VIP Go platform were down, and that includes Hackaday. For about 45 minutes you couldn’t load any content, and for a bit more than two hours after that all we could display was a default WordPress theme with an alarmingly bright background.

At first, we were looking at a broken home page with nothing on it. We changed some things around on the back end and we had a black text on white background displaying our latest articles. Not ideal, but at least you could catch up on your reading if you happened to check in right at that time.

But wait! Unintended consequences are a real drag. Our theme doesn’t have comments built into the front page and blog page views, but the WordPress stock themes do. So comments left on those pages were being blasted out to your RSS feeds. I’d like to apologize for that. Once it was reported, we turned off comments on those pages and deleted what was there. But if you have a caching RSS reader you’ll still see those, sorry about that.

As I type this, all should be back to normal. The front end was restored around 1:00 PM Pacific time. We’ve continued our normal publishing schedule throughout, and we hope you have had a good laugh at this debacle. It might be a few days before I’m able to laugh about it though.

55 thoughts on “The Day Hackaday’s Theme Was Broken

    1. Yeah. Had the new theme still been there today without any sort of “This is not by design, we’re working on it”, I probably would have stopped visiting HaD…

      1. I found that article interesting because I’m color blind. But why why do you call that red. For me any thing called red would start with C0 to FF and the other components would be below about 20 so about #C00000 to #FF2020

          1. I’ve been reading this site for years and I didn’t know this. I thought the colour scheme was dark gray and green.

            Now that I know, it doesn’t look green to me anymore.

            This is embarrassing.

  1. For a moment, I was hoping Hack-A-Day was Hacked and the Hack-A-Dayers could invite the Hacker who Hacked their Hacking site to write about the Hacker’s Hack on Hack-A-Day for all the wanna-be Hackers to learn how to Hack.

    1. I would like to know what happened on WordPress VIP’s side of things. They migrated to this new VIP Go platform back in April. We’d been rock solid for years on the old system, but have had a few short outages already on this one. Three hours is an eternity to be down, and across all sites that they host. Yikes.

    2. Couldn’t disagree more. I’ve found the comments section of HaD infinitely valuable. Its where I often find parallel technologies or alternative solutions to the problem posed in the article. There is still a bit of “Not a Hack” or “Doesn’t need an Arduino” but you take the bad with the good.

      Even if everyone spun up their own page for every thought they had, how on earth would you link them together without a comments or user feedback section to at least post a link? Honestly the more I think about and re-read your comment the more it seems like Trolling or at least a little ironic joke thats a little too dry?

        1. One reason was because they’re firmly in the Anthropogenic Global Warming camp and got tired of people posting all the inconvenient problems with the AGW theory.

    1. Agreed. I just went back to html, and so far I haven’t had any break-in attempts. ;) The difference between typing a minimalist website in html and a wordpress site in their editor is surprisingly small. Notepad++ also boots WAY faster with no login. Filezilla for a quick upload and I think I’m actually saving time and energy. I don’t think I’ll ever go back to php/sql.

  2. Here’s the latest from WordPress VIP. Sounds like this was a self-inflicted wound on their end:

    We’ve now fixed the 503 errors and availability issues impacting the VIP Go platform. We’re addressing any remaining issues on a per-site basis.

    At 17:01 UTC, we deployed an update to the VIP Go platform. Due to an issue in the deployment system, the filesystem mounts on some containers broke, resulting in missing client code. As a result, sites experienced a blank screen or loaded a default WordPress theme. To fix the issue, we restarted all web and CLI containers on the platform, which fixed the container mounts. We prioritized finding a definitive solution to this problem, which led to a slower process than we’d hoped.

    We’re in the process of investigating the causes of this disruption, and will share a full incident report this week.

    1. At least they fessed up to it. Hosting is *hard*. I’ve been doing it off and on for 20 years, actively for 15. It’s not trivial, especially at scale. I just hope that HaD has a backup plan and a full set of backups for when things break even worse than that. I know you guys probably make a living with this site. Don’t put all your eggs in one basket. Have your own copy of your own data. Make it current and don’t rely on anyone elses backups. Host based backups aren’t accessible if your host is down.

      1. How’s that a white wash? They’ve admitted fault and are investigating what happened, presumably they’re also going to investigate ways to stop it from happening again. Maybe your sarcasm tags weren’t rendered?

        1. I’m sure it was sarcasm,
          that bright white shining out of my monitor temporarily blinded my uhhh, coworkers, down here in the boiler room, interfering with our uhhh network penetration testing. Fortunately, some were able to adjust their balaclavas to protect their eyes.

  3. Something else that’s not strictly broken, but something that should be fixed – look at your site’s icon in Chrome’s dark mode. It’s the only icon in my bookmarks bar that goers stealth.

  4. I like uses static site generators for blogs like Pelican or hugo. No need for databases, backends blah blah blah. And yes no built-in commenting setup. I guess one could always integrate an external comment service.

    Just a bunch of html files and supporting resources. Heck you could just use http. no need for https since no one has to login e.t.c.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.