Today at about 10:00 AM Pacific time, Hackaday’s site host had an outage. All websites on the WordPress VIP Go platform were down, and that includes Hackaday. For about 45 minutes you couldn’t load any content, and for a bit more than two hours after that all we could display was a default WordPress theme with an alarmingly bright background.
At first, we were looking at a broken home page with nothing on it. We changed some things around on the back end and we had a black text on white background displaying our latest articles. Not ideal, but at least you could catch up on your reading if you happened to check in right at that time.
But wait! Unintended consequences are a real drag. Our theme doesn’t have comments built into the front page and blog page views, but the WordPress stock themes do. So comments left on those pages were being blasted out to your RSS feeds. I’d like to apologize for that. Once it was reported, we turned off comments on those pages and deleted what was there. But if you have a caching RSS reader you’ll still see those, sorry about that.
As I type this, all should be back to normal. The front end was restored around 1:00 PM Pacific time. We’ve continued our normal publishing schedule throughout, and we hope you have had a good laugh at this debacle. It might be a few days before I’m able to laugh about it though.
55 thoughts on “The Day Hackaday’s Theme Was Broken”
Oh thank god. I thought for a minute you were going with a new minimalist style.
I thought I was at the wrong site. Then it’s “when did they make these changes”?
Yeah. Had the new theme still been there today without any sort of “This is not by design, we’re working on it”, I probably would have stopped visiting HaD…
Glad it’s all been sorted.
For a moment I thought you have become some cheerful gadget blog. Scary stuff.
#f3bf10 4 lyfe!
I’ve got #1a1a1a on the brain.
Found my first tattoo. Mike, can I expense that?
I found that article interesting because I’m color blind. But why why do you call that red. For me any thing called red would start with C0 to FF and the other components would be below about 20 so about #C00000 to #FF2020
#f3bf10 is not red, its more of an amber.
I’ve been reading this site for years and I didn’t know this. I thought the colour scheme was dark gray and green.
Now that I know, it doesn’t look green to me anymore.
This is embarrassing.
@Tom- I think it used to be like that.
How this post wasn’t titled, “Hackaday announces Dark Theme” is beyond me.
Complete lack of creativity on the part of the editorial staff.
It’s not called Hackaday for nothing.
Just kidding. You guys are great.
For a moment, I was hoping Hack-A-Day was Hacked and the Hack-A-Dayers could invite the Hacker who Hacked their Hacking site to write about the Hacker’s Hack on Hack-A-Day for all the wanna-be Hackers to learn how to Hack.
I would like to know what happened on WordPress VIP’s side of things. They migrated to this new VIP Go platform back in April. We’d been rock solid for years on the old system, but have had a few short outages already on this one. Three hours is an eternity to be down, and across all sites that they host. Yikes.
Im sure thats it.
One of those….
Applying updates…. please wait.
(Pulls out hair)
Couldn’t disagree more. I’ve found the comments section of HaD infinitely valuable. Its where I often find parallel technologies or alternative solutions to the problem posed in the article. There is still a bit of “Not a Hack” or “Doesn’t need an Arduino” but you take the bad with the good.
Even if everyone spun up their own page for every thought they had, how on earth would you link them together without a comments or user feedback section to at least post a link? Honestly the more I think about and re-read your comment the more it seems like Trolling or at least a little ironic joke thats a little too dry?
Blogs, and let’s not forget Popular Science disabled commenting for a reason.
One reason was because they’re firmly in the Anthropogenic Global Warming camp and got tired of people posting all the inconvenient problems with the AGW theory.
WordPress is too big a target to safely use for anything now.
Everything needs to be on Post-it™ notes now.
There’s a hack for that! (TM)
Agreed. I just went back to html, and so far I haven’t had any break-in attempts. ;) The difference between typing a minimalist website in html and a wordpress site in their editor is surprisingly small. Notepad++ also boots WAY faster with no login. Filezilla for a quick upload and I think I’m actually saving time and energy. I don’t think I’ll ever go back to php/sql.
Here’s the latest from WordPress VIP. Sounds like this was a self-inflicted wound on their end:
At least they fessed up to it. Hosting is *hard*. I’ve been doing it off and on for 20 years, actively for 15. It’s not trivial, especially at scale. I just hope that HaD has a backup plan and a full set of backups for when things break even worse than that. I know you guys probably make a living with this site. Don’t put all your eggs in one basket. Have your own copy of your own data. Make it current and don’t rely on anyone elses backups. Host based backups aren’t accessible if your host is down.
blockquote>p:last-child:after needs some CSS fixes once the fire dies down over there :)
what a white wash
How’s that a white wash? They’ve admitted fault and are investigating what happened, presumably they’re also going to investigate ways to stop it from happening again. Maybe your sarcasm tags weren’t rendered?
I’m sure it was sarcasm,
that bright white shining out of my monitor temporarily blinded my uhhh, coworkers, down here in the boiler room, interfering with our uhhh network penetration testing. Fortunately, some were able to adjust their balaclavas to protect their eyes.
retro.hackaday still showing content from 2010 -2011
Yeah, we should do an update on that so that it’s at least showing 2012 ;-)
Go full retrograde and update it backwards, 2009-wards that is.
pics or it didn’t happen. :-)
This was what I saw yesterday: https://imgur.com/a/HOGzD0K
My heart stood still – another website switched to the great new fancy unreadable ugly layout :-(((
But it was just a bad dream, today I woke up to the ol’ good world: https://imgur.com/oSURvuK
What? You want to blind us again!
This was what I saw yesterday: https://imgur.com/a/HOGzD0K
My heart stood still – another website switched to the great new fancy unreadable ugly layout :-((( But it was just a bad dream, today I woke up to the ol’ good world: https://imgur.com/oSURvuK
Glad to see it’s sorted out! No harm done here, it was funny pretty quick once I realized what was happening to my RSS reader.
It might be a few days before I can see again.
I’m really surprised no one posted “butts” to the 233,000+ RSS subscribers.
Doesn’t accept pictures.
All this and still no EDIT capability for posts, like pretty much every other comments section on other sites.
i thought this was going to be about the time hackaday started allowing capital letters and coloured thumbnail images without bits of “tape” stuck over the corners
Oh I miss the tape! So edgy.
I thought HAD had been hacked to add in a decent comment system…
Something else that’s not strictly broken, but something that should be fixed – look at your site’s icon in Chrome’s dark mode. It’s the only icon in my bookmarks bar that goers stealth.
Did Joe Kim just whip up that banner graphic for this article?
Or, did HaD already have one on file?
I think I’ve seen that one before, so it would be something they just have kicking around for events like these.
Joe’s back catalog is extensive and awesome. This one is from about 4-5 years ago.
I like uses static site generators for blogs like Pelican or hugo. No need for databases, backends blah blah blah. And yes no built-in commenting setup. I guess one could always integrate an external comment service.
Just a bunch of html files and supporting resources. Heck you could just use http. no need for https since no one has to login e.t.c.
https is not only for protecting credentials, it’s also for protecting server’s reply.
It protects against MitM attack like injecting ads or malware.
really wish the browsers would ease up about self-signed certs then
for next time, easy to solve: send print screen of all hackaday articles by mail , and many users will use outage time cleaning their inbox :)
Please be kind and respectful to help make the comments section excellent. (Comment Policy)