Tales From The Sysadmin: Impending Hard Drive Doom

July 14, 2020 by Jonathan Bennett 70 Comments

It should have been another fine day, but not all was well in paradise. Few things bring a creeping feeling of doom like a computer that hardlocks and then refuses to boot. The clicking sound coming from the tower probably isn’t a good sign either. Those backups are up to date, right? Right?

There are some legends and old stories about hard drive repair. One of my favorites is the official solution to stiction for old drives: Smack it with a mallet. Another trick I’ve heard repeatedly is to freeze a hard drive before trying to read data off of it. This could actually be useful in a couple instances. The temperature change can help with stiction, and freezing the drive could potentially help an overheating drive last a bit longer. The downside is the potential for condensation inside the drive. Don’t turn to one of these questionable fixes unless you’ve exhausted the safer options.

For the purpose of this article, we’ll assume the problem is the hard drive, and not another component like a power supply or SATA cable causing problems. A truly dead drive is a topic for another time, but if the drive is alive enough to show up as a block device when plugged in, then there’s hope for recovering the data. One of the USB to SATA cables available on your favorite online store is a great way to recover data. Another option is booting off a Linux DVD or flash drive, and accessing the drive in place. If you’re lucky, you can just copy your files and call it a day. If the file transfer fails because of the dying drive, or you need a full disk image, it’s time to pull out some tools and get to work. Continue reading “Tales From The Sysadmin: Impending Hard Drive Doom” →

New Year Habits – What Do You Do For Data Storage?

January 20, 2020 by Bob Baddeley 109 Comments

2020 is a year of reflection and avoiding regret, and one of the biggest practices we all know we should do better is back up our data. Inevitably there will be a corruption or accident, and we mourn the loss of some valuable data and vow to never let it happen again, and then promptly forget about good data retention practices.

I believe life is about acquiring memories, so it makes sense to me to try to archive and store those memories so that I can reflect on them later, but data storage and management is a huge pain. There’s got to be a better way (cue black and white video of clumsy person throwing up arms in disgust).

Nice Cloud You Have There; Shame if Something Happened to It

The teens of the century saw a huge shift towards cloud storage. The advantages of instantly backing up files and using the cloud as the primary storage for all your devices is appealing. It’s now easier to transfer files via the cloud than with a cable. With Google Docs and WordPress we have our most important documents and writing stored as database blobs on someone else’s servers. Facebook and Google and Flickr record all of our memories as photo albums. Unlimited storage is common, and indexing is so good that we can find photos with a vague description of their contents.

These things are instantly accessible, but lack permanence. Gone are newspaper clippings and printed photos discovered in a shoebox. When we aren’t in control of those services, they can disappear without any warning. Even some big offerings have packed up shop, leaving people scrambling to back up data before the servers were shut down. Google Plus is closed, Yahoo Groups is closed, MySpace lost all content created prior to 2016, GeoCities closed in 2009, and Ubuntu One closed in 2014. It’s safe to say that no online content is safe from deletion. It’s also safe to say that cloud storage is a difficult location from which to extract your data.

With the risk of data leaks and privacy violations occurring daily, it’s also safe to say that some of your files should probably not be stored in the cloud in the first place. So, how do we do it well, and how do we get in the habit of doing it regularly?

Continue reading “New Year Habits – What Do You Do For Data Storage?” →

DIY Photo Backup In The Field

November 20, 2019 by Bryan Cockfield 29 Comments

They say a file isn’t backed up if it isn’t backed up twice. This is easy enough to do if you have access to your computer and a network, but if you’re a photographer you might end up in a place without either of these things and need a way to back up the files you just created. For that you’ll need a specialized photo backup tool which you can easily build yourself.

While commercial offerings are available which back up files locally from a camera’s SD card to another medium, they suffer from a high price. [André]’s solution can be had for a fraction of that cost. Using a Raspberry Pi Zero, a tiny USB hub, and a high capacity jump drive, a photographer can simply plug in an SD card and the Pi will handle the backups with varying levels of automation. The software that [André] made use of is called Little Backup Box written by [Dmitri Popov] and can be used typically as an automatic backup for any other device as well.

This is a great solution to backing up files on the go, whether they’re from a camera or any device that uses an SD card. Removable storage is tiny and easily lost, so it’s good to have a few backups in case the inevitable happens. Raspberry Pis are an ideal solution to data backup, and can even be battery powered if you’re really roughing it for a few days.

Ask Hackaday: What’s In Your Digital Bugout Bag?

April 9, 2018 by Dan Maloney 119 Comments

Your eyes pop open in the middle of the night, darting around the darkened bedroom as you wonder why you woke up. Had you heard something? Or was that a dream? The matter is settled with loud pounding on the front door. Heart racing as you see blue and red lights playing through the window, you open the door to see a grim-faced police officer standing there. “There’s been a hazardous materials accident on the highway,” he intones. “We need to completely evacuate this neighborhood. Gather what you need and be ready to leave in 15 minutes.”

Most people will live their entire lives without a scenario like this playing out, but such things happen all the time. Whether the disaster du jour is man-made or natural, the potential to need to leave in a big hurry is very real, and it pays to equip yourself to survive such an ordeal. The primary tool for this is the so-called “bugout bag,” a small backpack for each family member that contains the essentials — clothing, food, medications — to survive for 72 hours away from home.

A bugout bag can turn a forced evacuation from a personal emergency into a minor inconvenience, as those at greatest risk well know — looking at you, Tornado Alley. But in our connected world, perhaps it pays to consider updating the bugout bag to include the essentials of our online lives, those cyber-needs that we’d be hard-pressed to live without for very long. What would a digital bugout bag look like?

Continue reading “Ask Hackaday: What’s In Your Digital Bugout Bag?” →

Solar Power In A Can!

March 2, 2018 by James Hobson 17 Comments

When spending time camping, people often bring lanterns, flashlights, and the like — you might even bring along a solar charger. Instructables user [bennelson] is combining all your electrical powered needs by cramming solar power into a can.

Already designed to resist the elements, [bennelson] is using a 50cal. ammo can for a portable enclosure. Inside, he’s siliconed a 15AH, 12V lead-acid battery in the centre to maintain balance and to leave room for the wiring and storage. One cardboard mockup later, he laser-cut the top panel from 1/8″ plywood and secured a 20A solar charge controller, a four-in-one socket panel, and two banana plugs on its top face.

[bennelson] is using 12 AWG wire to match the 20A rating of the solar charge controller — including a fuse for safety — and lever lock-nut connectors to resolve some wiring complications. Industrial velcro keeps the top panel in place and easily removed should the need arise. When he’s out camping, he uses an 18V, 1A solar panel to charge, but can still use a DC power adapter to charge from the grid. Check out the full build video after the break!

Continue reading “Solar Power In A Can!” →

Fail Of The Week: GitLab Goes Down

February 2, 2017 by Jenny List 57 Comments

Has work been a little stressful this week, are things getting you down? Spare a thought for an unnamed sysadmin at the GitHub-alike startup GitLab, who early yesterday performed a deletion task on a PostgreSQL database in response to some problems they were having in the wake of an attack by spammers. Unfortunately due to a command line error he ran the deletion on one of the databases behind the company’s main service, forcing it to be taken down. By the time the deletion was stopped, only 4.5 Gb of the 300 Gb trove of data remained.

Reading their log of the incident the scale of the disaster unfolds, and we can’t help wincing at the phrase “out of 5 backup/replication techniques deployed none are working reliably or set up in the first place“. In the end they were able to restore most of the data from a staging server, but at the cost of a lost six hours of issues and merge requests. Fortunately for them their git repositories were not affected.

For 707 GitLab users then there has been a small amount of lost data, the entire web service was down for a while, and the incident has gained them more publicity in a day than their marketing department could have achieved in a year. The post-mortem document makes for a fascinating read, and will probably leave more than one reader nervously thinking about the integrity of whichever services they are responsible for. We have to hand it to them for being so open about it all and for admitting a failure of their whole company for its backup failures rather than heaping blame on one employee. In many companies it would all have been swept under the carpet. We suspect that GitLab’s data will be shepherded with much more care henceforth.

We trust an increasing amount of our assets to online providers these days, and this tale highlights some of the hazards inherent in placing absolute trust in them. GitLab had moved from a cloud provider to their own data centre, though whether or not this incident would have been any less harmful wherever it was hosted is up for debate. Perhaps it’s a timely reminder to us all: keep your own backups, and most importantly: test them to ensure they work.

Thanks [Jack Laidlaw] for the tip.

Rack server image: Trique303 [CC BY-SA 4.0], via Wikimedia Commons.