Fail of the Week is a Hackaday column which runs every Wednesday. Help keep the fun rolling by writing about your past failures and sending us a link to the story — or sending in links to fail write ups you find in your Internet travels.
Has work been a little stressful this week, are things getting you down? Spare a thought for an unnamed sysadmin at the GitHub-alike startup GitLab, who early yesterday performed a deletion task on a PostgreSQL database in response to some problems they were having in the wake of an attack by spammers. Unfortunately due to a command line error he ran the deletion on one of the databases behind the company’s main service, forcing it to be taken down. By the time the deletion was stopped, only 4.5 Gb of the 300 Gb trove of data remained.
Reading their log of the incident the scale of the disaster unfolds, and we can’t help wincing at the phrase “out of 5 backup/replication techniques deployed none are working reliably or set up in the first place“. In the end they were able to restore most of the data from a staging server, but at the cost of a lost six hours of issues and merge requests. Fortunately for them their git repositories were not affected.
For 707 GitLab users then there has been a small amount of lost data, the entire web service was down for a while, and the incident has gained them more publicity in a day than their marketing department could have achieved in a year. The post-mortem document makes for a fascinating read, and will probably leave more than one reader nervously thinking about the integrity of whichever services they are responsible for. We have to hand it to them for being so open about it all and for admitting a failure of their whole company for its backup failures rather than heaping blame on one employee. In many companies it would all have been swept under the carpet. We suspect that GitLab’s data will be shepherded with much more care henceforth.
We trust an increasing amount of our assets to online providers these days, and this tale highlights some of the hazards inherent in placing absolute trust in them. GitLab had moved from a cloud provider to their own data centre, though whether or not this incident would have been any less harmful wherever it was hosted is up for debate. Perhaps it’s a timely reminder to us all: keep your own backups, and most importantly: test them to ensure they work.
There’s a reason we often use the phrase “It ain’t Rocket Science”. Because real rocket science IS difficult. It is dangerous and complicated, and a lot of things can and do go wrong, often with disastrous consequences. It is imperative that the lessons learned from past failures must be documented and disseminated to prevent future mishaps. This is much easier said than done. There’s a large number of agencies and laboratories working on multiple projects over long periods of time. Which is why NASA has set up NASA Lessons Learned — a central, online database of issues documented by contributors from within NASA as well as other organizations.
Unfortunately, all of this body of past knowledge is sometimes still not enough to prevent problems. Case in point is a recently discovered issue on the ISS — a completely avoidable power supply mistake. Science payloads attach to the ISS via holders called the ExPRESS logistics carriers. These provide mechanical anchoring, electrical power and data links. Inside the carriers, the power supply meant to supply 28V to the payloads was found to have a few capacitors mounted the other way around. This has forced the payloads to use the 120V supply instead, requiring them to have an additional 120V to 28V converter retrofit. This means modifying the existing hardware and factoring in additional weight, volume, heat, cost and other issues when adding the extra converter. If you’d like to dig into the details, check out this article about NASA’s power supply fail.
Have you ever wired up a piece of test equipment to a circuit or piece of equipment on your bench, only to have the dreaded magic smoke emerge when you apply power? [Steaky] did, and unfortunately for him the smoke was coming not from his circuit being tested but from a £2300 Clare H101 HiPot tester. His write-up details his search for the culprit, then looks at how the manufacturer might have protected the instrument.
[Steaky]’s employer uses the HiPot tester to check that adjacent circuits are adequately isolated from each other. A high voltage is put between the two circuits, and the leakage current between them is measured. A variety of tests are conducted on the same piece of equipment, and the task in hand was to produce a new version of a switch box with software control to swap between the different tests.
This particular instrument has a guard circuit, a pair of contacts that have to be closed before it will proceed. So the switch box incorporated a relay to close them, and wiring was made to connect to the guard socket. At first it was thought that the circuit might run at mains voltage, but when it was discovered to be only 5V the decision was made to use a 3.5mm jack on the switch box. Inadvertently this was left with its sleeve earthed, which had the effect of shorting out a DC to DC converter in the HiPot tester and releasing the smoke. Fortunately then the converter could be replaced and the machine brought back to life, but it left questions about the design of the internal circuit. What was in effect a logic level sense line was in fact connected to a low current power supply, and even the most rudimentary of protection circuitry could have saved the day.
We all stand warned to be vigilant for this kind of problem, and kudos to [Steaky] for both owning up to this particular fail and writing such a good analysis of it.
[B Arnold] is hearing voices and needs help from the Hackaday community. But before any of you armchair psychiatrists run off to WebMD, rest assured that [B Arnold] suffers not from schizophrenia but rather has an RF coupling problem.
The project (which isn’t posted yet) is an attempt to turn a C.H.I.P into an Amazon Echo, for which [B Arnold] needed an audio amplifier. Turning to the junk bin, he unearthed an LM386, that venerable power amp chip that first appeared in the mid-70s. Dead simple and able to run off a 9-volt battery, the LM386 that has found its way into thousands of commercial products and countlesshacks.
Shortly after applying power to the amp, [B Arnold] started hearing things – faint, far-off voices, scratchy but discernible. A bit of repositioning of wires and hands improved the signal enough for a station ID – an FM talk radio station on 97.1 MHz. [B Arnold] doesn’t mention the call sign, but it might have been KFTK out of St. Louis, Missouri; in any case, it would be helpful to know the range from the transmitter to the inadvertent receiver. Two low-fidelity audio clips are included below for your listening pleasure – you’ll want your headphones on, and Sample 2 is better than Sample 1 – as are photos of the offending circuit.
What do you think is going on here? We’ve heard of RF coupling of AM radio stations before, but how would FM signals be making it into this circuit and out of the speaker? Is there anything [B Arnold] did wrong to get this result? Sound off in the comments and let us know your horror stories of RF coupling.
Would you use your tech prowess to cheat at the Pinewood Derby? When your kid brings home that minimalist kit and expects you to help engineer a car that can beat all the others in the gravity-powered race, the temptation is there. But luckily, there are some events that don’t include the kiddies and the need for parents to assume the proper moral posture. When the whole point of the Pinewood Derby is to cheat, then you pull out all the stops, and you might try building an electrodynamic suspension hoverboard car.
Fortunately for [ch00ftech], the team-building Derby sponsored by his employer is a little looser with the rules than the usual event. Loose enough perhaps to try a magnetically levitating car. The aluminum track provided a perfect surface to leverage Lenz’s Law. [ch00ftech] tried different arrangements of coils and drivers in an attempt to at least reduce the friction between car and track, if not outright levitate it. Sadly, time ran out and physics had others ideas, so [ch00ftech], intent on cheating by any means, tried spoofing the track timing system with a ridiculous front bumper of IR LEDs. But even that didn’t work in the end, and poor [ch00f]’s car wound up in sixth place.
So what could [ch00ftech] had done better? Was he on the right course with levitation? Or was spoofing the sensors likely to have worked with better optics? Or should he have resorted to jet propulsion or a propeller drive? How would you cheat at the Pinewood Derby?
Fail of the Week is a Hackaday column which celebrates failure as a learning tool. Help keep the fun rolling by writing about your own failures and sending us a link to the story — or sending in links to fail write ups you find in your Internet travels.
There are times when you set out to do one thing, and though you do not achieve your aim you succeed in making something else that’s just a bit special. [TheKhakinator] sent us something he described as a fail, but even though we’re posting it as one of our Fail Of The Week series we think the result still has something of the win about it. It may not be the amazing hack he hoped it would become, but that really does not matter in this case.
On his travels in China his attention was caught by an everyday electronic gadget, an electronic calculator that speaks the numbers and operations in Chinese as you use it. He bought a few of them, hoping that when he got them back to his bench he’d find an EEPROM containing the samples, which he could replace with his own for a cheap but low bitrate sampler.
Sadly this neat hack was not to be, for when he tore the surprisingly well-built calculators down he found only an epoxy blob concealing a single chip. All was not lost though, for while investigating the device’s features he discovered that as well as speaking Chinese numbers and operands it also had a selection of alarm tunes built-in, plus a mode in which it operated as a rudimentary electronic organ. He leaves us with a couple of videos we’ve posted below the break, first his teardown, and then a virtual orchestra of calculators playing dance music as he forgets the fail and concentrates on the win.
Is it possible to recycle failed 3D prints? As it turns out, it is — as long as your definition of “recycle” is somewhat flexible. After all, the world only needs so many coasters.
To be fair, [Devin]’s experiment is more about the upcycling side of the recycling equation, but it was certainly worth undertaking. 3D printing has hardly been reduced to practice, and anyone who spends any time printing knows that it’s easy to mess up. [Devin]’s process starts when the colorful contents of a bin full of failed prints are crushed with a hammer. Spread out onto a properly prepared (and never to be used again for cookies) baking sheet and cooked in the oven at low heat, the plastic chunks slowly melt into a thin, even sheet.
[Devin]’s goal was to cast them into a usable object, so he tried to make a bowl. He tried reheating discs of the material using an inverted metal bowl as a form but he found that the plastic didn’t soften evenly, resulting in Dali-esque bowls with thin spots and holes. He then flipped the bowl and tried to let the material sag into the form; that worked a little better but it still wasn’t the win he was looking for.
In the end, all [Devin] really ended up with is some objets d’art and a couple of leaky bowls. What else could he have done with the plastic? Would he have been better off vacuum forming the bowls or perhaps even pressure forming them? Or does the upcycling make no sense when you can theoretically make your own filament? Let us know in the comments how you would improve this process.