Fail of the Week: GitLab Goes Down

Has work been a little stressful this week, are things getting you down? Spare a thought for an unnamed sysadmin at the GitHub-alike startup GitLab, who early yesterday performed a deletion task on a PostgreSQL database in response to some problems they were having in the wake of an attack by spammers. Unfortunately due to a command line error he ran the deletion on one of the databases behind the company’s main service, forcing it to be taken down. By the time the deletion was stopped, only 4.5 Gb of the 300 Gb trove of data remained.

Reading their log of the incident the scale of the disaster unfolds, and we can’t help wincing at the phrase “out of 5 backup/replication techniques deployed none are working reliably or set up in the first place“. In the end they were able to restore most of the data from a staging server, but at the cost of a lost six hours of issues and merge requests. Fortunately for them their git repositories were not affected.

For 707 GitLab users then there has been a small amount of lost data, the entire web service was down for a while, and the incident has gained them more publicity in a day than their marketing department could have achieved in a year. The post-mortem document makes for a fascinating read, and will probably leave more than one reader nervously thinking about the integrity of whichever services they are responsible for. We have to hand it to them for being so open about it all and for admitting a failure of their whole company for its backup failures rather than heaping blame on one employee. In many companies it would all have been swept under the carpet. We suspect that GitLab’s data will be shepherded with much more care henceforth.

We trust an increasing amount of our assets to online providers these days, and this tale highlights some of the hazards inherent in placing absolute trust in them. GitLab had moved from a cloud provider to their own data centre, though whether or not this incident would have been any less harmful wherever it was hosted is up for debate. Perhaps it’s a timely reminder to us all: keep your own backups, and most importantly: test them to ensure they work.

Thanks [Jack Laidlaw] for the tip.

Rack server image: Trique303 [CC BY-SA 4.0], via Wikimedia Commons.

TruffleHog Sniffs Github for Secret Keys

Secret keys are quite literally the key to security in software development. If a malicious actor gains access to the keys securing your data, you’re toast. The problem is, to use keys, you’ve got to write them down somewhere – oftentimes in the source code itself. TruffleHog has come along to sniff out those secret keys in your Github repository.

It’s an ingenious trick — a Python script goes through the commit history of a repository, looking at every string of text greater than 20 characters, and analyzing its Shannon entropy. This is a mathematical way of determining if it looks like a relatively random string of numbers and letters. If it has high entropy, it’s probably a key of some sort.

Sharing source code is always a double-edged sword for security. Any flaws are out for all to see, and there are both those who will exploit the flaws and those who will help fix them. It’s a matter of opinion if the benefits outweigh the gains, but it’s hard to argue with the labor benefits of getting more eyes on the code to hunt for bugs. It’s our guess though, that a lot of readers have accidentally committed secret keys in a git repository and had to revert before pushing. This tool can crawl any publicly posted git repo, but might be just as useful in security audits of your own codebase to ensure accidentally viewable keys are invalidated and replaced.

For a real world example of stolen secret keys, read up on this HDMI breakout that sniffs HDCP keys.

Automate Git and Upgrade Your Battle Station With a Custom Peripheral

[mfaust] wakes up in the morning like a regular person, goes to work like a regular person, types in tedious commands for his software versioning utilities like a regular person, and then, as a reward, gets his coffee, just like rest of us. However, what if there was a way to shorten the steps, bringing us all closer to the wonderful coffee step, without all those inconvenient delays? Well, global industry is trying its best to blot out the sun, so mornings are covered there. [Elon Musk’s] thinktank proposed the hyperloop, which should help with the second step. [mfaust] built a control station for his versioning software. Raise your cup of joe high for this man’s innovative spirit.

He first laid out all the buttons, LED lights, and knobs he’d like on a panel to automate away his daily tasks. Using photoshop he ended up with a nice template. He laminated it to the top of a regular project box and did his best to drill holes in the right places without a workshop at his command. It’s pretty good looking!

Since this is the sort of thing an Arduino is best at he, in a mere two tries, wired everything up in such a way that it would all cram into the box. With everything blinking satisfactorily and all the buttons showing up on the serial out, he was ready for the final step.

Being a proficient and prolific enough developer to need a control panel in the first place, like a sort of software DJ, he wrote a nice interface for it all. The Arduino sits and waits for serial input while occasionally spitting out a packet of data describing its switch status. A Java daemon runs in the background of his computer. When the right bits are witnessed, a very nicely executed on screen display reports on the progress of his various scripts.

Now he can arrive at the hyperloop terminal during the appropriate work time slot in Earth’s perpetual night. After which he simply walks up to his computer, flips a few switches, glances quickly at the display for verification, and goes to drink some nice, hydroponically grown, coffee. Just like the rest of us.

Hackaday Links: June 7, 2015

I’ve said over and over again that Apple’s MagSafe port is the greatest advancement in laptop tech in the last 15 years. Those charger connectors break, though, so how do you fix it? With Lego, of course (Google translatrix). Use a light-colored 1×4 brick so the LED will shine through.

Want to learn Git commands? Here’s a great game that does just that. It’s a really well-designed game/tutorial that walks you through basic Git commands.

Lets say you’re just slightly paranoid about the Bad Guys™ getting into your computer with 0-days and roller blades. You’d like to connect this computer to the Internet, but you don’t want to leave it connected all the time. The solution? A timer for an Ethernet switch. It’s actually a better solution than doing the same thing with scripts: there’s a real, physical interface, and if the Bad Guys™ get in when you are connected, they could just enable the network adapter anyway. An extremely niche use case, but that’s 99% of the security hacks we see.

The DaVinci 3D printer is an okay printer if you’re cool with the Gilette model. The filament cartridges are chipped, and the software is proprietary. These problems have been solved, and now you can use a standard RepRap heated bed and glass with the DaVinci. At this point, people are buying the DaVinci just to tear it apart.

Monitor GitHub Activity with an RGB LED Matrix

tim-display

Ever wonder who is forking your code? [Jack] did, so he built a real time GitHub activity display for his company’s repositories. The display is based a Wyolum The Intelligent Matrix (TiM) board. The TiM is an 8 x 16 matrix of the ubiquitous WS2811/Smart Pixel/NeoPixel RGB LEDs with built-in controller. We’re seeing more and more of these serial LEDs as they drop in price. Solder jumpers allow the TiM to be used as 8 parallel rows of LEDs (for higher refresh rates), or connected into one long serial chain.

[Jack] wasn’t worried about speed, so he configured his board into a single serial string of LEDs. An Arduino drives the entire matrix with a single pin. Rather than reinvent the wheel, [Jack] used Adafruit’s NeoMatrix library to drive his display. Since the TiM uses the same LEDs as the Adafruit NeoPixel Matrix, the library will work. Chalk up another victory for open source hardware and software!

An Electric Imp retrieves Github data via WiFi and passes it on to the Arduino. This is a good use of a microcontroller such as the AVR on the Arduino. [Jack’s] display has a scrolling username. Every step in the scroll animation requires all the pixel data be clocked out to the TiM board. The Arduino can handle this while the IMP takes care of higher level duties.

Continue reading “Monitor GitHub Activity with an RGB LED Matrix”

Git with Eagle: Add meaning to Diff

a-glimpse-of-git-with-eagleWe love Git. We know everyone has their favorite version tracking tools. But even those that don’t care for Git should see the value of getting meaningful Diff data from tracking Eagle layout files.

Was that last sentence just gibberish to you? Let’s take a step back. A few years ago it was impossible to use version control with Eagle at all because the schematic and PCB layout software used to save its files as binaries. But then Cadsoft transitioned to saving Eagle files as XML. This opened the door for things like scripting to rename parts en masse and to track the files under version control. One problem with the latter has been that performing a Diff on two different versions of a file results in XML changes that are probably not human readable. [Patrick Franken] wrote this script to add at least a glimmer of meaning.

We’d love to see some kind of side-by-side highlighting on the schematic or board renderings themselves. But that’s quite a ways off if we ever actually see it. For now his script will take the Diff and print out the tables seen above denoting which types of changes were made from one version to the next. It’s a start, and we hope it inspires even more work in this area.

Carry a Git server in your pocket

android-git-server

We love using Git for its superior version control. We often host our more advanced projects in a public Github repository. But the bulk of our little experiments are simply local repos. This is fine if you’re always at home, but if we are away from home we find ourselves having to SSH into our server to copy over the Git files. [Andrew] found a way around this slightly awkward process. He used an old Android phone as a Git server.

This actually makes a lot of sense when you start to think about it. Most Android phone have a microSD card slot to provide a huge storage bin (the lack of this on the Nexus 4 is baffling) so you don’t need to worry about running out of space. All of these devices have WiFi, making it easy to use them as an AP when there isn’t any other WiFi around. And the web-connected nature of the device will make syncing your repo over the Internet a snap.

Most of the behind the scenes work is done using Debian packages. This provides a few issues which [Andrew] walks through one by one. We also like his pointers like using ‘noatime’ on your EXTx file systems to avoid wear on the SD card.