On February 20th, servers hosting the Linux Mint web site were compromised and the site was modified to point to a version of Mint with a backdoor installed. Very few people were impacted, fortunately; only those who downloaded Mint 17.3 Cinnamon on February 20th. The forum user database was also compromised.
What is most impressive here is not that Linux Mint was compromised, but the response and security measures that were already in place that prevented this from becoming a bigger problem. First, it was detected the same day that it was a problem, so the vulnerability only lasted less than a day. Second, it only affected downloads of a specific version, and only if they clicked a specific link, so anyone who was downloading from a direct HTTP request or a torrent is unaffected. Third, they were able to track down the names of three people in Bulgaria who are responsible for this hack.
As far as the forum compromise, the breech netted usernames, emails, and encrypted passwords, as well as personal information that forum users may have entered in signatures or private messages. It’s always nice to see when compromised sites are not storing passwords in plain text, though.
There is one security measure which should have protected against this and failed for a couple of reasons, and that’s the signature. Normally, the file download is accompanied by a signature which is generated from the file, like an MD5 or SHA checksum. By generating the checksum of the downloaded ISO file and comparing it to the reported signature on the web site, one can confirm that the file has downloaded correctly and that it is the same file. In this case anyone downloading the bad ISO should have caught that the downloaded file was not the official one because the signatures did not match. This can fail. Most people are too lazy to check (and there is no automated checking process). More importantly, because the attackers controlled the web site, they could change the site to report any signature they wanted, including the signature for the bad ISO file.
If you are affected by this, you should change your password on the forum and anywhere you use the same email/password. More importantly, as great as the verification signature is, shouldn’t there be a better way to verify so that people use it regularly and so that it can’t be compromised so easily?
Digest != Signature
Exactly. Digests are pretty much as vulnerable as the file itself. It wouldn’t take much more for the attacker to replace the digest alongside the file to match.
Signatures, however, are cryptographically linked to the signer’s key which is usually kept very safely.
Yeah. Digests (MD5, SHA1, etc.) are only useful when posted in a separate place, and even in that case mostly just help the aftermath after a breach. I post mine to google groups, which is hopefully relatively hard to modify after-the-fact.
Very pleased that Mint disclosed the breach so quickly.
Ye,as someone that uses Mint on all three of my machines I’m appalled this was allowed to happen, but indeed pleased it was dealt with swiftly and transparently.
“I’m appalled this was allowed to happen”
That is a bit harsh. Do you think they knew about the security weakness and did nothing to fix it and invited the hackers in?
I would certainly hope they were not aware of the security weakness and chose to ignore it, on the other hand I also hope that they would have considered the possibility and been more proactive. However, and more to the point, the linux world in general has been relatively free of this sort of nonsense, and now that is a critical mass of users, I suspect that we will be targeted more often, and it is that, more than anything else, is what I am appalled about.
I strongly disagree, it is impossible to make a site unhackable, their are always unknown vulnerabilities, so what’s truly important is how quickly they shut down the hack and patch the holes. In this case they did a superb job, and I believe the hack came about through no fault of their own.
Did you even bother to read what I wrote?
I did, I believe that the hack came about through no fault of their own, and that they couldn’t have actually prevented it, not that it’s mostly not their fault.
I ignored the second part of your comment because I don’t have anything to add on that point. If I overlooked any other part of your comment, I would prefer if you would be more specific.
Well, yes, sort of. Go look at the history of linuxmint.com at Netcraft. Right after the breach they apparently made an update to Apache 2.4.7 on Ubuntu, but the last record before that, with a November timestamp, shows linuxmint.com running Apache 2.2.9 (which dates from 2008) on Debian Lenny (which hasn’t had security updates since 2012). That’s really just negligent.
Time to replace my Mint forum password.
Assume your password was cracked. They were hashed but only relatively weakly (phpass).
What about update?
As soon as an update is signaled I install it? Could the faked site have reported an update that contained the backdoor?
Update is unaffected. This was only a link to an ISO file, which is not related to the update process.
the public-key cryptography used for signing each package means repos can get thrashed, and apt was designed to detect when it sources were compromised. Blessings upon our BSD and GNU forefathers for they were truly wise.
This is why the kids that did this chose to drop an additional payload rather than attempt to break the echo system of the cryptography used for package managers (way harder to pull off).
It is a good idea to manually download debsums.deb from your trusted repo and scan to check if anything like apt sig files were jacked…
> sudo debsums -as
Note configuration files you changed yourself will show, but they should still be checked for lameness.
I doubts that the crooks are actually Bulgarian. They just used Bulgarian hosting, and the IP reservation info is obviously made up: Pernik 97 str., appartment 18, Sofia
Crooks come from all countries. But you are right that the ip might be bogus.
They are just jealous that the HaD conference is in Belgrade and not Sophia. B^)
Where ever they come I hope they are found and identified. Ideally I want to know who they are, where they have worked and will be working as they changes jobs. Then what??? Boycott dealing with any company that hires them in a role where they access to computers. In my twisted mind Linux is sacrosanct and it’s as vile as pedophile clergy.
… Excuse me?
are you kidding, he’ll probably get hired by Microsoft, or NSA, or …., there’s some other motives.
Arch uses PGP signatures on the packages and the monthly ISO to help prevent this kind of thing
I sometimes think that a totally septate website with no php, no databases, no fancy stuff at all, that just presents the basic text of md5/sha1/sha256 hashes of file names and nothing else would be very nice.
%s/septate/separate/g
Would a blockchain of ‘distributions’ work – each time a distribution needed to do a new push they could add it to the blockchain and have the rest authenticate it?
Why complicate things when what exists already works well enough?
The attack was really crude and ineffective in the first place. It’s rather a shame for Mint that it was possible in the first place – as far as security breaks go, it falls in the category of “Oops I forgot the CD of state secrets on the bus”.
repeat after me… “blockchain is not the answer”. It is a bad solution (vulnerable to a concentration of hashing power, as we’ve seen) to the wrong question (distributed agreement, not source authentication), and horrifically inefficient at that.
I’m not sure that what exists does work well enough. And yes, exactly, posting an MD5 digest alongside a link on a website != hack/virus/defacement prevention.
That said, it would also be nice if there was a transparent way to use a digest to (help) ensure that a download was successful and glitch-free. Maybe include an md5=”xxx” value in the tag, so a browser can check auto-magically?
Ugh, should read: …in the (triangle bracket) a (close triangle bracket) tag…
SHA-256 and better only, people
So I just make sure the hash “checksum” on the real Mint download page matches the install archive I downloaded and I would catch this before it ever hit my machine – right? If yes, then that means the people who downloaded and installed the hacked archive file were not following safe practices.
So if I’m not re-using passwords, then all I have to do is change my Mint Forum password and I’m safe? Right?
This is a real shame! These hackers are horrible people! I think one of the problems with a project like Linux mint is that it has many users but only a few centralized maintainers. I think that they might be getting stretched a bit too thin. I love Linux Mint and think that it’s the most usable Linux distro out there. But perhaps the project maintainers should consider delegating things like website maintenance to others that have the time to do it.
BTW in case you didn’t already know, WordPress sucks. For simple blogs I prefer to use static site generators like Hexxo or Pelican. For larger websites with forums e.t.c perhaps a more secure CMS ought to be used.
I’ve been online since 1999. My list of sites I visit and am registered to is massive…In 17 years, my details have never been breached– until Linux Mint forums! AND– I didn’t even download the damned ISO! I think this “breach” by the sheer ease of it is shocking. Great- they figured it out- but it shouldn’t have been that easy. You can check YOUR email for breach here: https://haveibeenpwned.com/