The xz backdoor is naturally still the top story of the week. If you need a refresher, see our previous coverage. As expected, some very talented reverse engineers have gone to work on the code, and we have a much better idea of what the injected payload does.
One of the first findings to note is that the backdoor doesn’t allow a user to log in over SSH. Instead, when an SSH request is signed with the right authentication key, one of the certificate fields is decoded and executed via a system()
call. And this makes perfect sense. An SSH login leaves an audit trail, while this backdoor is obviously intended to be silent and secret.
It’s interesting to note that this code made use of both autotools macros, and the GNU ifunc, or Indirect FUNCtions. That’s the nifty feature where a binary can include different versions of a function, each optimized for a different processor instruction set. The right version of the function gets called at runtime. Or in this case, the malicious version of that function gets hooked in to execution by a malicious library.
Who is Jia Tan?
One of the more intriguing mysteries is the real identity of the programmer or programmers behind the Jia Tan identity. I’m of the opinion that it’s more than one developer, we’ll get into why in a bit. The first clue is the timezone on the Jia commits, which is UTC +8. That’s the timezone of China, Mongolia, part of Russia, Western Australia, and the island nations in-between. East Asian name, East Asian timezone, so it must be an East Asian hacker, presumably Chinese. Except, the commit times don’t make any sense.
I’ve seen some theories that Jia was committing in a +2 or +3 timezone. Nope. Check the git log for yourself, and see the graph. There’s an obvious pattern here, starting at 12:00 noon UTC. Now which time zone would that correspond to an office worker starting work at 8:00 AM? UTC -4 is the Eastern US — but only during half the year. What’s notably missing from that chart is the twice-a-year time change for Daylight Savings Time that the US endures. It’s also worth noting that git commit times are easy to change before making a push.
And then there’s the series of changes with oddball times towards the end of the graph. Of interest are the version bumps to 5.6.0 and 5.6.1, and also the Valgrind “fix”. Which may possibly suggest a second programmer controlling the Jia Tan puppet account. There is a notable gap in commits right around the Christmas/New Year holiday each year, too. But all things considered, there is no smoking gun pointing at a specific country. Just a few hints. But the complete lack of information about Jia Tan in any other venues is a good sign that it’s a fabricated persona, and using pretty good opsec at that.
So What Now?
The next question is, what can we do to keep this from happening again? Developers and distro maintainers are definitely thinking through this question right now. One conclusion is that tarballs cannot be thoughtlessly trusted. Checking tarball contents against git tags, and regenerating the autotools
output are good, too. Several developers are looking hard at moving away from hard-to-understand tools like gnulib
and autotools
altogether.
The Debian policy of requiring in-person meetings before GPG attestation suddenly seems like a good policy. Perhaps this should become an industry standard practice to gain maintainership in any existing open-source project.
An interesting point was made at about 37 1/2 minutes into this week’s FLOSS Weekly interview, where Joshua explained that only two or three people have commit access to the Asterisk codebase, and only Sangoma employees are permitted to be project maintainers. Will more corporate sponsorship of individual Open Source projects be part of the solution? Time will tell.
AT&T Finally Owns Up
We now know that back in 2019, customer data from AT&T customers was grabbed in a breach. What’s notable here is that it took until this March for AT&T to publicly acknowledge that it was indeed data leaked from their customer records. The way this was confirmed is particularly impressive.
Credit goes to [Sam Croley] for figuring this out. Each account record included address, phone number, birthdays, and Social Security number. And then for each account, a numeric passcode was encrypted and included. That encryption is strong enough to prevent properly cracking it. (We’d bet it’s actually a hashing function.) But [Sam] is particularly clever. He took just the encrypted values, sorted, and de-duplicated them. The result? just about 10,000 unique records, pretty much what you would expect if 70 million users all picked a 4 digit pin. The hashing or encryption algorithm used here wasn’t keyed or salted per entry.
And as a result we get part two of the world’s greatest crossword puzzle — the numeric edition. To put it simply, every time a customer chose 1337 as their PIN, the value in the leak was the same. All it takes is a few users out of 70 million with a phone number, birthday, or social security number ending in 1337 to figure out that’s what a given hash value corresponds to. And even if you picked that number completely randomly, it still gets figured out with all the rest.
Hackers’ Letters of Marque
There is a fascinating saga playing out, along side the ground war in Ukraine: Hackers have been given a sort of Letter of Marque from the Ukranian government, and now some physical letters have been sent out as well. Drawing on my knowledge of history gained from early computer games like Sid Meier’s Pirates!, I know that a Letter of Marque is authorization from a government given to a private citizen to attack and claim foreign ships and assets. So what we have here is a modern, digital take on the idea. It may be worth noting that Ukraine is not a signatory to the treaty that banned the practice.
Typosquatting Adds AI Hallucinations
It seems like we need to add another entry to the standard list of warnings given to Linux users. Don’t run code copied directly from a website, and now, don’t run code copied directly from ChatGPT. Apparently there’s a commonly hallucinated pip package, huggingface-cli
, referring to part of the huggingface project. That package doesn’t exist, but the command has sneaked into a few online guides anyways.
And that’s where the typosquatting comes in. Thankfully this time it’s by a security researcher rather than a malware author. [Bar Lanyado] grabbed that package name as a test, and scored 15,000 download attempts in a mere three months. So far there isn’t a real attack using this technique, but it’s likely just a matter of time.
But you don’t need AI for traditional typosquatting: PyPI finds itself again in the cross-hairs. This is a collection of 566 fake packages, in an automated campaign. The uploads happened in a pair of bursts over a couple days, and ended when PyPI maintainers suspended both new project and new user creation.
Bits and Bytes
The DOMPurify project is intended to take “dirty” HTML and turn it into a string of sanitized, XSS-free HTML. There was a little catch, in that HTML code is a bit different from XML. XML is much stricter in what it considers parseable tags, but will pass the contents of those tags, even if it contains valid HTML. The takeaway is that researchers found several DOMPurify bypasses that took advantage of this parsing gap. Patches have been applied, so the up-to-date DOMPurify package should be good to go.
Filed in the “awkward” category, the OWASP foundation was accidentally hosting some very old member resumes available to the public. It’s awkward because OWASP is the foundation dedicated to improving open source security. The upside is the information leaked is so out of date, most of the people affected can’t even be contacted. So we get a public announcement.
Microsoft has received a reaming from the Cyber Safety Review Board over the 2023 hack of hosted Exchange accounts. It appears there are still unanswered questions, like exactly which crash dump contained the key used in the final attack. Still, it’s not a good day when your actions are described as a “cascade of … avoidable errors.”
“Perhaps this should become an industry standard practice to gain maintainership in any existing open-source project.”
Takes away the open-source advantage anyone can contribute.
Everyone can contribute, but maybe only a verified developer should get commit access.
Anybody should be able to fork and do interesting things or submit a ‘I found a problem, this is my suggested fix’ type of message. But it would require one of the official maintainers to actually review and accept that change into the parent project that everyone relies on.
So I don’t think it will make any difference, to the ‘anyone can contribute’ really, just perhaps slows down the spread of a good contribution.
If I am reading the git commit chart correctly, there are some other notable commit log gaps. 4th O’July sees a week off. Memorial Day sees a week+ off. Labor Day sees ~2weeks off (maybe they attended burning man).
> Labor Day sees ~2weeks off (maybe they attended burning man).
… or Dragon*Con.
If I was running a team of state actor hackers, I’d put them on another country’s time zone and holidays. Seems pretty basic. So I’m not sure we can learn too much from this.
This!
“Let us make a cunning plan to conquer the world and follow it for some years. And HR is reminding you, that office hours are from 9 to 5 and don’t forget the paid holidays!”
Yes, I would do the same. But if you ran a team of state actor hackers, you would be a civil servant in charge of other civil servants and so much of the ‘I know how to get around this’ falls on the floor due to legal and cultural factors. Also, civil servants are notoriously inflexible around work hours and holidays.
Also, if this was a ‘blame the USA’ false flag, I imagine they would pick a more western sounding name. Thanks a lot cheaper the overtime and holiday pay.
Other notes:
The “time off” blocks seem to roughly equate to federal employee vacation quota.
No significant time off for Lunar/Chinese new year.
However in 2023 there is time off around the Chinese Golden Week (Oct 1st to 7th), although they did work during the Golden Week in 2022 an 2021, so this might not be a strong signal.
Maybe they had a tool which would make commits and send emails on national holidays when staff where out, but that feels a little far fetched for civil servants.
The are a lot of hints that it’s not Chinese, despite those who want it to be. It’s not like they’re aren’t other states with expertise and motivation or mercenary hackers like those NK employs.
When you consider the potential impact of having root command access to basically the entire internet as virtually all servers are Linux based the amount of effort that would be put into this idea, and keeping it quiet enough to get this far it could easily have come from anybody. You don’t have to actually be in China to work for them or keep to the ‘normal office hours’ of the place you live etc.
All you need to pull this off is a single bright spark that notices the vulnerable point and can get the funding and any help they need on the social engineering to pull it off. Not impossible that no nation state was involved at all, might even be the most likely situation though I’d not like to rule out anybody – detected too soon and without enough information to really draw any conclusions IMO.
I really hope AT&T gets in a lot of trouble for not salting, that’s huge potential for abuse if even a small percentage of pins get worked out, if all 70 million get decrypted it’s a truly epic fail.