In breaking news that dropped just after our weekly security column went live, a backdoor has been discovered in the xz package, that could potentially compromise SSH logins on Linux systems. The most detailed analysis so far seems to be by [Andres Freund] on the oss-security list.
The xz release tarballs from 5.6.0 in late February and 5.6.1 on March 9th both contain malicious code. A pair of compressed files in the repository contain the majority of the malicious patch, disguised as test files. In practice, this means that looking at the repository doesn’t reveal anything amiss, but downloading the release tarballs gives you the compromised code.
This was discovered because SSH logins on a Debian sid were taking longer, with more CPU cycles than expected. And interestingly, Valgrind was throwing unexpected errors when running on the liblzma library. That last bit was first discovered on February 24th, immediately after the 5.6.0 release. The xz-utils package failed its tests on Gentoo builds.
One of the xz maintainers, [Jia Tan], weighed in on that Gentoo bug, suggesting that it was a GCC bug causing the Valgrind errors. This is the same developer that pushed the malicious archive files and minted the tainted releases. And as if to clear up any remaining doubts, the developer doubled down in a GitHub commit, working around the Valgrind errors, and linking to a completely unrelated GCC bug report claiming it to be this issue.
At this point, the only reasonable conclusion is that the person in control of the [JiaT75] GitHub account is a malicious actor and is completely untrustworthy. What’s unclear is if this is still the same developer that has been co-maintaining the project since August 2022. It’s possible that [Jia Tan] has always been a bad actor, or that account may be completely compromised.
What About SSH?
What may not be clear is the connection to SSH. And it’s a trip. Many Linux distros patch sshd to add systemd features, and libsystemd pulls the liblzma library. That means the liblzma initialization code gets run when sshd starts. In the malicious code, the library checks argv[0], which is the name of the program being executed, for /usr/bin/sshd. Additionally it seems to check for debugging tools like rr and gdb. If the checks are green, liblzma replaces a few function calls with its own code. It’s a complicated dance, but the exploit is specifically looking to replace RSA_public_decrypt.
That’s a very interesting function to clobber, as it is one of the functions used to validate SSH keys. It’s not hard to imagine how malicious code here could check for a magic signature, and bypass the normal login process. The full analysis is still being done, and expect more information in the coming days.
But the bottom line is that a machine with a patched sshd binary, that also has xz packages version 5.6.0 or 5.6.1, is vulnerable to unauthenticated SSH logins. The good news is that only a few distributions have shipped the 5.6.x series of xz packages. Fedora Rawhide/41 them, and Debian Testing and Unstable shipped these versions as well. If you’re on an affected system, look for an update right away.
It’s unclear what the path forwards is for the xz project. This is obviously an important system utility for Linux systems, and its current maintainers seem to be asleep at the wheel — or intentionally steering towards disaster. Expect one or more hard forks, and then a lot of cleanup work.
This is a developing story. For more, see the Redhat security alert, the Debian alert, and the oss-security thread on the subject.
>It’s unclear what the path forwards is for the xz project
Yes, im sure “libsystemd pulls the liblzma library” is blameless
systemd has assimilated so much, it is now a massive attack surface. I worked out that one thing that systemd does far better than anything else.
Add another entry into the “why systemd is an abomination” omnibus. Advocates will rail against sane “alternatives” until they’re blue in the face, but they still can’t argue against systemd’s massive attack surface.
It almost seems deliberate at this point.
Oh, they will. It’s more like a cult than a piece of software at this point.
I’m down for blaming systemd when it’s at fault but rpm itself depends on liblzma.
Does it also depend on the RSA_public_decrypt?
systemRichard has incorporated everything and the kitchen sink, and so the vulnerabilities of the kitchen sink, too.
Not specifically but it’s pretty easy to imagine how patching RPM with a compromised liblzma is just as much a vector for unauthorised remote access given that RPM maintains all system packages on relevant distributions, including, y’know, sshd
The main issue is that most of the places where liblzma gets pulled in without introspection are due to libsystemd0 depending on it.
A build of openssh that isn’t built to use libsystemd0 is also not linked against liblzma. Even when linked against libsystemd0, sshd is not directly linked against liblzma. Instead, libsystemd depends on it.
Also, it looks like the xz acting maintainer is an artificial identity, and that xz/liblzma was targeted specifically because it’s early in the runtime dynamic symbol resolution, so it can manipulate the symbol tables before they’re “sealed”/made read-only.
In other words, the bad actor found a useful vector, and spent a couple of years building trust, to gain the opportunity to control that vector.
libsystemd0 is a reverse dependency of almost everything on a modern Debian box, thus is a great place to start looking for an attack vector, no matter what your actual target is.
apt-cache rdepends libsystemd0 on my system reports 349 packages (mostly services, daemons, or desktop-related). Most are involved in separation-of-authority or security contexts in one way or another.
Example of the library dependency tree for sshd on a debian stable system (too old to be affected, but demonstrates the attack vector):
/usr/sbin/sshd
├── libcrypt.so.1 [ld.so.conf]
├── libz.so.1 [ld.so.conf]
├── libcrypto.so.3 [ld.so.conf]
├── libcom_err.so.2 [ld.so.conf]
├── libkrb5.so.3 [ld.so.conf]
│ ├── libk5crypto.so.3 [ld.so.conf]
│ │ └── libkrb5support.so.0 [ld.so.conf]
│ ├── libresolv.so.2 [ld.so.conf]
│ ├── libkeyutils.so.1 [ld.so.conf]
│ ├── libkrb5support.so.0 [ld.so.conf]
│ └── libcom_err.so.2 [ld.so.conf]
├── libgssapi_krb5.so.2 [ld.so.conf]
│ ├── libkrb5.so.3 [ld.so.conf]
│ ├── libkrb5support.so.0 [ld.so.conf]
│ ├── libcom_err.so.2 [ld.so.conf]
│ └── libk5crypto.so.3 [ld.so.conf]
├── libselinux.so.1 [ld.so.conf]
│ └── libpcre2-8.so.0 [ld.so.conf]
├── libsystemd.so.0 [ld.so.conf]
│ ├── libcap.so.2 [ld.so.conf]
│ ├── liblz4.so.1 [ld.so.conf]
│ ├── libzstd.so.1 [ld.so.conf]
│ ├── liblzma.so.5 [ld.so.conf] <——- ATTACK VECTOR
│ │ └── libpthread.so.0 [ld.so.conf]
│ └── libgcrypt.so.20 [ld.so.conf]
│ └── libgpg-error.so.0 [ld.so.conf]
├── libpam.so.0 [ld.so.conf]
│ └── libaudit.so.1 [ld.so.conf]
│ └── libcap-ng.so.0 [ld.so.conf]
├── libaudit.so.1 [ld.so.conf]
└── libwrap.so.0 [ld.so.conf]
└── libnsl.so.2 [ld.so.conf]
└── libtirpc.so.3 [ld.so.conf]
└── libgssapi_krb5.so.2 [ld.so.conf]
The real problem is that distro maintainers shouldn’t just be trivially patching sshd to pull in more dependencies to fix minor issues. It was bound to result in something like this eventually.
It pulls if you tell</a≥ it to.
>It’s unclear what the path forwards is for the xz project
This is an unfortunate way of seeing things.
I am convinced this has happened before and it’s becoming more common. Blaming ‘them’ while thinking this would never happen to ‘us’ is making everyone more vulnerable to these attacks. This can happen to you, and if the open source project you maintain is placed im an interesting place there will be attempts.
Code reviews are not effective at catching these issues. Remember the kernel bugs introduced Minnesota university, in fact, every bug in software that uses reviews systematically passed the review. We need better systems to find these issues, we need software architectures to contain their impact, and we need them yesterday.
I remember the firestorm it caused and it (mostly) wasn’t about code reviews and their deficiencies.
Maybe something along the lines of including fuzzing in unit-tests?
I’m not much of a coder so the idea is at most half-baked but the way I understand your criticism that idea ^^ came up and passed my *smell test*…
xz was fuzzed.
Jia disabled a feature that would have caught it, with some plausible-sounding excuse: https://github.com/google/oss-fuzz/commit/d2e42b2e489eac6fe6268e381b7db151f4c892c5
I think it’s more about the xz maintainer starting this saga burned out, and the only help they got was malicious. I doubt this will make them less burned out, and I doubt this will make people help them, so what’s the path forward?
One more reason to only use RTEMS or any other lightweight system where you can inspect ALL the code, not just some single throat-smearing executable.
Umm, you want to replace a full blown rich desktop environment with an RTOS?
Is this a big downside to open source we will see more of in the future?
Much easier for “bad actors” to embed malware deep in the system than becoming an employee of a big tech company and doing so from there, or writing a virus that has to go propogate itself and go undetected.
Absolutely. This issue is a successful proof if concept. Much more developers will be working on backdoors from now on.
I doubt it will really be ‘a big downside’ – yes it clearly happens, even takes a while to get caught from time to time but it does get caught eventually (or just overwritten by new code as progress is made), and the quality of code usually required to be accepted into a OS project makes avoiding accidental errors and following the intended logic easier.
With closed source the bad actor (which may even be the company itself) can do what it likes, or just code really shoddily thanks to huge time pressures on the developers to please the shareholders and nobody outside those directly involved with making it that way really know. But as seen plenty of times the white/black hats will notice something and if they don’t disclose it it could take a major event for anybody else to look hard enough to trace the issue.
Or the short versions – obscurity isn’t security, Open source will usually get spotted and fixed – in many cases long before the bad new stuff gets to the LTS distro of most normal user, where in closed source you probably won’t get fixes quickly or even know how and why your system was broken…
Exactly. People still think that big companies are somehow better than open source projects. Just look at Meta and the recently revealed court case documents stipulating they wholeheartedly went with a full blown man in the middle attack, at least against Snapchat, but could be worse than that. Sure, they got fined, a measley $90 mil, not really a deterrent to stop them for doing similar in the future.
Is it? That is actually that is an _upside_ of opensource. With open source someone has the chance of spotting the malicious code, report it and get it fixed. And yes, we will see more of that in the future. I hope so. I hope vulnerabilities are spotted and promptly fixed instead of hidden. With proprietary code it will stay there forever nobody will ever know.
Maybe, but the hackers demonstrate all the time that “proprietary” isn’t the barrier people think it is. And one might even argue the latter requires a deeper understanding than open-source presents.
I’d argue that what happened here is the complete opposite of what you’re arguing. In this case, a malicious actor managed to take over a project which is used by other large projects – and, while that could lead to mass compromise, it also meant that that project had extreme visibility, causing concern when code fuzzers like Valgrind started throwing seemingly spurious errors.
The malicious actor’s method of hiding the exploit to evade early detection proved to be his downfall, and that’s because so many people were involved in this process.
That’s a resounding endorsement of open source if there ever was one.
Lets face it: if there is an organized group of bad actors (controlled by governments) with a carefully orchestrated approach, then none would notice for a long time.
I wouldn’t attack the OS itself. Way too many egomaniacs searching for honor and exposure.
Capture an end-user project that is well liked and almost dead. Something like an editor, game thingy, artist tool, music/video cutter. Tools that not so tech-savvy users download, update, install mods and plugins for without much thinking.
Slowly, very slowly install your own group of bad actors. Collect data. Who of us hasn’t his “passwords.txt” on their desktop?
> I wouldn’t attack the OS itself.
That may well have happened here. Apparently JiaT75 may have made some recent xz commits to the kernel as well. We’ll likely know soon whether they’re malicious or not – I have very little doubt that everything he’s touched is being scrutinized closely.
I stand by what I said – while there seems to be a good chance that JiaT75 is a state actor (which further means that ‘he’ is more likely a ‘they’, being an actual team behind the JiaT75 account), this shows that even state actors will have a hard time subverting open source code. Impossible, no, but the more central any given project is to other projects, the harder it’ll be to sneak changes like this under our noses – and yet, for state actors to make a given attack worthwhile, they need to find projects like xz, with little maintainer involvement and central to larger projects.
@Jan I’d have to suggest the way open source works doesn’t easily allow for a new group of state sponsored computer terrorist to take over anything important – look at the pushback something like systemd gets, where the developer is a known ‘trusted’ entity. To get something actually new or manage to cuckoo into an existing important project to the level it won’t be relatively easily noticed or left entirely unused as ‘we are quite happy with the current system thanks’ would be the work of decades probably.
And as you would have to actively be putting out good contributions that actually benefit everyone without revealing your long term goal to screw it all up, surviving the scrutiny and inevitable patch/feature submitted by somebody outside the group you can’t actually ignore, because if you do your project is dead and the fork isn’t yours… The web of trust can’t easily be invaded and broken without being noticed, unless you are targeting something so on the periphery – like perhaps a specific driver for a family of USB devices that are probably only really meant to be sold inside the authoritarian regime… As there odds are great almost nobody has the skills, and cares about these devices to look.
Much harder for bad actors to hide code in a codebase that they don’t know where it will end up, who compiles it on what architecture and with what, than hiding in a large “secret” codebase that will become a Big Product.
Hiding in plain sight is hard. Hiding in plain sight while your hiding place is a playground, a work bench and a meeting point at the same time is even harder.
name@PC:~$ liblzma –version
liblzma: command not found
name@PC:~$ xz-utils –version
xz-utils: command not found
So I’m safe from this?
liblzma is a library, not an executable.
xz-utils is a package, not an executable.
If you’re on a Debian or Ubuntu-based distro (any distribution which uses .deb packages, and apt/apt-get/synaptic for package management), you can do:
$ dpkg -l liblzma\*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================
un liblzma2 (no description available)
ii liblzma5:amd64 5.4.1-0.2 amd64 XZ-format compression library
Debian stable (bookworm, Debian 12) isn’t vulnerable, as it still uses liblzma5 5.4.x. Only Debian unstable (sid) and testing pulled 5.6.0 in – and those are now reverted to 5.4.5, if you’ve applied updates since this issue became public yesterday:
$ dpkg -l liblzma\*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-================-===================-============-=================================
un liblzma2 (no description available)
ii liblzma5:riscv64 5.6.1+really5.4.5-1 riscv64 XZ-format compression library
Note the version here is ‘5.6.1+really5.4.5-1’, since apt doesn’t allow package downgrades without causing significant issues. If you see a package version like that, you should be fine.
Shows the sense in Linux Mint’s philosophy of being behind the curve with updates, users of bleeding edge distros have discovered this nasty before it reaches us.
xz -V should let you check if you’re affected, Mint 21 ain’t.
Any stable (non-rolling, production-ready) distro should similarly be unaffected. Debian 12 standardized on liblzma5 5.4.x, so it will never be affected, unlike Debian unstable and testing (which pulls packages in from unstable until the freeze occurs when the devs get ready to push it to production).
Apparently Arch had pulled the compromised package in, given that it’s a rolling distribution – but since the exploit explicitly looked for deb and rpm package builds (not to mention Arch’s openssh doesn’t have a reliance on liblzma, unlike Debian and RH!), it doesn’t appear to have included the exploit in the package build for Arch.
xz a dependency in many many MacOS Homebrew packages (22 installed packages on my system). Today I ran ‘brew update’ and it downgraded my xz:
==> Upgrading 2 outdated packages:
…
xz 5.6.1 -> 5.4.6
Just check macport installed packages:
port installed
…
xz @5.4.6_0 (active)
…
So safe version installed for the moment.
Now, this post explains that the malicious code in liblzma is checking if it has been runned explicitely by sshd (and if rr and gdb are available), and in this case (only?) is doing its nefarious job.
So it seems that even if anything else than ssd is running infected versions of xz, nothing bad should happen.
And since there’s no sshd port in macports, it seems quite unlikely to have any problem on macOS because of this virus.