Yet another entry in the “why we can’t have nice things” category, Retbleed was announced this week, as yet another speculative execution vulnerability. This one is mitigated in hardware for AMD’s Zen 3 and Intel Generation 9 and later. For earlier devices the performance hit in mitigation is quite painful. What exactly makes this different from previous weaknesses, and why didn’t the previous mitigations cover this problem?
Spectre V2 abuses a CPU’s indirect branch prediction, to trigger speculative execution of code that shouldn’t be executed in a given context. Even though the CPU eventually catches up and rolls back the bogus execution, there are still fingerprints left behind in the cache contents. The idea is that reading those fingerprints leaks actual data that the attacker process shouldn’t have access to at all. The solution in the Linux kernel is the “retpoline“, a portmanteau of “return” and “trampoline”. This gadget replaces a jmp instruction with some setup, and finally a ret call instead. This looked to be an inexpensive solution to the problem.
What Retbleed brings to the table is a method to poison the speculative execution of these return instructions as well. Their full paper (PDF) describes the technique, which boils down to manipulating a processor to use the vulnerable Branch Target Buffer (BTB) instead of the safer Return Stack Buffer (RSB). On vulnerable Intel systems, this means filling the RSB with enough data to pop the actual target return out of the buffer. When the series of jumps and returns unwind, the final return actually uses the BTB as the RSB has emptied, or underrun. Vulnerable AMD systems seem to simply always use a BTB to predict returns, making the exploitation much easier there.
Windows machines use a more aggressive mitigation policy, Indirect Branch Restricted Speculation (IBRS), which does seem to fully mitigate this particular issue, though there may be more coming in the next few weeks. On Linux, the retpoline mitigation is finally being replaced by IBRS by default, leading to the performance hit discussed above.
If you’re on one of the affected CPUs, there are some kernel parameters you can use to control what mitigations are put in place. retbleed=off uses the existing retpoline mitigation if appropriate, but doesn’t further degrade performance, at the expense of vulnerability to this attack. The default, retbleed=auto will use the full mitigations to make the machine as safe as practical without disabling Simultaneous MultiThreading (SMT). And finally retbleed=auto,nosmt will actually disable SMT on the few models that technically require it for full mitigations. This isn’t set by default, as it is even more crippling to the machine’s performance.
NIST Goes Post-Quantum
While the quantum crypto-Apocalypse has yet to be realized, the various agencies responsible for standards are working to stay ahead of the curve by sponsoring research and then selecting cryptography schemes to be used as next-gen standards. In that vein, NIST has released an update to their Post-Quantum Cryptography Standardization Process. The big news here is that a few algorithms have been selected. Falcon, SPHINCS, CRYSTALS-Kyber, and CRYSTALS-Dilithium. There definitely seems to be some sci-fi geeks among the entrants, based on the names.
CRYSTALS-Kyber is a Key Encapsulation Mechanism (KEM), a method to share a private key using only publicly sent messages, a la Diffie-Hellman. CRYSTALS-Dilithium and the others are signature schemes, useful for verifying data. We look forward to these standards rolling out through the different projects and applications we use daily.
PyPI, 2FA, and a Grumpy Dev
In an effort to avoid security problems, the PyPI repository has rolled out a security policy that requires maintainers of critical projects to use Two-Factor Authentication for their accounts, and are even sending out free hardware keys. The criteria for inclusion is to be in the top 1% of downloads for six months. There’s a relevant question in the FAQ on the page: “Can a project opt-out or become non-critical in any way?” The official answer states, “No, once the project has been designated as critical it retains that designation indefinitely.” At least one developer discovered a workaround, with interesting results.
nice, I just deleted the atomicwrites package, then uploaded a new version. now it's no longer a critical project
— Markus Unterwaditzer (@untitaker) July 8, 2022
Read that again. One of the most popular packages on PyPI was deleted and a new version uploaded. Every project and every dependency that pointed to a pinned version of atomicwrites got their installation process broken. Sure, it’s no longer marked as a critical package, and the author doesn’t have to use 2FA, but until the PyPI admins stepped in and rectified the problem, a bunch of developers around the world were in quite the pickle. The primary dev, [unitaker] has since decided to leave the package alone, but deprecate it.
Let us know what you think, is PyPI in the wrong, requiring volunteer devs to use 2FA, or is the supply chain issue important enough to make it worth the hassle?
Cuteboi
And if you want to know why PyPi is worried about security — there’s been some odd activity spotted on NPM, to the tune of over a thousand malicious packages mass-uploaded to the service. It’s all under randomly named users, and seems to be a variation of eazyminer, which is a simple coin mining package for Node.js. The name comes from the configuration of that miner, which sends to a pool named “cute”.
This seems to have been a test of an automated tool to create users and upload packages, maybe gearing up for a more malicious run. Or it could be a fellow researcher, who plans to publish a report about how to automate user creation. Either way, there’s even a site available to track the bot’s activity, appropriately named cuteboi.info. (I know, that looks like a terrifying URL. I promise, it’s safe for work.)
Bits and Bytes
We’ve been following the OpenSSL AVX512 memory corruption story for several weeks now. There’s finally a pin in this issue, as OpenSSL 3.0.5 has been released, containing the fix. As far as we know, a full RCE exploit does not exist for this bug. So while it could be a very serious problem when viewed in isolation, this one will almost certainly a real-world impact that rounds to zero.
QNAP has published an advisory about a new ransomware attack against their NAS devices, and the recommended action boils down to, “Do not expose SMB to the internet.” The attack seems to be a dictionary attack against exposed services. An update is available to harden the service, and QNAP recommends using a VPN instead of exposing services to the net.
Microsoft is finally trying to leave the past behind, and block macros by default in office documents. Office has a logic flow to determine whether to run macros, starting with whether the document is explicitly trusted, whether the macros are signed by a trusted publisher, etc. What’s new is the default behavior when the flow falls off the end of the chart. In the past, it would fail open, running the macros by default. The new behavior fails safe, refusing to run. The craziest thing about this is that they’ve received enough pushback that this change has been rolled back for now, so you can still push macro viruses through. Party like it’s 1999, Melissa!
And if that isn’t enough security craziness, check out the story of Aerojet Recketdyne which hired [Brian Markus] as “senior director of cybersecurity, compliance, and controls.” Rocketdyne was falsifying reports, like reporting that they did have a certain piece of security equipment, when it was still in the box. All of this in pursuit of government contracts. [Markus] refused to sign off on a document stating the company had complied with the appropriate rules. He refused to sign, and was fired as a result. After multiple year of legal wrangling, the parties have settled, with [Markus] receiving over $2.6 million from the company, and the US gov’t receiving twice that much. Some times, apparently, it pays to do the right thing. (Via Heise.de).
Rocketdyne was falsifying reports, like reporting that they did have a certain piece of security equipment, when it was still in the box.
Well… technically they did have it. They just never deployed it. The things you learn from Lawyers. (#1 to loathe them)
“is PyPI in the wrong, requiring volunteer devs to use 2FA, or is the supply chain issue important enough to make it worth the hassle?”
[Markus Unterwaditzer] is correct that they’re imposing an additional cost on something they get for free. It appears the PyPI team failed to use any of the well-known best practices for dealing with creators you aren’t paying, like communicating ideas and getting buy-in _before_ declaring them mandatory.
It also looks like the PyPI team is dumping the workload for their problem (security and consistency of the overall collection) onto unpaid contributors. At minimum, they should have a well-reasoned, persuasive explanation for why they aren’t willing to do the work themselves.
I’m not that familiar with PyPI, but they seem to be a non-profit that is providing a free service to the python community general, to both users and developers (including providing free distribution of Markus package). It also seems that PyPI are acting in the community’s best interest by trying to improve security (and picking up the cost of required hardware).
It could be seen as immature on Markus’s part to just delete the package causing issues for package users (and refuse to use 2FA), certain it doesn’t make him look like he has good intentions towards the community…
Marcus could use any of the literal hundreds of coding sites. While PyPi may be a non-profit, that doesn’t mean they don’t make money and pay their employees. The Susan G. Komen foundation is also a non-profit, yet less than 2% of their intake goes towards cancer research.
Non-profit is a misnomer and misunderstood, they make money and are making money off Marcus’ free work.
PyPI’s primary income seems to come from large donations from other foundations (like Mozilla). It seems like it would take some mental gymnastics to claim PyPI is profiting from Marcus – I’m sure they would still get the same donations if Marcus hosted his work on a different site… But they would probably get less donations if they don’t take security seriously…
Susan G Komen: from one data point you are saying all non-profits are embezzling/misappropriating funding?
While I couldn’t care less about PyPI before this thread, I think they are doing the right thing for the community, and bonus that their actions will shake out whiny/grumpy/over-entitled/malaise open source authors who think the world owes them and will smite their users if they don’t get their own way.
I also don’t know Marcus, nor know his circumstances, which editor he likes, or which licenses he prefers, but his actions don’t paint him in the best light. I would avoid depending on software he maintains given the chose.
Actually, it does seem that Marcus realized deleting the package wasn’t the best move, and did enable 2FA in the end.
PyPI should probably make the ‘critical’ designation as opt-in for the developer/maintainer, and notify down stream users to as to their SOC2 compliance (dependency graph). Perhaps even sponsoring Marcus to make atomicwrites compliant (although I’m not sure how much does using 2FA actual costs) if there is enough down stream demand.
It’s a wonderful showcase for the difference between people who really believe in open-source and freedom of choice and people who get paid to believe in open-source and freedom of choice.
In general it’s a good idea to improve security of important base libraries and I’m pretty sure PyPI tried their best to help the open-source community by providing 2FA to maintainers for free.
But in this specific case the way of communication seems to me like a total disaster. They haven’t understood how important freedom of choice is for open-source community.
If the Python Package Index adds a requirement that package maintainers do any specific thing, it’s unsurprising that some package maintainers would simply remove their package from the index rather than complying. Sharing software you wrote is a volunteer effort, and volunteers have the right to decide that the burden has become excessive. Anything that increases that burden increases the likelihood that some of the volunteers would make this decision.
As Markus said in the issue: “I decided to deprecate this package. While I do regret to have deleted the package and did end up enabling 2FA, I think PyPI’s sudden change in rules and bizarre behavior wrt package deletion doesn’t make it worth my time to maintain Python software of this popularity for free. I’d rather just write code for fun and only worry about supply chain security when I’m actually paid to do so.”
When you choose to use free software, you accept any costs that the people who actually wrote it are unwilling to bear. I’ve been on both sides of this, and it’s important to remember that the guy who shared something with you *for free* doesn’t actually owe you anything at all. Forcing a volunteer to navigate arbitrary hoops (even those that are actually good ideas) will inevitably simply make some of them quit volunteering anything.
Such concerns aren’t just limited to free software. If you pay for software, your rights are limited to those enumerated by the license agreement. In many cases there’s nothing in there about future versions, bugfixes, continued distribution, etc. Markus’s actions would not have actually resulted in actionable damages in such a case, even if there was an explicit commercial agreement.
Ultimately, open source is simply a matter of voluntary sharing: “Hey, I did this cool thing. If you find it useful, you can use it too, but there’s no warranty expressed or implied, and I have no liability to you for anything related to this stuff I’ve shared with you.” If you don’t want people to take their marbles and go home, it’s important to avoid anything that would annoy them. Of course, you always have the option to not use their stuff and go find something else or write your own instead.
+1.
Anyway it’s a larger problem with PyPI I think, if there are critical libraries on this repository, critical in the sense that it’s not possible to do basic Python programming without them, then these libraries should be moved to the python standard library and maintained by the core team, thus liberating the volunteer from maintaining his work forever, and lowering the risk of security breaches that arise due to the fact that volunteers can not always maintain proper security.
> We’ve been following the OpenSSL AVX512 memory corruption story for sever weeks now.
Sever.
I had to read this 3 times to see the typo. Thanks!