Netflix isn’t the first name to come to mind when considering security research firms, but they make heavy use of FreeBSD in their content delivery system and do security research as a result. Their first security bulletin of the year, not surprisingly, covers a FreeBSD vulnerability that happens to also affect Linux kernels from the last 10 years. This vulnerability uses SACKs and odd MSS values to crash a server kernel.
To understand Selective ACKs, we need to step back and look at how TCP connections work. TCP connections provide guaranteed delivery, implemented in the from of ACKnowledgement (ACK) packets. We think of a TCP connection as having a dedicated ACK packet for every data packet. In reality, the Operating System makes great effort to avoid sending “naked” ACK packets, and combines multiple ACKs in a single packet. An ACK is simply a flag in a packet header combined with a running total of bytes received, and can be included in a normal data packet. As much as is possible, the ACK for data received is sent along with data packets flowing in the opposite direction.
One problem with this approach is that when a transmission failure occurs, it’s not clear which packet was dropped, and multiple packets must be re-transmitted. Another strategy for handling ACKs is to use Selective ACKs, or SACKs. A SACK will include the ACK flag, the total number of bytes, as well as the TCP sequence numbers. When data is dropped, the SACK packet specifies precisely which packets were lost.
The other term important to understand is the Maximum Segment Size (MSS). This value is usually specified during the initial TCP handshake, and specifies how much data can be transmitted in a single TCP segment. A MSS set to a lower number often results in data being split into multiple segments.
Netflix outlined several problems related to SACK , but the most serious vulnerability is triggered when an attacker makes a TCP connection to a Linux or FreeBSD server, and sets the MSS to the lowest possible value. After data is transferred, the attacker sends a sequence of SACK packets, requesting the re-transfer of specific multiple packets. This specially crafted series of packets causes the multiple fragmented messages to overflow the server’s outgoing buffer. It appears this attack cannot lead to code execution, but it does cause an immediate kernel panic, which essentially knocks the target machine offline.
Patches fixing the problem have been released,
but aren’t yet available for easy install on live systems. The patches haven’t yet been part of an official kernel release, but most distributions have already backported the patches and made them available as updates. For more information, see a very helpful comment from an anonymous commenter below.
As a workaround, Netflix suggests either disabling SACK altogether, or filtering packets with very low MSS values. More information about these mitigations is available in their bulletin.
Building on the concepts of Rowhammer, Rambleed attacks the memory of other processes, but by reading that memory instead of just writing to it. Just as with Rowhammer, the central idea is that modern RAM is so dense that individual bits have a detectable effect on nearby bits. Rowhammer allowed an attacker to flip nearby bits even though they may have belonged to a different process, or even the kernel itself.
Rambleed depends on the physical layout of memory — it’s essentially a two dimensional grid. The bits above and below have an effect on the bit flips of a given bit. If an attacker can control a row of memory, a Rowhammer attack can be mounted on one of the bits of that row. By measuring how effective that attack was, the status of the bits above and below can be statistically determined.
Historically, physical RAM attacks of this nature is defeated by ECC memory. The Rambleed researchers suggest two approaches to overcome ECC. The first is to flip multiple bits so that the ECC algorithm still evaluates the pattern as correct. The second technique is a timing attack, where an error-corrected read takes measurably longer than an uncorrected read. Since the presence or absence of a flipped bit is enough to determine the target bit’s value, the ECC mechanism is defeated. As their coup de grâce, the authors demonstrated Rambleed by recovering an RSA-2048 key from an OpenSSH 7.9 server.
Have I Been Pwned… For Sale?
First off, if you haven’t already, go check out Have I Been Pwned. Give the website an email address, and it will return the list of websites that have been compromised where an account was using that email address. It’s extremely useful to keep track of where your accounts have been scraped and exposed. While some hits are benign, like your email address scraped from public Github data, you might just discover an old forum or service that leaked an important password or other data.
As useful as this service is, it’s surprising to see a virtual for sale sign show up. [Troy Hunt] has been running the site single-handedly for over 5 years. He now measures traffic by the millions, and records by the billions, and recently had the epiphany that personal burnout was looming on the horizon, unless changes were made. He’s looking for a parent organization or company to acquire HIBP, stay true to his core principles, and let him make some changes to keep the ship afloat.
Oracle Weblogic is actively being targeted with a Java deserialization attack. If that sounds familiar, it’s because we talked about it right here not long ago.
An April commentary on the vulnerability seems particularly apt, given the current resurgence of the problem. [Rob VandenBrink] observed that Oracle’s resolution for the problem is simply to blacklist the specific attack vector, rather than take action to fix the underlying deserialization problem.
Firefox has released two point releases in the last week, patching two vulnerabilities that are reported to be actively used in an attack against Coinbase employees. Not all the details have been released yet, so look forward to more details next week. For now, just make sure your version of Firefox is at least 67.0.4.
13 thoughts on “This Week In Security: SACK Of Death, Rambleed, HIBP For Sale, And Oracle Weblogic — Again!”
If you’re like me and stayed on an old version of Firefox for add-on support, just switch to Waterfox. Works great and supports legacy add-ons
I hope Troy finds a non commercial partner for HIBP like the EFF, ACLU or other equivalent associations. Someone who is not going to try and commercialize the database and thus make it worthless in its function. I get that he wants to stay incharge but any commercial entity will not stand for that for very long as someone will try to maximise that revenue stream and generally they will have more expensive lawyers than he is able to afford.
Money in vs money out. Imagine that? Good things cost money.
quote: Patches fixing the problem have been released, but aren’t yet available for easy install on live systems.
This isn’t true. Patches were released by most Linux distributions on the 17th, the same day the vulnerability was announced. These links came from a comment on hacker news (I don’t know how to link directly to the comment).:
gravitas 3 days ago | parent | on: SACK Panic – Multiple TCP-based remote denial-of-s…
I’m collecting vendor links internally for work:
Red Hat / CentOS
https://linux.oracle.com/errata/ELSA-2019-4686.html (RHCK kernel)
https://linux.oracle.com/errata/ELSA-2019-4685.html (UEK5 kernel)
https://linux.oracle.com/errata/ELSA-2019-4684.html (UEK4 kernel)
https://alas.aws.amazon.com/ALAS-2019-1222.html (Linux 1)
https://alas.aws.amazon.com/AL2/ALAS-2019-1222.html (Linux 2)
SUSE / SLES
Doh. You’re absolutely correct. I’ll add a note to point that out.
LinuxMint didn’t have an update last night when I checked…
It won’t show an update until the Mint developers backport the Ubuntu patch to their branch.
That being said, Mint is based on Ubuntu, so downloading the patch for Ubuntu and applying it via dpkg should work just as well.
“This specially crafted series of packets causes the multiple fragmented messages to overflow the server’s outgoing buffer. ”
Buffer overflows. Let us count the ways. :-)
Question on Rambleed/Rowhammer, if I was to overspec my RAM size by double, and have unused rows between the rows I use, would that stop the problem dead? Would be some hack though.
That’s a really interesting thought. I would imagine it would take some kernel hacking to make it work, but in theory I would think that could work.
That’s basically impossible with how modern memory interfaces are wired. The architecture just doesn’t allow it.
I’m not even remotely qualified or experienced enough to answer this question. But I’ve never let that stop me before. Hopefully someone with relevant knowledge will point out all the ways I’m wrong.
It sounds like that would be a very hardware specific question. Does your server utilize RAM blocks randomly (as the name suggests), or does it fill up blocks sequentially. In either case, unless your hardware happens to leave empty blocks between used blocks if you keep the attack up long enough you’ll probably luck into a scenario where you can cause problems for the server. As with other attacks it seems like it would be trivial from the attackers side to cause the server to spin up extra instances until your available RAM was saturated. Then the trick becomes finding a sector not utilized by your attack.
Thanks for the post here and I loved this so much.
Please be kind and respectful to help make the comments section excellent. (Comment Policy)