At the 33rd annual Chaos Communications Congress, [Antonio Barresi] and [Erik Bosman] presented not one, not two, but three (3!!) great hacks that were all based on exploiting memory de-duplication in virtual machines. If you’re interested in security, you should definitely watch the talk, embedded below. And grab the slides too. (PDF)
Memory de-duplication is the forbidden fruit for large VM setups — obviously dangerous but so tempting. Imagine that you’re hosting VMs and you notice that many of the machines have the same things in memory at the same time. Maybe we’re all watching the same cat videos. They can save on global memory across the machines by simply storing one copy of the cat video and pointing to the shared memory block from each of the machines that uses it. Notionally separate machines are sharing memory. What could go wrong?
Basically, it takes slightly longer to access that memory when it’s de-duplicated, because the VM has to go lookup the address of memory that’s outside of itself. If an attacker were interested in finding out if another VM was watching cat videos, he could put the cat video in his memory, wait for the VM manager to de-duplicate it, and then time how long it takes to make a modification in his memory. If it’s longer than some threshold, someone else is watching cat videos. The rest of the talk explains three exploits of this vulnerability.
The CAIN attack allows the attacker to figure out what address a given memory page has in a neighboring VM. Think, for instance, of a Windows DLL. The basic idea is as explained above, but figuring out the offset of the code within a memory page is hard, but they brute-force it by writing the same code fragment at all offsets, and figuring out which one matches. This attack so far just leaks the memory location of programs running on another virtual machine, but think of it as a stepping-stone.
The second attack adds a hardware bug, the Rowhammer attack, to exploit a process running on the same machine. In particular, they’re running Javascript code to exploit Microsoft’s “secure” Edge browser. They get the address of a code and heap pointer using their memory de-duplication attack, create a code object, and use Rowhammer to turn the address of the object into a pointer and run it. All thanks to memory de-duplication.
Finally, “Flip Feng Shui” writes new data into the victim VM by corrupting a local copy of shared memory that then gets mirrored back to the victim. Because a bit flipped in Rowhammer is unpredictable but repeatable, the first stage is to figure out where bits are going to flip, and then align a copy of the data you want changed on the victim’s VM in memory. Then the memory is Rowhammered, and because it hasn’t been written to explicitly, after a while it can percolate back to the victim.
The demonstration includes flipping a few bits in a victim’s SSH public key to turn it into a key that’s easily factorable, and then logging in. In a second attack, they combine the key bit flip with possession of the web domain “ubunvu.com” to install arbitrary software on the victim’s VM using its automatic upgrade mechanism. Holy cow.
This talk is a bit heavy at times, but the takeaways are stunning. It’s “obvious” that memory de-duplication should be a problem, but actually exploiting it (three ways!) is a tour de force. They don’t say that they were responsible, but it’s worth noting that Windows 10 doesn’t use memory de-duplication anymore.
Page sharing between VMs has been disabled on vSphere for a few years now. They added the option to use a salt which by default is unique for each VM so nothing is shared. You can set the same salt on a group of VMs if you want some to share or just disable the salt entirely so everything shares.
The addition of Rowhammer is interesting.
I read this stuff and so want to understand but alas I am just a mere mortal .
The best I can manage is memory duplication in VM’s is bad.
I feel the same, but then again some non tech people think I’m a “genius” because I can make a simple Arduino project. the thing is things like this sound insanely hard but if you have a little know how or learn one or two things the rest falls into place. You read Hackaday I bet most people in your day to day life think you are some sort of super genius.
Don’t feel bad, I got lost after they ditched line numbers.
The things is memory is so cheap deduplication shouldn’t be needed.
Memory stops being cheap when you want to run hundreds of VMs on a single host machine.
While I understand the idea of density, and the fantastic efficiency, I have noticed that far before I fit hundreds of machines on a single host, I am memory-starved.
Not to mention the fact that I fanatically avoid “single” anything in the pursuit of redundancy and load-sharing.
I am also curious how these deduplication vulnerabilities work when the machines migrate.
I imagine that it would be fairly important to target the specific machines, so if a machine were to require more resources than the host had available, and they or even the attacking machine get migrated to a different host, they no longer share the same memory pool, and this exploit seems like it would no longer be effective.
It never hurts to have more of it :-)
thought that project zero found that they couldnt flip bits on ECC and that newer ram uses TRR and MAC to mitigate rowhammer
Is memory deduplication the problem or how it was implemented? I suspect the latter. Rowhammer is fixed in newer RAM, and even with that, you can deduplicate read-only memory segments no problem. Deduplication detection can be removed with asynchronous deduplication – I’m not saying that’s easy, but if it means more VMs and no security flaw, someone will build it.
What are the chances that deduplication is disabled on Windows and others by request of hardware partners? Security is the great excuse. Even though they could fix the vulnerabilities, they don’t because of the cost, hardware partner pressure and kudos for disabling it and looking responsible.