Speculative Execution Was A Troublemaker For Xbox 360

Part of why people can’t stop talking about Meltdown/Spectre is the fact that all the individual pieces have been sitting in plain sight for a long time. When everyone saw how it all came together last week, many people (and not even necessarily security focused people) smacked themselves on the forehead: “Why didn’t I see that earlier?” Speculative execution has caused headaches going way back. [Bruce Dawson] tells one such story he experienced back in 2005. (Warning: ads on page may autoplay video.)

It’s centered around Xbox 360’s custom PowerPC processor. Among the customization on this chip was the addition of an instruction designed to improve memory performance. This instruction was a hack that violated some memory consistency guarantees held by the basic design, so they knew up front it had to be used very carefully. Even worse: debugging problems in this area were a pain. When memory consistency goes wrong, the code visible in the debugger might not be the actual code that crashed.

Since we’re talking about the dark side of speculative execution, you can already guess how the story ends: no matter how carefully it was used, the special instruction continued to cause problems when speculatively executed outside the constrained conditions. Extensive testing proved that instructions that were not being executed were causing crashes. That feels more like superstition than engineering. As far as he can recall, it ended up being more trouble than it was worth and was never used in any shipped Xbox 360 titles.

[Main image source: AnandTech article on Xbox 360 hardware]

28 thoughts on “Speculative Execution Was A Troublemaker For Xbox 360

  1. I planned to buy a new Intel CPU this month but guess I’ll pass until fixed Ice Lake comes. I like my good old E8400 overclocked to 4500 MHz, I call it “pocket goatse”.

    1. I honestly dont think we will see the hardware fix’s implemented for another few CPU generations, that intel has coming out this year will likely still have physical bug present. Intel sits on so much hardware wise compared to what they have set for release.

      1. It’s also possible a full bug fix may end up requiring a new socket series as it may not be usable on current motherboard design. This may take a few years anyway since they’d need to work on a new socket design so the new CPU can’t be shoehorned into old sockets, new design tested for bugs and such, design finalized and spec released, and motherboard makers would need to design new line of motherboards around the new sockets.

        I am quite happy with my 5 year old i7 2700k. One of the few that can safely run at 5GHz without requiring elaborate cooling setup.

        1. I kinda of doubt it will require a new socket. So much is integrated into the cpu these days, that most of what is coming off it the chip as far as pins, is power, pcie, and i2c. Which is why we only see a socket change on the on intels Tick cycle this last decade or so. Even when your talking about the Tick cycle of cpu revisions, they are only changing by 1 or two pins. and the location of the indexing notches.

          I haven’t torn into the design docs for AMD cpu’s as much as i have the intel ones. Cause i was chopping and rerouting lanes for a while on motherboards for overclocking w/video benchmarks. Which till the ryzen reared it head wasnt worth its time on amd hardware since the PCIE performance was so poor and generations behind.

          1. Think 771 and 775, basically the same chips, one for Power XEON line other for consumer, you could with a slight mod get one chip fully working in another socket. So there is very little “need” for a new socket, but their will be one.

    2. Might I suggest a Xeon process based on Harpertown architecture (E/X54xx)? Used X5470 are going for $40 a piece.

      delidded.com has your modding instruction and it’s like a breath of fresh air for an old LGA775 rig. Great overclockability on these babies too, I moved one rig from a 4.2GHz E8400 to one of these and it flies.

  2. “Since we’re talking about the dark side of speculative execution, you can already guess how the story ends: no matter how carefully it was used, the special instruction continued to cause problems when speculatively executed outside the constrained conditions.”

    Considering prior conversations speculative execution may not have a light side. Looks like we may have to find some other ways to speed up our processors.

    1. No. The highest performing processors are all using out of order execution. The main advantage (IIRC 80%+) of the performance advantage of OoO processors over in-order ones are better speculative execution.

      We simply can’t replace it while running existing software. But we don’t need to as the problem isn’t in speculative execution itself, it is just used as a tool.

      Meltdown: Intel allows speculative loads to bypass protection checks allowing an attacker to read all mapped memory ignoring protection mechanisms.
      The problem: not doing a protection check.
      Solution: do a protection check before doing an access even when speculating.

      Spectre: branch predictors being shared between two protection domains allows an attacker to target the branch prediction outcome of one or more branches in the victim domain. Speculative execution may expose data the attacker isn’t supposed to know but is accessible by the victim.
      The problem: not taking into account the combination of sharing and speculative execution.
      The solution: many. Making sure the caller can’t influence the callee with non-architectural state is the best, making it unrealistically hard to do is acceptable, patching software to not expose sensitive branches is the quick fix.

      And the X360 problem isn’t in the speculative execution either, it just exposed the design flaw elsewhere.

    2. The “light side” of speculative execution would be the performance improvement. As I understand it, Spectre/Meltdown could be mitigated by either preventing code speculatively “doesn’t” execute from causing cache updates or preventing speculative execution from speculatively crossing normally sacred memory boundaries. Either would likely require extensive rework to existing designs, and both would be a good idea.

      I also wonder whether ASLR (which happens in software) might mitigate any real-world implementation of these vulnerabilities.

      Spectre/Meltdown aren’t so scary because they’ve been found in use They’re so scary because they’re flaws “written in stone” in existing silicon, they’re ubiquitous, and because security weaknesses only become more exploitable over time, as techniques to exploit them become more advanced and synergy between exploits or techniques is discovered.

      1. ASLR wouldn’t help at all for “meltdown” and little for “spectre”. ASLR is just a dirty hack to make some buggy software harder to exploit.

        Meltdown can’t be improved – it can already read the whole memory map. The solution is to reduce what is mapped in any instance so that a user process can only read what it is allowed to read anyway, sadly it isn’t really possible to do in practice in traditional systems.

        Spectre will be less likely to succeed as software is rewritten to avoid potential security leaks.

  3. I miss the days when there was Cyrix, VIA , AMD, Intel and others to choose from and so just like the phone market with little to no choice of what you can buy, this in turn has changed everything and not for the good.

      1. You’d be hard pressed to find someone who wants a noticeably lesser performing processor for the same price. AMD is keeping afloat because it offers a generally more affordable alternative (in the recent years/generations) since it doesn’t outperform Intel where it matters for most consumer.

        Let’s say the fault is 1 part consumer 1 part chip maker for the death of Cyrix, VIA and others.

    1. It’s a read only only bug. So it limits what you can do with it. It may let you extract AES keys for encrypted firmware. Mainly, the intel and Arm chips effect by meltdown. However, it does not get around a signed boot loader or signed OS. We still don’t have the private key. Spectre is not as helpful since it’s really only allows access of memory within a single process. A problem for VM’s that are trying to isolate tasks in a single process. Aka Javascript or VM like VMWare.

      It may make some exploits easier though that require knowing if a program is in a certain state to exploit it. Otherwise, it sadly does not help with a jail break.

    2. Broke my Xbox360 open long ago. Didn’t have to touch this bug to do it. I went through the SATA bus before an extra layer of security was bolted on to mitigate that route.

  4. Xbox 360 and Xbox One have write-back hashing and even encryption on bit isolated page tables for the hypervisor and a lot of loaders(the hashes and keys never come out of MMU SDRAM). You can actually dump external system RAM on both using clock-down and SPI on DRAM chips to build a complete map for partial IDA reverse engineering.. It’ll only find you bugs in low privladge code though and they have to be libc style.. You have to get code exec as HV or bootrom to defeat the chain of trust.

    MS has had better security than Nintendo and Sony since the beginning. Recent Apple security design was in x360 over a decade before..

    Nintendo Switch and PS4 both are just Trustzone loader auth and ASLR.. Both fully jailbroken..

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.