We’ve all heard about cosmic rays flipping bits here and there, but by and large, it occurs rarely enough that we don’t worry too much about it on a day-to-day basis. However, it seems just such a ray happened to flip a crucial bit that assisted a speedrunner in the middle of a competition.
The flip happened to [DOTA_Teabag], who suddenly found Mario flying upward to a higher part of a level, completely unexpectedly. Testing by [pannenkoek12] seems to indicate that this may have been due to a single-bit change to Mario’s height value, from C5837800 to C4837800, leading to the plucky Italian plumber warping upwards through the level. The leading theory is that this bit flip was caused by a cosmic ray event, though the likelihood of such an event is exceedingly rare.
It’s possible that there remains another cause for the flip, though after much work from the community replicating the situation in emulation, none has been found. Other suggestions involve electrical noise or other malfunctions causing the flip, though one would rarely expect such an occurrence to change just one bit of RAM. For now, the jury remains out, but who knows – maybe in the future we’ll find out it was a hidden, undiscovered exploit all along. Of course, if Nintendo doesn’t get you going, try speedrunning Windows 95. Video after the break.
>though the likelihood of such an event is exceedingly rare.
“Studies by IBM in the 1990s suggest that computers typically experience about one cosmic-ray-induced error per 256 megabytes of RAM per month.”
N64 has 4MB of RAM, so once every 5.333 (repeating of course) years.
Since Super Mario 64 runs on an N64 and doesn’t make use of the 4MB RAM expansion pack, then statistically that would reduce to one cosmic-ray-induced error per N64 per 64 months, or about once every 5 and 1/3 years.
which is not 0
Good thing no one said it was.
Good thing no one ever said they did.
With 32 million units sold, there is a potential 16,541 units being subject to a cosmic ray event every day. Whether one of those happens to belong to a youtube speedrunner is a matter of luck, but then again, someone usually wins the lottery as well.
I have for a fair bit of time been curious to how cosmic ray induced bit errors are these days on more modern hardware.
I know that SRAM is a lot more radiation hardened than DRAM, but SRAM isn’t particularly cheap.
DRAM + ECC should on the other hand fix most of these errors.
But it is still an interesting topic.
The error doesn’t have to occur in RAM. Some cosmic rays actually have the energy of a flying base ball and can trip a bit while it’s being transmitted between chips, or mess up with some logic, etc.
It’s more like, one event per month per equivalent chip area to 256 MB of RAM (in 1990’s densities).
“Some cosmic rays actually have the energy of a flying base ball”
Considering that the LHC reaches a particle energy of 6.5 TeV (the world record for accelerators), or in Joules, about 1µJ for a given particle.
A baseball moving at a casual 1m/s has about 145mJ of energy. And at this speed, it won’t really be flying through the air, rather just casually rolling off a table. Subatomic particle reaching these energies is exceptionally rare to say the very least.
There is some subatomic particles that have been observed at a bit over 1J.
But the phrase “the energy of a flying base ball” seems like it implies it being thrown.
For a roughly 145g ball moving at a typical throwing speed. (apparently about 90-120 km/h if one asks google.) Lets say something more modest, like 50 km/h. Then our ball would have an energy of 28 J, or over an order of magnitude more than the highest ever observed among cosmic rays.
In regards to flipping bits on signal conductors.
That tends to be a bit hard. Considering how most signal conductors are driven by output drivers. Giving a fairly low impedance connection to either of our power rails.
Though, not saying that “cosmic rays can’t flip these bits!”, more along the lines of, “I would consider signal traces substantially less sensitive than for an example DRAM.”
Considering how DRAM isn’t actively driven to a supply rail. Its cells are simply left floating, if a subatomic particle rips through the nearby insulation, it can cause a conductive path for that charge to escape via. (A fair few particle detectors work in this fashion as well.)
Then we have Flash and (E)EPROM have floating gates within them as well, and would be sensitive to radiation for the same reason.
But here is the main reason for my question how modern hardware fairs.
For an example, a big electrolytic, even a film capacitor won’t suddenly fully discharge just because we expose it to some radiation. The insulation will let some energy through with each cosmic ray interacting with it, but the capacitor has a fair bit of charge making this loss of energy mostly negligible.
Now, older DRAM chips weren’t have a lot of charge stored in them compared to a much larger film capacitor. But comparing DRAM from the 80’s and 90’s to modern ones is also a fairly huge difference in the amount of stored charge.
So for modern DRAM and Flash, the amount of stored charge if fairly tiny. The amount of insulation as well. Not to mention how a lot of DRAM/Flash chips have 3D stacked cells and at times also a stack of chips as well. Then this could theoretically mean that a given particle could erase more than just one bit for a given event. For a planar array, we can have the same thing happen if the particle travels in the mostly same plane, but for a 3D stacked structure, there is more opportunity for a given event to interact with more cells along its course. (Especially for 3D stacked cells where the Z height between cell layers is measured in sub µm.)
But even SRAM can suffer bit errors due to radiation. Since if the radiation discharges the active side of our latch then it could toggle over into the other state. but considering how both sides are actively driven, then this should be fairly robust against radiation, even if the given structure is fairly tiny.
In short, a modern chip could potentially suffer multiple bit errors from the same high energy particle, leading to more severe problems than what an older chip would have suffered.
For DRAM we can fairly trivially fix this by error correction. But Flash tends to be a bit less nice to just go through and repair, due to the block structure of it, wear leveling and other issues making rewriting data a bit of a can of worms.
I’m an amateur, but according to the Wikipedia page on cosmic rays, the flux varies inversely with the energy. So the ~50J particle that people talk about as having energy roughly equivalent to a fastball pitch in baseball is rare.
But I think the phenomenon is more related to disturbance of transistors by the presence of positive charges passing through (cosmic rays are mostly protons or He nuclei) which can disturb the flow of currents and charges (holes and electrons). So I doubt there would be any impact on circuit voltages outside of the sensitive and amplifying transistor.
This of course was highly dependant on the size of the RAM cells at the time. With smaller cells I wonder how today’s error rate has changed.
Yes, individual bits are smaller, but there is more of them, so average area of memory could be about the same. In other words, chance of flipping exact single bit is smaller, but chance of flipping some bit in a computer is about the same as then. But this is just my guess, it would be interesting to see similar analysis on modern hardware, this is three decades old…
Considering the 3D stacked nature of modern DRAM, then I would expect there to be a risk of more bit errors for a given event.
Since on a given memory chip, a manufacturer can build 8-16 layers of memory cells on top of each other. Then at times stack a bunch of these into the same package.
I remember parity-checked RAM on the original IBM 5150 PC (1980s). If the parity didn’t check, the screen cleared, it printed “PARITY CHECK ERROR”, and hung. So if just one bit was wrong, you lost everything. Infuriating. Even more infuriating was the thought that on average every 1 in 9 times it would have been the parity bit RAM itself that was wrong, so you lost all your work for nothing.
No such thing as that happening in your favour. Just using Occam’s Razor tells me it just might have been an external human source or a glitch in the code itself.
Of course making it sound like space magic to not have yourself dethroned is favourable than just facing reality.
One of the most misunderstood things about statistics and randomness – extraordinarily rare things happen all the time. If you only look after the fact for *any* unusual things happening you’ll find plenty of seemingly rare things have in fact happened by chance.
But sure, bit flips aren’t actually all *that* rare but other causes seem more likely.
Yup! In fact once in a hundred years type events do happen two consecutive years in a row from time to time.
And there was a study recently that says “century floods” actually happen every 20-30 years on average – they just sometimes leave big gaps. Just because it’s random doesn’t mean it has to be evenly distributed over time either.
That study is actually what led me to post my comment :)
Missed the point where he said: “in your favor”
That part is not really true either. Random happenings don’t care about you, but they can still be in your favor. People win the lottery.
“No such thing as that happening in your favour.”
Sorry?
“Dethroned” in a casual speedrunning race? Yeah, ok. Also, this completely screwed him up and didn’t save any time.
We need Billy Mitchell to chime in. https://en.wikipedia.org/wiki/Billy_Mitchell_(video_game_player)
Except he’s a fraud, has never legitimately held a single record, is a rampant cheater, has no game knowledge of SM64, doesn’t speedrun and is a disgrace to the community. Also, this occurrence is very old, this happened like 5 years ago; a better person would be Cheese, or Puncayshun, or an actual respected name in the community. But all those people have already made their input, because; once again, this news is many years old and there was already a significant money bounty placed on finding the solution.
Billy Mitchell is a disgrace to gaming as a whole.
That ray was not traced here, I saw nothing happening. Magic mushrooms are much more powerful to see cosmic rays…
See also https://en.wikipedia.org/wiki/Bitsquatting. Bit flips do indeed happen.
And as for the possibility of cheating, there was this case recently of a speedrunner that apparently modified the item drop probabilities in Minecraft: https://www.youtube.com/watch?v=8Ko3TdPy0TU
Bitsquatting could fairly trivially be solved by just doing a bit better error checking and correction.
Just sending the data with a checksum would root out most of these issues, but then such would need to be part of the DNS standard though, so not the most trivial thing to just implement.
I guess it is another one of these remnants of the good old times when one could blindly trust other systems.
DNS goes over UDP, and UDP does have a checksum. So it seems more likely that bitsquatting errors occur at the client side, before the packet is sent. On server side ECC ram is probably used.
You’re seriously bringing that stupid debunked Dream Speedrunning Drama up? It’s been months. The run is now in the double digits on the leaderboards. It doesn’t matter if it happened or not, Let it die already.
Debunked? You mean all but confirmed?
lmao, yeah, that dude cheated.
Definitely a flaw in the game. It’s likely a convergence of a game event with an animation and needs exact timing to occur. Digging through the source code should help identify what could possibly result in Mario being dropped again. Then you just have to figure out what could have possibly triggered one of those events. Could be as simple as a flaw in collision detection.
Yeah we all know that video games have extensive hardware error protection, they only use the highest quality components, and glitches Never, Ever happen! :-}
Evolution works similarly with most bit flips caused by cosmic Ray’s resulting in cancer. But rarely will increases fitness of the organism.
Also there are arguments that evolution could not be correct because most mutations cause cancer. I see similar fallacious arguments in this thread.
Likely a collision detection bug. Reminds me of a bug in the old driving game Stunts where if you crashed your car really hard, it would occasionally ascend to heaven in a backward spiral. The original Left 4 Dead had some similar bugs that players would intentionally exploit.
That’s pretty much how I played stunts all the time! Get the car flying through the tracks! I love stunts!
There has been a lot of work done to explain this happening in code alone. There are people who understand the movement mechanics probably on the same level as the developer and no exploitation has been found.
Except this is one of the most heavily deconstructed games ever. The source code has already been completely decompiled and there was a several thousand dollar bounty on finding the solution that has stood for many years.
If only this would happen to my bank account.
Yeah because when money goes missing at the bank, they never bother to track it down. And even if they do, what’s a few years in prison? Nothing much! Go for it!
OK, then a utility company account. They’re hideously disorganized. Used to work for an “English” one on industrial billing. Pretty sure all bills had errors and some of the team made very little effort to bring even vague realism to the bills.
It was temp work, was only there for a week and hope my efforts leaned in the direction of accuracy.
Prison for something that happened on *their* hardware with none of your doing? Don’t be ridiculous.
The actual reason why this would be unlikely to happen is that there are several consistency checks.
Do you mean losing 0x1000000 in an instant?
I can see speedrunners hacking up smoke alarms to put the radioactive sources next to their N64 CPUs…
A lot of bit flips can lead to the console/game crashing. So being lucky to first get one, and have it in a good place is very rare.
All hail RNGesus!
Unsure whether you think that might induce big flips or if you think gamers might think that, but nothing will happen. Beta particles emitted by Americium are too low energy and can’t penetrate but a few microns into solid matter.
Cosmic rays can have a lot more energy, so it’s possible for them to travel through the roof of your house, into the plastic case of your N64, through the RAM package, and smash into all the sensitive little bits.
I do not believe for one second this was a random bit flip. The vast majority of random bit flips will simply crash the game, locking it up or sending the whole thing to obvious la-la land, or even have no effect. For a random bit flip like this to benefit the player to be plausible, N64’s would have to be crashing and going wacko over random bit flips all over the place. Since I have never seen anything about this being a recurring problem it is much more plausible that something *cough* arranged the bit to flip deliberately, with the possibility of it being random as a fig leaf. I will just note that for a 100 MHz processor it would be short work for a modern FPGA to detect the desired address-data combo and trigger the bit crowbar operation, perhaps with other conditions (such as of course do it only once). Not that I’m saying the lucky gamer did that, of course. But I know which possibility I’m putting my money on when the bookie offers odds.
I don’t know anyone who won the lottery, but people are know to win from time to time and there are tales of people winning more than once…
“Game crashes” isn’t newsworthy and generally offers no information about exactly how it crashed, gamer curses, power cycles and it doesn’t happen in exactly the same way again. You’re supposed to get a random bit flip every couple of years of computing, but most do nothing remarkable. Not many people are gonna jump to cosmic rays to report crashes either, they probably think it was a power glitch.
The number of N64s produced and the number of years since then they’ve been in use, make it practically certain that each bit of RAM has been flipped somewhere at some time. You should be more suspicious if it had never been detected.
“It was unlikely therefore I don’t believe it” is a terrible argument. That bit was just as likely to flip as any other
Ha, cosmic rays. Yes very sensational. But nah, I’ve seen this happening myself many times back in the day. Can’t tell what the exact cause or trigger is. But it is a lot more likely that bit got flipped cause of an overflow elsewhere.
I watched a game streamer (tyty23) win a race at ExileCon using a wall glitch that literally was done in front of millions of people, and no-one caught it. I only caught it because I had accidentally done it myself, and put in the time to replicate it.
I can pull it off 100% of the time now. If this guy found a similar weird combo/animation cancel/triggered skill that jumped him to the top just one time, it would only be a matter of hours to perfect the replication, and make it look “accidental”.
And then 5 years go by with a 1000 buck bounty on someone figuring how to replicate it but the runner was so good on making the TTC upwarp look “accidental” that nobody ever figured how to make it happen. Right?
I should’ve not used Google news cause damn these comments are way too skeptical. The reason a bitflip is the leading hypothesis is because after multiple years there was no progress in finding the glitch responsible for what happened. One of the most knowledgeable people of the SM64 game, pannenkoek2012 aka UncommentatedPannen, spent a lot of time on this and even put a bounty of $1000 of reproducing this bug: https://youtu.be/aNzTUdOHm9A. Instead of just deciding that a cosmic ray was the reason, two possible explanations had been hacked to occur to compare the output. When these results were compared to the actual output, see the first video linked in this article, a single bitflip seems to match up awfully well. Different reasons for this bitflip have been proposed, but many of these would corrupt more than just 1 bit. Although a cosmic ray having a positive effect is obviously rare this is the best explanation so far. I think the article referenced at the start of this article explains it really well.
And if you are 100% certain it’s a glitch, why don’t you try to get that $1000 bounty, it’s money for at least a weeks with of pay.
Yep, combinatorial explosions are everywhere. If the probability isn’t zero, then stuff happens. The binomial distribution is an interesting beast with regards to rare events.
There is a problem with the referenced article.
“In extreme cases, these upsets have caused planes to fall out of the sky and elections to give thousands of
impossible additional votes to candidates….”.
Could the author provide explicit evidence of these two claims.
..did you try clicking on said claims, which link to articles detailing such evidence?
Yes.
“It was most likely a single event upset (SEU)” = No explicit evidence.
“However, the Australian Transport Safety Board (ATSB) found it had “insufficient evidence to estimate the likelihood” that an SEU was the cause” = No explicit evidence.
This glitch didn’t actually save him any time since it was so random and interrupted him trying to do something else. Also, the article makes no mention of the fact that this happened back in 2013, or the fact that people have been trying to replicate this since 2015 with zero results. There’s a $1000 bounty if you think you can figure this out.
Whether or not it saved him time on this occasion is beside the point. The speed runners want to be able to do it every time, they don’t want it to be a cosmic ray interaction, they want it to be a bug that can be exploited by a human using a controller.
simply SW bug
Cool, you figured out what other people have been trying to figure out for 8 years. You should collect the $1000 bounty that no one else has been able to.
Why did they make this article now when the bit flip happened 4 years ago??