[Andrew Peterson] was looking for a way to indulge in his retro gaming passions in a more contemporary manner. His 3D NES emulator “N3S” for Windows brings Nintendo classics to the HoloLens, turning pixels into voxels, and Super Mario into an augmented reality gingerbread man.
To run NES games on the HoloLens, [Andrew’s] emulator uses the Nestopia libretro core. Since AR glasses cry for an augmentation of the game itself, the N3S re-emulates the NES’ picture processing unit (PPU), allowing it to interpret a Nintendo game’s graphics in a 3D space. [Andrew] also put together a comprehensive explanation of how the original Nintendo PPU works, and how he re-implemented it for the HoloLens.
The current version of the N3S PPU emulator automatically generates voxels by simply extruding the original pattern data from the game’s ROM, but [Andrew] is thinking about more features. Users could sculpt their own 3D versions of the original graphic elements in an inbuilt editor, and model sets could then be made available in an online database. From there, players would just download 3D mods for their favorite games and play them on the HoloLens.
According to [Andrew], the emulator reaches the limits of what the current pre-production version of the HoloLens can render fluently, so the future of this project may depend on future hardware generations. Nevertheless, the HoloLens screen capture [Andrew] recorded makes us crave for more augmented retro gaming. Enjoy the video!
Holy… now that I hope is the future, sure beats mounting projectors on industrial robots!
I wish [Andrew Peterson] had filmed through the Hololens instead of using the in-system capture tool, which makes it look better than the actual display.
The author should seriously talk to the person who wrote an emulator that automatically guesses 3D geometry from NES tiles — https://geod.itch.io/3dnes
That’s a really cool project, but after playing with it for a few minutes, I realize it guesses poorly The good results you see are mostly edtted files where someone spent a bit of time modifying how the layers should be displayed, which gets stored in 3DN files that you can then share so others can see it rendered the same way (separate from the rom file)
Amazing work! This is a really cool idea and very well implemented. Was waiting to see mario ‘flat’ against the hedge!