Apple has developed a proprietary — even mysterious — “fisheye” projection format used for their immersive videos, such as those played back by the Apple Vision Pro. What’s the mystery? The fact that they stream their immersive content in this format but have provided no elaboration, no details, and no method for anyone else to produce or play back this format. It’s a completely undocumented format and Apple’s silence is deafening when it comes to requests for, well, anything to do with it whatsoever.
Probably those details are eventually forthcoming, but [Mike Swanson] isn’t satisfied to wait. He’s done his own digging into the format and while he hasn’t figured it out completely, he has learned quite a bit and written it all up on a blog post. Apple’s immersive videos have a lot in common with VR180 type videos, but under the hood there is more going on. Apple’s stream is DRM-protected, but there’s an unencrypted intro clip with logo that is streamed in the clear, and that’s what [Mike] has been focusing on.
[Mike] has been able to determine that the format definitely differs from existing fisheye formats recorded by immersive cameras. First of all, the content is rotated 45 degrees. This spreads the horizon of the video across the diagonal, maximizing the number of pixels available in that direction (a trick that calls to mind the heads in home video recorders being tilted to increase the area of tape it can “see” beyond the physical width of the tape itself.) Doing this also spreads the center-vertical axis of the content across the other diagonal, with the same effect.
There’s more to it than just a 45-degree rotation, however. The rest most closely resembles radial stretching, a form of disc-to-square mapping. It’s close, but [Mike] can’t quite find a complete match for what exactly Apple is doing. Probably we’ll all learn more soon, but for now Apple isn’t saying much.
Videos like VR180 videos and Apple’s immersive format display stereoscopic video that allow a user to look around naturally in a scene. But to really deliver a deeper sense of presence and depth takes light fields.
The reason is rather simple, less distortion on the edges while still using all available pixels
Hackaday’s highlighting of the deep diving being done by individuals into big questions, including big tech’s schemes, is why I like hackaday.
Kudos to Mike Swanson and Donald Papp for this immensely satisfying dive!
Agreed, just a shame there isn’t more details from Apple themselves – really can’t see the advantage to them in locking the format they want to use away in this case. Their walled garden I can see why even if I don’t like it – more cut for them all the time and a captive audience forever as it too much work to cross platform. But if you want folks to buy into your vision pro etc you need content for it and to be able to tempt users of other platforms with how nice the content you generate is. That really would be easier if it used a ‘standard’ format or at least was well enough understood that a conversion can be done.
I’m wondering if their inability to find a complete match may be because the format is actively being distorted to match the display optics on the vision pro. Would seem silly to me to do it that way, but if you have zero desire to allow any other device to ever playback this stuff it should lower the computational load a bit.
Even if it was documented, it’s under DRM, you can’t generate without Apple private keys, so it would be useless anyway. But as all other proprietary DRM content, this is going to wade in the oblivion with time passing.
The whole point is Apple aught to want it to work, so just because it can be DRM doesn’t mean it must always and only ever work with DRM content. If Apple want to make it possible that should be trivial, assuming it won’t already just work when the DRM is absent.
But the mystery is a big part of the fun of reverse engineering.
I suspect they might not want to release it until they’ve locked down the format internally. Until they release it they can develop and mess with the format; once they release it, they have to keep supporting that version.
Perhaps, but that does mean anything their users are now making will need conversion or backwards compatibility modes in the future versions still. Also this Apple, hyper locked down with massive markups is their default for everything…
I wonder if they’re using a Peirce quincuncial or related projection
Most ‘fisheye’ format files are porn. Ref: https://youtu.be/LTJvdGcb7Fs?si=7TIa2iMdJkdscHjY&t=120
If apple didn’t provide a way to play standard format VR porn they really did miss the point of VR.
Stereoscopic “VR video” is the two-colour-process of VR image storage. If it doesn’t at the very lease contain depth data for reprojection (with lightfield storage being idea to preserve surface BRDF behaviour) in order to not completely break on head motion, then it does not deserve the moniker ‘VR’.
Wait until you accidently find a female perspective VR porn vid.
They are VR enough. Coming right at you. Argh. Fight or flight.
Off topic. Thank you Parker and Stone. For not rereleasing BASEketball in 3d.