Unraveling The Secrets Of Apple’s Mysterious Fisheye Format

April 18, 2024 by Donald Papp 12 Comments

Apple has developed a proprietary — even mysterious — “fisheye” projection format used for their immersive videos, such as those played back by the Apple Vision Pro. What’s the mystery? The fact that they stream their immersive content in this format but have provided no elaboration, no details, and no method for anyone else to produce or play back this format. It’s a completely undocumented format and Apple’s silence is deafening when it comes to requests for, well, anything to do with it whatsoever.

Probably those details are eventually forthcoming, but [Mike Swanson] isn’t satisfied to wait. He’s done his own digging into the format and while he hasn’t figured it out completely, he has learned quite a bit and written it all up on a blog post. Apple’s immersive videos have a lot in common with VR180 type videos, but under the hood there is more going on. Apple’s stream is DRM-protected, but there’s an unencrypted intro clip with logo that is streamed in the clear, and that’s what [Mike] has been focusing on.

Most “fisheye” formats are mapped onto square frames in a way similar to what’s seen here, but this is not what Apple is doing.

[Mike] has been able to determine that the format definitely differs from existing fisheye formats recorded by immersive cameras. First of all, the content is rotated 45 degrees. This spreads the horizon of the video across the diagonal, maximizing the number of pixels available in that direction (a trick that calls to mind the heads in home video recorders being tilted to increase the area of tape it can “see” beyond the physical width of the tape itself.) Doing this also spreads the center-vertical axis of the content across the other diagonal, with the same effect.

There’s more to it than just a 45-degree rotation, however. The rest most closely resembles radial stretching, a form of disc-to-square mapping. It’s close, but [Mike] can’t quite find a complete match for what exactly Apple is doing. Probably we’ll all learn more soon, but for now Apple isn’t saying much.

Videos like VR180 videos and Apple’s immersive format display stereoscopic video that allow a user to look around naturally in a scene. But to really deliver a deeper sense of presence and depth takes light fields.

Fisheye Lens For Your SLR

April 20, 2010 by Mike Szczys 13 Comments

[Bhautik Joshi’s] fisheye lens hack works well and looks OK too. It uses a door peephole from the hardware store as the fisheye and a slide projector lens to enlarge the image for proper sizing on the camera’s sensor. He included an EOS lens adapter so that it is easy to install and remove, then grabbed a soda can and some foam for the rest of the build. This will take those fun bendy pictures but don’t forget that you can correct for that in software if you wish.

Spherical And Stereoscopic Photography

January 14, 2010 by Mike Szczys 6 Comments

[Ryubin’s] experiments with spherical video continue. This time around he’s using two cameras, both with fisheye lenses, to capture 360 degree by 360 degree video. The two cameras mount back to back and each record a 360 degree vertical view in a 180 degree horizontal range. By stitching the two recordings together and synchronizing them by comparing moving objects a seamless spherical video is produced.

He’s got one more trick up his sleeve with this setup. The tripod mount has a pivot point that allows the two cameras to shoot side-by-side instead of back-to-back. This produces a hemispherical video that is stereoscopic. That’s a pretty cheap way to make this type of 3D imaging compared to some of the CES offerings.

There are a few example videos up on his webpage. If you missed it earlier this month, he’s the guy that build a spherical video setup using a light bulb.

Removing Fisheye Distortion

December 21, 2008 by Eliot 14 Comments

fisheye

Reader [alex] had a commercial plugin for fisheye lens correction and wondered exactly what kind of magic was behind it. Was it actually doing line detection? He dropped in a square grid to see what it spit out. The warped result indicated that the transformation was completely independent of the photo’s content. Using this result as a guide he was able to create a similar transform using Warp and save it as a script. The script generates almost identical results and now he knows exactly how little magic is involved.