Unraveling The Secrets Of Apple’s Mysterious Fisheye Format

April 18, 2024 by Donald Papp 12 Comments

Apple has developed a proprietary — even mysterious — “fisheye” projection format used for their immersive videos, such as those played back by the Apple Vision Pro. What’s the mystery? The fact that they stream their immersive content in this format but have provided no elaboration, no details, and no method for anyone else to produce or play back this format. It’s a completely undocumented format and Apple’s silence is deafening when it comes to requests for, well, anything to do with it whatsoever.

Probably those details are eventually forthcoming, but [Mike Swanson] isn’t satisfied to wait. He’s done his own digging into the format and while he hasn’t figured it out completely, he has learned quite a bit and written it all up on a blog post. Apple’s immersive videos have a lot in common with VR180 type videos, but under the hood there is more going on. Apple’s stream is DRM-protected, but there’s an unencrypted intro clip with logo that is streamed in the clear, and that’s what [Mike] has been focusing on.

Most “fisheye” formats are mapped onto square frames in a way similar to what’s seen here, but this is not what Apple is doing.

[Mike] has been able to determine that the format definitely differs from existing fisheye formats recorded by immersive cameras. First of all, the content is rotated 45 degrees. This spreads the horizon of the video across the diagonal, maximizing the number of pixels available in that direction (a trick that calls to mind the heads in home video recorders being tilted to increase the area of tape it can “see” beyond the physical width of the tape itself.) Doing this also spreads the center-vertical axis of the content across the other diagonal, with the same effect.

There’s more to it than just a 45-degree rotation, however. The rest most closely resembles radial stretching, a form of disc-to-square mapping. It’s close, but [Mike] can’t quite find a complete match for what exactly Apple is doing. Probably we’ll all learn more soon, but for now Apple isn’t saying much.

Videos like VR180 videos and Apple’s immersive format display stereoscopic video that allow a user to look around naturally in a scene. But to really deliver a deeper sense of presence and depth takes light fields.

Stereo Photography With Smartphones Made Better With Syncing

February 21, 2024 by Maya Posch 16 Comments

Stereo photography has been around for almost as long as photography itself, and it remains a popular way to capture a scene in its 3D glory. Yet despite the fact that pretty much everyone carries one or more cameras with them every day in the form of a smartphone, carrying a stereo photography-capable system with you remains tricky. As [Pascal Martiné] explains in a How-To article, although you can take two smartphones with you, syncing up both cameras to get a stereo image isn’t so straightforward, even though this is essential if you want to prevent jarring shifts between the left and right image.

Custom made twin shutter. (Credit: Pascal Martiné)

Fortunately, having two of the exact same smartphone with the exact same camera modules is not an absolute requirement, as apps like i3DStereoid offer auto-adjustments. But activating the camera trigger on each phone is essential. The usual assortment of wireless remote triggers don’t work well here, and the twin-pairing in i3DStereoid had too much delay for dynamic scenes. This left the wired remote trigger option, but with a dearth of existing stereo trigger options [Pascal] was forced to make his own for two iPhones out of Apple Lightning cables and wired earbud volume controls.

Although the initial prototype more or less worked, [Pascal] found that each iPhone would often ‘decide’ to release the trigger at a slightly different time, requiring multiple attempts at the perfect shot. This led him down a rabbit hole of investigating different camera apps and configurations to make shutter delay as deterministic as possible. Much of this turned out to be due to auto exposure and auto focus, with enabling AE/AF lock drastically increasing the success rate, though this has to be done manually before each shot as an extra step.

With this one tweak, he found that most of the stereo photo pairs are now perfectly synced, while occasionally there is about a ~3 ms jitter, the cause of which he hasn’t tracked down yet, but which could be due to the camera app or iOS being busy with something else.

In the end, this iPhone-based stereo photography setup might not be as reliable or capable as some of the purpose-built rigs we’ve covered over the years, but it does get extra points for portability.

Photography, The Stereo Way

December 19, 2022 by Bryan Cockfield 19 Comments

Most consumer-grade audio equipment has been in stereo since at least the 1960s, allowing the listener to experience sounds with a three-dimensional perspective as if they were present when the sound was originally made. Stereo photography has lagged a little behind the stereo audio trend, though, with most of the technology existing as passing fads or requiring clumsy hardware to experience fully. Not so with the DIY stereoscopic cameras like this one produced by this group of 3D photography enthusiasts, who have also some methods to view the photos in 3D without any extra hardware.

The camera uses two imaging sensors to produce a stereo image. One sensor is fixed, and the other is on a slider which allows the user to adjust the “amount” of 3D effect needed for any particular photo. [Jim] is using this camera mostly for macro photography, which means that he only needs a few millimeters of separation between the two sensors to achieve the desired effect, but for more distant objects more separation can be used. The camera uses dual Raspberry Pi processors, a lithium battery, and a touch screen interface. It includes a ton of features as well including things like focus stacking, but to get a more full experience of this build we’d highly recommend checking out the video after the break.

As for viewing the photographs, these stereoscopic 3D images require nothing more than a little practice to view them. This guide is available with some simple examples to get started, and while it does at first feel like a Magic Eye puzzle from the late 90s, it quickly becomes intuitive. Another guide has some more intricate 3D maps at the end to practice on as well. This is quite the step up from needing to use special glasses or a wearable 3D viewer of some sort. There are also some methods available to create 3D images from those taken with a regular 2D camera as well.

Thanks to [Bill] for the tip and the additional links to the guides for viewing these images!

Continue reading “Photography, The Stereo Way” →

Raspberry Pi Cameras Stand In For Stereo Microscope

July 9, 2021 by Tom Nardi 9 Comments

Handling tiny surface mount components and inspecting PCBs is a lot easier with a nice stereo microscope, but because of their cost and bulk, most hobbyists have to do without. At best they might have a basic digital microscope, but with only one camera, they can only show a 2D image that’s not ideal for detail work.

The team behind [Stereo Ninja] hopes to improve on the situation by developing a stereoscopic vision system that puts tiny objects up on the big screen in three dimensions. Utilizing the Raspberry Pi Compute Module, a custom carrier board that enables the use of both MIPI CSI camera interfaces, and a 3D gaming monitor, their creation combines the capabilities of a traditional stereo microscope with the flexibility of a digital solution.

With two Raspberry Pi cameras suspended over the work area, and the addition of plenty of LED light, Stereo Ninja is able to generate the 3D image required by the monitor. While the camera’s don’t have the same magnification you’d get from a microscope, they’re good enough for enlarging SMD parts, and looking at a big screen monitor certainly beats hunching over the eyepiece of a traditional microscope. Especially if you’re trying to show something to a group of people, like at a hackerspace.

Of course, not everyone has a large 3D gaming monitor on their workbench. In fact, given how poorly the tech went over with consumers the last time it was pushed on us, we’d wager more hackers have stereo microscopes than 3D displays. Which is why the team’s next step is to have the Raspberry Pi generate the signals required by the shutter glasses, allowing Stereo Ninja to show a three dimensional image on 2D monitors; bringing this valuable capability to far larger audience than has previously been possible.

Continue reading “Raspberry Pi Cameras Stand In For Stereo Microscope” →

Scanner Captures View-Master Discs As Glorious 3D Videos

May 12, 2021 by Dan Maloney 25 Comments

The toys of the past may have been cheesy, but you can’t deny the creativity needed to build something engaging without any electronics. One stalwart toy from this category is View-Master, the little stereoscopic slide viewer that brought the world to life in seven vibrant scenes. And digitizing these miniature works of art is the purpose of this neat View-Master reel scanner project.

If you haven’t had the pleasure of using a View-Master, the gist is that a flat cardboard disc ringed with 14 color transparencies was inserted into a plastic viewer. Binocular eyepieces showed scenes from opposing pairs of slides, which were illuminated by a frosted screen and room lighting. The scenes were photographed from slightly different angles, leading to a stereoscopic image that was actually pretty good quality.

In the video below, project creator [W. Jason Altice] describes View-Master as “the YouTube of the 1950s.” We partially agree; with only seven frames to tell a story, we’d say it’s more like TikTok than YouTube. Regardless, capturing these mini-movies requires quite a bit of complexity. All the parts for the reel carousel are 3D-printed, with a small stepper to advance the reel and an optical sensor to register its position. A ring of RGB LEDs beneath the reel illuminates the slides; being able to control the color of the light helps with color balancing for slides with faded colors. An 8-megapixel camera captures each slide, and some pretty slick software helps with organizing the image pairs, tweaking their alignment, capturing the captions from the disc, and stitching everything into a video.

There’s a whole YouTube channel devoted to View-Master captures, which are best viewed with a Google Cardboard or something similar. Even without the 3D effect, it’s still pretty cool to watch [Popeye] beat up a nuke again.

Continue reading “Scanner Captures View-Master Discs As Glorious 3D Videos” →

Robots Invade Your Personal Space

September 5, 2018 by Brian McEvoy 13 Comments

If you have ever had to complete a task such as building a LEGO model over a remote connection, you will know that the challenges are like an absurd grade school group project. The person giving directions often has trouble describing what they are thinking, and the person doing the work has trouble interpreting what the instructor wants. “Turn the blue block over. No, only half way. Go back. Now turn it. No, the other way. NO! Not clockwise, downward. That’s Upward! Geez. Are you even listening‽” Good times.

While you may not be in this situation every day, the Keio University of Japan has an intuitive way to give instructors a way to physically interact with an instructee through a Moore/Swayze experience. The instructor has a camera in typical pirate parrot placement over the shoulder. Two arms are controlled by the instructor who can see through stereoscopic cameras to have a first-person view from across the globe. This natural way to interact with the user’s environment allows muscle memory to pass from the instructor to the wearer.

For some of the other styles of telepresence, see this deep-sea bot and a cylindrical screen that looks like someone is beaming up directly from the holodeck.

Continue reading “Robots Invade Your Personal Space” →

Immersive VR With A 200-Degree Stereoscopic Camera

October 29, 2017 by James Hobson 14 Comments

VR is in vogue, but getting on board requires a steep upfront cost. Hackaday.io user [Colin Pate] felt that $800 was a bit much for even the cheapest commercial 360-degree 3D camera, so he thought: ‘why not make my own for half that price?’

[Pate] knew he’d need a lot of bandwidth and many GPIO ports for the camera array, so he searched out the Altera Cyclone V SOC FPGA and a Terasic DE10-Nano development board to host it. At present, he has four Uctronics OV5642 cameras on his rig, chosen for their extensive documentation and support. The camera mount itself is a 3D-printed octagon so eight of the OC5642 can capture a full 360-degree photo.

Next: producing an image!

Continue reading “Immersive VR With A 200-Degree Stereoscopic Camera” →