Re-imagining Telepresence With Humanoid Robots And VR Headsets

July 30, 2024 by Donald Papp 20 Comments

Don’t let the name of the Open-TeleVision project fool you; it’s a framework for improving telepresence and making robotic teleoperation far more intuitive than it otherwise would be. It accomplishes this in part by taking advantage of the remarkable technology packed into modern VR headsets like the Apple Vision Pro and Meta Quest. There are loads of videos on the project page, many of which demonstrate successful teleoperation across vast distances.

Teleoperation of robotic effectors typically takes some getting used to. The camera views are unusual, the limbs don’t move the same way arms do, and intuitive human things like looking around to get a sense of where everything is don’t translate well.

A stereo camera with gimbal streaming to a VR headset complete with head tracking seems like a very hackable design.

To address this, researches provided a user with a robot-mounted, real-time stereo video stream (through which the user can turn their head and look around normally) as well as mapping arm and hand movements to humanoid robotic counterparts. This provides the feedback to manipulate objects and perform tasks in a much more intuitive way. In short, when our eyes, bodies, and hands look and work more or less the way we expect, it turns out it’s far easier to perform tasks.

The research paper goes into detail about the different systems, but in essence, a stereo depth and RGB camera is perched with a 3D printed gimbal atop a humanoid robot frame like the Unitree H1 equipped with high dexterity hands. A VR headset takes care of displaying a real-time stereoscopic video stream and letting the user look around. Hand tracking for the user is mapped to the dexterous hands and fingers. This lets a person look at, manipulate, and handle things without in-depth training. Perhaps slower and more clumsily than they would like, but in an intuitive way all the same.

Interested in taking a closer look? The GitHub repository has the necessary code, and while most of us will never be mashing ADD TO CART on something like the Unitree H1, the reference design for a stereo camera streaming to a VR headset and mirroring head tracking with a two-motor gimbal looks like the sort of thing that would be useful for a telepresence project or two.

Continue reading “Re-imagining Telepresence With Humanoid Robots And VR Headsets” →

Stereo Photography With Smartphones Made Better With Syncing

February 21, 2024 by Maya Posch 16 Comments

Stereo photography has been around for almost as long as photography itself, and it remains a popular way to capture a scene in its 3D glory. Yet despite the fact that pretty much everyone carries one or more cameras with them every day in the form of a smartphone, carrying a stereo photography-capable system with you remains tricky. As [Pascal Martiné] explains in a How-To article, although you can take two smartphones with you, syncing up both cameras to get a stereo image isn’t so straightforward, even though this is essential if you want to prevent jarring shifts between the left and right image.

Custom made twin shutter. (Credit: Pascal Martiné)

Fortunately, having two of the exact same smartphone with the exact same camera modules is not an absolute requirement, as apps like i3DStereoid offer auto-adjustments. But activating the camera trigger on each phone is essential. The usual assortment of wireless remote triggers don’t work well here, and the twin-pairing in i3DStereoid had too much delay for dynamic scenes. This left the wired remote trigger option, but with a dearth of existing stereo trigger options [Pascal] was forced to make his own for two iPhones out of Apple Lightning cables and wired earbud volume controls.

Although the initial prototype more or less worked, [Pascal] found that each iPhone would often ‘decide’ to release the trigger at a slightly different time, requiring multiple attempts at the perfect shot. This led him down a rabbit hole of investigating different camera apps and configurations to make shutter delay as deterministic as possible. Much of this turned out to be due to auto exposure and auto focus, with enabling AE/AF lock drastically increasing the success rate, though this has to be done manually before each shot as an extra step.

With this one tweak, he found that most of the stereo photo pairs are now perfectly synced, while occasionally there is about a ~3 ms jitter, the cause of which he hasn’t tracked down yet, but which could be due to the camera app or iOS being busy with something else.

In the end, this iPhone-based stereo photography setup might not be as reliable or capable as some of the purpose-built rigs we’ve covered over the years, but it does get extra points for portability.

Clever Stereo Camera Uses Sony Wireless Camera Modules

April 30, 2022 by Jenny List 13 Comments

Stereophotography cameras are difficult to find, so we’re indebted to [DragonSkyRunner] for sharing their build of an exceptionally high-quality example. A stereo camera has two separate lenses and sensors a fixed distance apart, such that when the two resulting images are viewed individually with each eye there is a 3D effect. This camera takes two individual Sony cameras and mounts them on a well-designed wooden chassis, but that simple description hides a much more interesting and complex reality.

Sony once tested photography waters with the QX series — pair of unusual mirrorless camera models which took the form of just the sensor and lens. A wireless connection to a smartphone allows for display and data transfer. This build uses two of these, with a pair of Android-running Odroid C2s standing in for the smartphones. Their HDMI video outputs are captured by a pair of HDMI capture devices hooked up to a Raspberry Pi 4, and there are a couple of Arduinos that simulate mouse inputs to the Odroids. It’s a bit of a Rube Goldberg device, but it allows the system to use Sony’s original camera software. An especially neat feature is that the camera unit and display unit can be parted for remote photography, making it an extremely versatile camera.

It’s good to see a stereo photography camera designed specifically for high-quality photography, previous ones we’ve seen have been closer to machine vision systems.

OpenCV And Depth Camera Spots Weeds

January 31, 2021 by Donald Papp 25 Comments

Using vision technology to identify weeds in agriculture is an area of active development, and a team of researchers recently shared their method of using a combination of machine vision plus depth information to identify and map weeds with the help of OpenCV, the open-source computer vision library. Agriculture is how people get fed, and improving weed management is one of its most important challenges.

Many current efforts at weed detection and classification use fancy (and expensive) multispectral cameras, but PhenoCV-WeedCam relies primarily on an OAK-D stereo depth camera. The system is still being developed, but is somewhat further along than a proof of concept. The portable setups use a Raspberry Pi, stereo camera unit, power banks, an Android tablet for interfacing, and currently require an obedient human to move and point them.

It’s an interesting peek at the kind of hands-on work that goes into data gathering for development. Armed with loads of field data from many different environments, the system can use the data to identify grasses, broad leaf plants, and soil in every image. This alone is useful, but depth information also allows the system to estimate overall plant density as well as try to determine the growth center of any particular plant. Knowing that a weed is present is one thing, but to eliminate it with precision — for example with a laser or mini weed whacker on a robot arm — knowing where the weed is actually growing from is an important detail.

PhenoCV-WeedCam (GitHub repository) is not yet capable of real-time analysis, but the results are promising and that’s the next step. The system currently must be carried by people, but could ultimately be attached to a robotic platform made specifically to traverse fields.