Immersive Virtual Reality From The Humble Webcam

March 12, 2023 by Joseph Long 12 Comments

[Russ Maschmeyer] and Spatial Commerce Projects developed WonkaVision to demonstrate how 3D eye tracking from a single webcam can support rendering a graphical virtual reality (VR) display with realistic depth and space. Spatial Commerce Projects is a Shopify lab working to provide concepts, prototypes, and tools to explore the crossroads of spatial computing and commerce.

The graphical output provides a real sense of depth and three-dimensional space using an optical illusion that reacts to the viewer’s eye position. The eye position is used to render view-dependent images. The computer screen is made to feel like a window into a realistic 3D virtual space where objects beyond the window appear to have depth and objects before the window appear to project out into the space in front of the screen. The resulting experience is like a 3D view into a virtual space. The downside is that the experience only works for one viewer.

Eye tracking is performed using Google’s MediaPipe Iris library, which relies on the fact that the iris diameter of the human eye is almost exactly 11.7 mm for most humans. Computer vision algorithms in the library use this geometrical fact to efficiently locate and track human irises with high accuracy.

Generation of view-dependent images based on tracking a viewer’s eye position was inspired by a classic hack from Johnny Lee to create a VR display using a Wiimote. Hopefully, these eye-tracking approaches will continue to evolve and provide improved motion-responsive views into immersive virtual spaces.

Human Vs. AI Drone Racing At The University Of Zurich

November 26, 2022 by Joseph Long 12 Comments

[Thomas Bitmatta] and two other champion drone pilots visited the Robotics and Perception Group at the University of Zurich. The human pilots accepting the challenge to race drones against Artificial Intelligence “pilots” from the UZH research group.

The human pilots took on two different types of AI challengers. The first type leverages 36 tracking cameras positioned above the flight arena. Each camera captures 400 frames per second of video. The AI-piloted drone is fitted with at least four tracking markers that can be identified in the captured video frames. The captured video is fed into a computer vision and navigation system that analyzes the video to compute flight commands. The flight commands are then transmitted to the drone over the same wireless control channel that would be used by a human pilot’s remote controller.

The second type of AI pilot utilizes an onboard camera and autonomous machine vision processing. The “vision drone” is designed to leverage visual perception from the camera with little or no assistance from external computational power.

Ultimately, the human pilots were victorious over both types AI pilots. The AI systems do not (yet) robustly accommodate unexpected deviation from optimal conditions. Small variations in operating conditions often lead to mistakes and fatal crashes for the AI pilots.

Both of the AI pilot systems utilize some of the latest research in machine learning and neural networking to learn how to fly a given track. The systems train for a track using a combination of simulated environments and real-world flight deployments. In their final hours together, the university research team invited the human pilots to set up a new course for a final race. In less than two hours, the AI system trained to fly the new course. In the resulting real-world flight of the AI drone, its performance was quite impressive and shows great promise for the future of autonomous flight. We’re betting on the bots before long.

Continue reading “Human Vs. AI Drone Racing At The University Of Zurich” →

Brainstorming

November 19, 2022 by Elliot Williams 5 Comments

One of the best things about hanging out with other hackers is the freewheeling brainstorming sessions that tend to occur. Case in point: I was at the Electronica trade fair and ended up hanging out with [Stephen Hawes] and [Lucian Chapar], two of the folks behind the LumenPnP open-source pick and place machine that we’ve covered a fair number of times in the past.

Among many cool features, it has a camera mounted on the parts-moving head to find the fiducial markings on the PCB. But of course, this mean a camera mounted to an almost general purpose two-axis gantry, and that sent the geeks’ minds spinning. [Stephen] was talking about how easy it would be to turn into a photo-stitching macrophotography rig, which could yield amazingly high resolution photos.

Meanwhile [Lucian] and I were thinking about how similar this gantry was to a 3D printer, and [Lucian] asked why 3D printers don’t come with cameras mounted on the hot ends. He’d even shopped this idea around at the East Coast Reprap Festival and gotten some people excited about it.

So here’s the idea: computer vision near extruder gives you real-time process control. You could use it to home the nozzle in Z. You could use it to tell when the filament has run out, or the steppers have skipped steps. If you had it really refined, you could use it to compensate other printing defects. In short, it would be a simple hardware addition that would open up a universe of computer-vision software improvements, and best of all, it’s easy enough for the home gamer to do – you’d probably only need a 3D printer.

Now I’ve shared the brainstorm with you. Hope it inspires some DIY 3DP innovation, or at least encourages you to brainstorm along below.

CV Based Barking Dog Keeps Home Secure, Doesn’t Need Walking

July 29, 2022 by Ryan Flowers 26 Comments

Meet [Tanner]. [Tanner] is a hacker who also appreciates the security of their home while they’re out of town. After doing some research about home security, they found that it doesn’t take much to keep a house from being broken into. It’s true that truly determined burglars might be more difficult to avoid. But, for the opportunistic types who don’t like having their appendages treated like a chew toy or their face on the local news, the steaks are lowered. All it might take is a security camera or two, or a big barking dog to send them on their way. Rather than running to the local animal shelter, [Tanner] used parts that were already sitting around to create a solution to the problem: A computer vision triggered virtual dog.

Continue reading “CV Based Barking Dog Keeps Home Secure, Doesn’t Need Walking” →

Computer Vision Extracts Lightning From Footage

July 1, 2022 by Bryan Cockfield 16 Comments

Lightning is one of the more mysterious and fascinating phenomenon on the planet. Extremely powerful, but each strike on average only has enough energy to power an incandescent bulb for an hour. The exact mechanism that starts a lightning strike is still not well understood. Yet it happens 45 times per second somewhere on the planet. While we may not gain a deeper scientific appreciation of lightning anytime soon, but we can capture it in various photography thanks to this project which leverages computer vision ~~machine learning~~ to pull out the best frames of lightning.

The project’s creator, [Liam], built this as a tool for stormchasers and photographers so that they can film large amounts of time and not have to go back through their footage manually to pull out the frames with lightning strikes. The project borrows from a similar project, but this one adds Python 3 capabilities and runs on a tiny netbook for more easy field deployment. It uses OpenCV for object recognition, using video files as the source data, and features different modes to recognize different types of lightning.

The software is free and open source, and releases are supported for both Windows and Linux. So far, [Liam] has been able to capture all kinds of electrical atmospheric phenomenon with it including lightning, red sprites, and elves. We don’t see too many projects involving lightning around here, partly because humans can only generate a fraction of the voltage potential needed for the average lightning strike.

Performing Magic With A Little High-Tech Help

June 30, 2022 by Bryan Cockfield 13 Comments

Doing magic with cards involves a lot of precise dexterity to know which card is where. For plenty of tricks, this is often knowledge and control of a single card or a small number of cards. But knowing the exact position of every single card in the deck could certainly be helpful, so the Nettle Magic Project was created to allow magicians to easily identify the location of cards in the deck.

The system works through the use of computer vision to identify a series of marks on the short edge of a stack of cards. The marks can be printed in IR- or UV-sensitive ink to make them virtually invisible, but for demonstration these use regular black ink. Each card has landmarks printed on either side of a set of bit markers which identify the cards. A computer is able to quickly read the marks and identify each card in order while the deck is still stacked, aiding the magician in whichever trick they need to perform.

The software only runs on various Apple devices right now, including iPhones and iPads, but the software is readily available fore experimentation if you are a magician looking to try something like this out. Honestly, we don’t see too many builds focusing on magic, sleight-of-hand or otherwise, and we had to go back over a decade to find a couple of custom magical builds from a magician named [Mario].

Thanks to [Tim] for the tip!

Another Rubik’s Cube Robot Is Simple But Slow

June 13, 2022 by Al Williams 5 Comments

[AndreaFavero] says that the CuboTino emphasizes simplicity and cost-savings over speed. However, solving the puzzle in about 90 seconds is still better than we can do. The plucky solver uses a Pi and a camera to understand what the cube looks like and then runs it through a solver to determine how to move.

Watching the video below, we were impressed with the mechanics. The titled surface solves a few problems and makes manipulation easier. The way the mechanics are arranged, it only takes a pair of servos to flip the cube around as you like. Continue reading “Another Rubik’s Cube Robot Is Simple But Slow” →