Watching Tony Stark wave his hands to manipulate projected constructs is an ever-approaching reality — at least in terms of gesture-tracking. Lift — a prototype built by a team from UC Irvine and FX Palo Alto Laboratory — is able to track up to ten fingers with 1.7 mm accuracy!
Lift’s gesture-tracking is achieved by using a DLP projector, two Arduino MKR1000s, and a light sensor for each digit. Lift’s design allows it to work on virtually any flat surface; the projected image acts as a grid and work area for the user. As their fingers move across the projected surface, the light sensors feed the information from the image to the Arduinos, which infers the location of each finger and translate it into a digital workspace. Sensors may also be mounted on other objects to add functionality.
So far, the team has used Lift as an input device for drawing, as well as using it to feign gesture controls on a standard laptop screen. The next step would be two or more projectors which would allow Lift to function fully and efficiently in three dimensions and directly interacting with projected media content. Can it also operate wirelessly? Yes. Yes, it can.