Are you ready to elevate your interactive possibilities without breaking the bank? If so, explore [Caio Bassetti]’s tutorial on creating a full 3D hand controller using only a webcam, MediaPipe Hands, and Three.js. This hack lets you transform a 2D screen into a fully interactive 3D scene—all with your hand movements. If you’re passionate about low-cost, accessible tech, try this yourself – not much else is needed but a webcam and a browser!
The magic of the project lies in using MediaPipe Hands to track key points on your hand, such as the middle finger and wrist, to calculate depth and positioning. Using clever Three.js tricks, the elements can be controlled on a 3D axis. This setup creates a responsive virtual controller, interpreting hand gestures for intuitive movement in the 3D space. The hack also implements a closed-fist gesture to grab and drag objects and detects collisions to add interactivity. It’s a simple, practical build and it performs reliably in most browsers.
For more on this innovation or other exciting DIY hand-tracking projects, browse our archive on gesture control projects, or check out the full article on Codrops. With tools such as MediaPipe and Three.js, turning ideas into reality gets more accessible than ever.
Neat, I have to take a look at this. I tried to use this same media pipe tool to prototype a custom 6dof “sci-fi ship” controller I wanted to build during a brief stint of playing star citizen. Existing 6dof is like two joysticks + pedals and is extremely unsatisfying. I want something that feels like the way space ship controls work in movies. Move hands closer together, rotate, press forward back, etc.
Anyway the media pipe thing worked enough to validate the idea but needed a lot more filtering for noise than would be practical it seemed to actually use. Interested to see how this deals with that.
Another option would be to use a Spacemouse.
It didn’t work in Chrome for me (no webcam window showed up) but did work in Firefox. But recognition was imperfect: it kept on dropping the objects that I was trying to drag and drop.