DIY Eye Tracking For VR Headsets, From A To Z

Eye tracking is a useful feature in social virtual reality (VR) spaces because it really enhances presence and communication when one’s avatar has a realistic gaze. Most headsets lack this feature, but EyeTrackVR has a completely open source solution ready for anyone willing to put it together.

Camera is visible in lower right corner.

EyeTrackVR is a combination of hardware, software, and 3D printable mounts for attaching a pair of microcontroller boards, cameras, and IR LEDs to just about any existing VR headset out there. An ESP32-based board and tiny camera module watches each eyeball, and under IR illumination the pupil presents as an easily-identified round black area. Software takes care of turning the camera’s view of the pupil into a gaze direction value that can be plugged into other software.

The project is still under active development, but in its current state is perfectly suitable for creating a functional system that can integrate into a variety of existing headsets with printed mounting brackets. Interested? Check out the intro and if it sounds up your alley, dive into the build guide which spells out everything you need to know. Check out the video below for a demo of EyeTrackVR working in VRChat, along with an overview of software support.

We’ve seen headsets built to custom specs that integrate eye tracking, but even if one is repackaging an existing headset that’s a perfect opportunity to include this feature.

Continue reading “DIY Eye Tracking For VR Headsets, From A To Z”

Hackaday Prize 2023: Eye-Tracking Wheelchair Interface Is A Big Help

For those with quadriplegia, electric wheelchairs with joystick controls aren’t much help. Typically, sip/puff controllers or eye-tracking solutions are used, but commercial versions can be expensive. [Dhruv Batra] has been experimenting with a DIY eye-tracking solution that can be readily integrated with conventional electric wheelchairs.

The system uses a regular webcam aimed at the user’s face. A Python script uses OpenCV and a homebrewed image segmentation algorithm to analyze the user’s eye position. The system is configured to stop the wheelchair when the user looks forward or up. Looking down commands the chair forward. Glancing left and right steers the chair in the given direction.

The Python script then sends the requisite commands via a TCP connection to an ESP32, which controls a bunch of servos to move the wheelchair’s joystick in the desired manner. This allows retrofitting the device on a wheelchair without having to modify it in an invasive manner.

It’s a neat idea, though it could likely benefit from some further development. A reverse feature would be particularly important, after all. However, it’s a great project that has likely taught [Dhruv] many important lessons about human-machine interfaces, particularly those beyond the ones we use every day. 

This project has a good lineage as well — a similar project, EyeDriveOMatic won the Hackaday prize back in 2015.

Hackaday Prize 2023: Eye Tracking On A Budget

There is a lot to be learned from the experience of building something functional, and even better if doing so doesn’t break the bank. [Sergej Stoetzer]’s 20€ DIY-Eyetracker aims to be an educational process that covers everything from hardware to functional software in an accessible way.

Hardware based on an economical USB endoscope, and can be used as-is or repackaged with IR illumination.

The eye tracker is based on an economical USB endoscope, which is a small camera optimized for up-close applications. By attaching the camera to a pair of common safety glasses so that it looks at one’s eye, some OpenCV and Python code can do simple tracking and interfacing with other projects.

Basic eye tracking — like determining whether a user is looking up, down, left, or right — can be all that’s needed depending on one’s application. That means that it’s possible to get something working with very little hardware and some easy-to-use OpenCV functions.

Even better performance can be had by adding IR illumination and repackaging the camera into a 3D printed enclosure. The pupil of the eye is an aperture in the iris that appears as a black circle, and that’s even more true under IR illumination which is invisible to the naked eye. If you’re curious about what’s inside those USB endoscope cameras and how to remove their IR filter, there are some good pictures of that process in this project.

The ability to get something prototyped quickly and working well enough to learn new things is a valuable skill, and that’s why re-engineering Education is one of the challenges in the 2023 Hackaday Prize.

Eye-Tracking Device Is A Tiny Movie Theatre For Jumping Spiders

The eyes are windows into the mind, and this research into what jumping spiders look at and why required a clever device that performs eye tracking, but for jumping spiders. The eyesight of these fascinating creatures in some ways has a lot in common with humans. We both perceive a wide-angle region of lower visual fidelity, but are capable of directing our attention to areas of interest within that to see greater detail. Researchers have been able to perform eye-tracking on jumping spiders, literally showing exactly where they are looking in real-time, with the help of a custom device that works a little bit like a miniature movie theatre.

A harmless temporary adhesive on top (and a foam ball for a perch) holds a spider in front of a micro movie projector and IR camera. Spiders were not harmed in the research.

To do this, researchers had to get clever. The unblinking lenses of a spider’s two front-facing primary eyes do not move. Instead, to look at different things, the cone-shaped inside of the eye is shifted around by muscles. This effectively pulls the retina around to point towards different areas of interest. Spiders, whose primary eyes have boomerang-shaped retinas, have an X-shaped region of higher-resolution vision that the spider directs as needed.

So how does the spider eye tracker work? The spider perches on a tiny foam ball and is attached — the help of a harmless and temporary adhesive based on beeswax — to a small bristle. In this way, the spider is held stably in front of a video screen without otherwise being restrained. The spider is shown home movies while an IR camera picks up the reflection of IR off the retinas inside the spider’s two primary eyes. By superimposing the IR reflection onto the displayed video, it becomes possible to literally see exactly where the spider is looking at any given moment. This is similar in some ways to how eye tracking is done for humans, which also uses IR, but watches the position of the pupil.

In the short video embedded below, if you look closely you can see the two retinas make an X-shape of a faintly lighter color than the rest of the background. Watch the spider find and focus on the silhouette of a tasty cricket, but when a dark oval appears and grows larger (as it would look if it were getting closer) the spider’s gaze quickly snaps over to the potential threat.

Feel a need to know more about jumping spiders? This eye-tracking research was featured as part of a larger Science News article highlighting the deep sensory spectrum these fascinating creatures inhabit, most of which is completely inaccessible to humans.

Continue reading “Eye-Tracking Device Is A Tiny Movie Theatre For Jumping Spiders”

Hands-Free Page Turning

For people who can’t lift a finger to turn the page on their ebooks, a solution is at hand. Seoul based technology company Visual Camp has adapted their eye tracking algorithms to an ebook reader. (Video, embedded below.) Reportedly this is the first time an ebook reader has been so equipped.

If your eye lingers on the page turn button, it will turn the page. While this particular application seems innocuous, some of the other applications being touted seem a little contrived if not invasive. For example, applying gaze analysis while you are reading a book, they claim to be able to make targeted recommendations for other books.

We’ve discussed eye tracking devices before, but they have utilized hardware. Visual Camp claims their AI-based technology only requires a color camera and can be integrated into existing camera-equipped devices, such an this ebook reader. They also offer a SDK for developers who want to add eye tracking control into their apps. Eye tracking is hard, though, and the devil is in the details. It’d be neat to see what they’re up to.

Continue reading “Hands-Free Page Turning”

Seek And Ye Shall Command

If we count all the screens in our lives, it takes a hot minute. Some of them are touchscreens, some need a mouse or keyboard, but we are accustomed to all the input devices. Not everyone can use the various methods, like cerebral palsy patients who rely on eye-tracking hardware. Traditionally, that only works on the connected computer, so switching from a chair-mounted screen to a tablet on the desk is not an option. To give folks the ability to control different computers effortlessly [Zack Freedman] is developing a head-mounted eye-tracker that is not tied to one computer. In a way, this is like a KVM switch, but way more futuristic. [Tony Stark] would be proud.

An infrared detector on the headset identifies compatible screens in line of sight and synchs up with its associated HID dongle. A headset-mounted color camera tracks the head position in relation to the screen while an IR camera scans the eye to calculate where the user is focusing. All the technology here is proven, but this new recipe could be a game-changer to anyone who has trouble with the traditional keyboard, mouse, and touchscreen. Maybe QR codes could assist the screen identification and orientation like how a Wii remote and sensor bar work together.

Give Me A Minute, My Eyes Are Busy

Social cues are tricky, but humans are very good at detecting where someone is looking; that goes a long way toward figuring out where someone is placing their attention. All of this goes right out the window though, when you’re talking with somebody who uses eye-tracking software to speak. [Matthew Oppenheim] with Lancaster University, UK wants to give listeners the message of Give Me a Minute with an easy-to-recognize indicator. His choice is a microBit, which displays a rotating arrow on the LED array while someone composes their speech. He chose the microBit because they are readily available, and you can get cases to fit people’s personalities. After the break, you can see a demonstration, but the graphic appears scrambled because of the screen flicker. The rotating arrow is a clear indicator that someone is writing, whereas a clock might suggest a frozen computer, and a progress bar could not be accurate.

[Matthew] wrote a program for the interpreting computer which recognizes when a message is forming by monitoring the number of black pixels in the composition field. If it changes, someone must be composing a sentence. Many people will try to peek over the speaker’s shoulder and see if they are working, but we’re sure that most readers would join the users of such tech in being unhappy if someone blatantly looks at theirr computer screen while they are typing.

Wheelchairs don’t always have to come from a hospital or supply store, and they don’t have to stay on the ground.

Continue reading Give Me A Minute, My Eyes Are Busy”