Experiencing Visual Deficits And Their Impact On Daily Life, With VR

March 29, 2024 by Donald Papp 14 Comments

Researchers presented an interesting project at the 2024 IEEE Conference on Virtual Reality and 3D User Interfaces: it uses VR and eye tracking to simulate visual deficits such as macular degeneration, diabetic retinopathy, and other visual diseases and impairments.

Typical labels and pill bottles can be shockingly inaccessible to a variety of common visual deficits.

VR offers a unique method of allowing people to experience the impact of living with such conditions, a point driven home particularly well by having the user see for themselves the effect on simple real-world tasks such as choosing a pill bottle, or picking up a mug. Conditions like macular degeneration (which causes loss of central vision) are more accurately simulated by using eye tracking, a technology much more mature nowadays than it was even just a few years ago.

The abstract for the presentation is available here, and if you have some time be sure to check out the main index for all of the VR research demos because there are some neat ones there, including a method of manipulating a user’s perception of the shape of the ground under their feet by electrically-stimulating the tendons of the ankle.

Eye tracking is in a few consumer VR products nowadays, but it’s also perfectly feasible to roll your own in a surprisingly slick way. It’s even been used on jumping spiders to gain insights into the fascinating and surprisingly deep perceptual reality these creatures inhabit.

DIY Eye Tracking For VR Headsets, From A To Z

August 5, 2023 by Donald Papp 3 Comments

Eye tracking is a useful feature in social virtual reality (VR) spaces because it really enhances presence and communication when one’s avatar has a realistic gaze. Most headsets lack this feature, but EyeTrackVR has a completely open source solution ready for anyone willing to put it together.

Camera is visible in lower right corner.

EyeTrackVR is a combination of hardware, software, and 3D printable mounts for attaching a pair of microcontroller boards, cameras, and IR LEDs to just about any existing VR headset out there. An ESP32-based board and tiny camera module watches each eyeball, and under IR illumination the pupil presents as an easily-identified round black area. Software takes care of turning the camera’s view of the pupil into a gaze direction value that can be plugged into other software.

The project is still under active development, but in its current state is perfectly suitable for creating a functional system that can integrate into a variety of existing headsets with printed mounting brackets. Interested? Check out the intro and if it sounds up your alley, dive into the build guide which spells out everything you need to know. Check out the video below for a demo of EyeTrackVR working in VRChat, along with an overview of software support.

We’ve seen headsets built to custom specs that integrate eye tracking, but even if one is repackaging an existing headset that’s a perfect opportunity to include this feature.

Continue reading “DIY Eye Tracking For VR Headsets, From A To Z” →

Hackaday Prize 2023: Eye-Tracking Wheelchair Interface Is A Big Help

May 14, 2023 by Lewin Day 2 Comments

For those with quadriplegia, electric wheelchairs with joystick controls aren’t much help. Typically, sip/puff controllers or eye-tracking solutions are used, but commercial versions can be expensive. [Dhruv Batra] has been experimenting with a DIY eye-tracking solution that can be readily integrated with conventional electric wheelchairs.

The system uses a regular webcam aimed at the user’s face. A Python script uses OpenCV and a homebrewed image segmentation algorithm to analyze the user’s eye position. The system is configured to stop the wheelchair when the user looks forward or up. Looking down commands the chair forward. Glancing left and right steers the chair in the given direction.

The Python script then sends the requisite commands via a TCP connection to an ESP32, which controls a bunch of servos to move the wheelchair’s joystick in the desired manner. This allows retrofitting the device on a wheelchair without having to modify it in an invasive manner.

It’s a neat idea, though it could likely benefit from some further development. A reverse feature would be particularly important, after all. However, it’s a great project that has likely taught [Dhruv] many important lessons about human-machine interfaces, particularly those beyond the ones we use every day.

This project has a good lineage as well — a similar project, EyeDriveOMatic won the Hackaday prize back in 2015.

Hackaday Prize 2023: Eye Tracking On A Budget

May 2, 2023 by Donald Papp 4 Comments

There is a lot to be learned from the experience of building something functional, and even better if doing so doesn’t break the bank. [Sergej Stoetzer]’s 20€ DIY-Eyetracker aims to be an educational process that covers everything from hardware to functional software in an accessible way.

Hardware based on an economical USB endoscope, and can be used as-is or repackaged with IR illumination.

The eye tracker is based on an economical USB endoscope, which is a small camera optimized for up-close applications. By attaching the camera to a pair of common safety glasses so that it looks at one’s eye, some OpenCV and Python code can do simple tracking and interfacing with other projects.

Basic eye tracking — like determining whether a user is looking up, down, left, or right — can be all that’s needed depending on one’s application. That means that it’s possible to get something working with very little hardware and some easy-to-use OpenCV functions.

Even better performance can be had by adding IR illumination and repackaging the camera into a 3D printed enclosure. The pupil of the eye is an aperture in the iris that appears as a black circle, and that’s even more true under IR illumination which is invisible to the naked eye. If you’re curious about what’s inside those USB endoscope cameras and how to remove their IR filter, there are some good pictures of that process in this project.

The ability to get something prototyped quickly and working well enough to learn new things is a valuable skill, and that’s why re-engineering Education is one of the challenges in the 2023 Hackaday Prize.

Eye-Tracking Device Is A Tiny Movie Theatre For Jumping Spiders

October 28, 2021 by Donald Papp 18 Comments

The eyes are windows into the mind, and this research into what jumping spiders look at and why required a clever device that performs eye tracking, but for jumping spiders. The eyesight of these fascinating creatures in some ways has a lot in common with humans. We both perceive a wide-angle region of lower visual fidelity, but are capable of directing our attention to areas of interest within that to see greater detail. Researchers have been able to perform eye-tracking on jumping spiders, literally showing exactly where they are looking in real-time, with the help of a custom device that works a little bit like a miniature movie theatre.

A harmless temporary adhesive on top (and a foam ball for a perch) holds a spider in front of a micro movie projector and IR camera. Spiders were not harmed in the research.

To do this, researchers had to get clever. The unblinking lenses of a spider’s two front-facing primary eyes do not move. Instead, to look at different things, the cone-shaped inside of the eye is shifted around by muscles. This effectively pulls the retina around to point towards different areas of interest. Spiders, whose primary eyes have boomerang-shaped retinas, have an X-shaped region of higher-resolution vision that the spider directs as needed.

So how does the spider eye tracker work? The spider perches on a tiny foam ball and is attached — the help of a harmless and temporary adhesive based on beeswax — to a small bristle. In this way, the spider is held stably in front of a video screen without otherwise being restrained. The spider is shown home movies while an IR camera picks up the reflection of IR off the retinas inside the spider’s two primary eyes. By superimposing the IR reflection onto the displayed video, it becomes possible to literally see exactly where the spider is looking at any given moment. This is similar in some ways to how eye tracking is done for humans, which also uses IR, but watches the position of the pupil.

In the short video embedded below, if you look closely you can see the two retinas make an X-shape of a faintly lighter color than the rest of the background. Watch the spider find and focus on the silhouette of a tasty cricket, but when a dark oval appears and grows larger (as it would look if it were getting closer) the spider’s gaze quickly snaps over to the potential threat.

Feel a need to know more about jumping spiders? This eye-tracking research was featured as part of a larger Science News article highlighting the deep sensory spectrum these fascinating creatures inhabit, most of which is completely inaccessible to humans.

Continue reading “Eye-Tracking Device Is A Tiny Movie Theatre For Jumping Spiders” →

Hands-Free Page Turning

January 7, 2021 by Chris Lott 11 Comments

For people who can’t lift a finger to turn the page on their ebooks, a solution is at hand. Seoul based technology company Visual Camp has adapted their eye tracking algorithms to an ebook reader. (Video, embedded below.) Reportedly this is the first time an ebook reader has been so equipped.

If your eye lingers on the page turn button, it will turn the page. While this particular application seems innocuous, some of the other applications being touted seem a little contrived if not invasive. For example, applying gaze analysis while you are reading a book, they claim to be able to make targeted recommendations for other books.

We’ve discussed eye tracking devices before, but they have utilized hardware. Visual Camp claims their AI-based technology only requires a color camera and can be integrated into existing camera-equipped devices, such an this ebook reader. They also offer a SDK for developers who want to add eye tracking control into their apps. Eye tracking is hard, though, and the devil is in the details. It’d be neat to see what they’re up to.

Continue reading “Hands-Free Page Turning” →

Seek And Ye Shall Command

October 2, 2020 by Brian McEvoy 3 Comments

If we count all the screens in our lives, it takes a hot minute. Some of them are touchscreens, some need a mouse or keyboard, but we are accustomed to all the input devices. Not everyone can use the various methods, like cerebral palsy patients who rely on eye-tracking hardware. Traditionally, that only works on the connected computer, so switching from a chair-mounted screen to a tablet on the desk is not an option. To give folks the ability to control different computers effortlessly [Zack Freedman] is developing a head-mounted eye-tracker that is not tied to one computer. In a way, this is like a KVM switch, but way more futuristic. [Tony Stark] would be proud.

An infrared detector on the headset identifies compatible screens in line of sight and synchs up with its associated HID dongle. A headset-mounted color camera tracks the head position in relation to the screen while an IR camera scans the eye to calculate where the user is focusing. All the technology here is proven, but this new recipe could be a game-changer to anyone who has trouble with the traditional keyboard, mouse, and touchscreen. Maybe QR codes could assist the screen identification and orientation like how a Wii remote and sensor bar work together.