Virtual reality is becoming more of a thing, now that we have high quality headsets and the computing power to generate attractive environments. Many VR systems use controllers held in mid air, or camera-based systems that track limbs and hands for interaction. However, productivity scenarios often require prolonged interactions over a long period of time, which typically necessitates working at surfaces that allow the body to rest intermittently. To help facilitate this, a group of researchers at ETH Zurich developed TapID, including a preprint paper (PDF) that will be presented at IEEE VR 2021 later this month.
TapID consists of a wristband that carries two motion sensors, with one worn on each wrist. This allows TapID to detect taps from each of the user’s fingers individually, thanks to a machine learning algorithm that analyses the unique vibrations through your skeletal system. This is demonstrated as being useful for VR environments, where the user can type into a virtual keyboard, or interact with virtual objects on a surface, using their fingers as they would in the real world. This is a sensor fusion with the features of modern VR headsets that include hand tracking. The TapID wristbands deliver granularity and detection of small motions that is not nearly as accurate through headset-mounted senors and camera-based detection.
We’re not entirely convinced of the utility of sitting down in a virtual environment to type at a fake keyboard when monitors and real keyboards are more tactile and cheaper. However, having a device that can accurately determine individual finger interactions is sure to have applications in VR. And whether or not the demonstrated use cases are viable, the technology does indeed work.
It’s exciting to see the wrist-band form factor. It brings to mind the possibility of improving tap interactions in smart watches for non-VR uses. We envision chorded keyboard type gestures that detect which fingers are tapping but don’t need positional accuracy.
Those experimenting in VR interfaces may find it useful to reverse engineer what’s already out on the market, as we’ve featured before. Or, you can simply build your own! Video after the break.
I have spent quite a few hours working in ImmersedVR for the Oculus Quest 2. It takes a bit of getting used but working with your desktop and virtual screens is pretty cool. You can sync up the position of your physical keyboard with a projected keyboard which is nice.
The biggest advantage for me comes from the total lack of distractions when you are working while floating in the middle of a nebula.
I am surprised that we haven’t seen at least some attempt by the Autodesk or Dassault at a VR plugin which I think could really boost productivity.
Please can you tell us more, it sounds fantastic! What is the minimum equipment required to work like this? Can it be done facebook-free? Can it adapt for prescription lenses and things like astigmatisms?
If your eyes are very poor at stereo vision the 3d effect of VR won’t work well if at all. And you might not find the whole thing might not be very readable – its taking something like a high end phone screen and blowing it up to a nearly normal FOV focused something around the 75cm from your face region – so an effective screen size that is pretty large…
But most headsets can get prescription lens inserts, and many accommodate glasses just fine anyway (just be careful to not scratch the lens).
If you can its worth trailing it somewhere first, personally I do get on pretty with with my Original Vive. Though my IPD is slightly larger than it will go to, it is noticeably front heavy after a while which gets a bit uncomfortable where the straps dig in, and being a slightly older generation it can’t really render text that well below something like a perceived 16pt font size – its not unreadable, just not nice to read, I understand the latest and greatest ones have bumped the pixel count up enough to be much nicer. I’m still running on a 980ti, for many things its enough, though I was on course to upgrade last year, as it was showing its age.. Boy is that not happening for a while as it stands, bloody waste of electricity crypto bollocks…
Quest 2 can’t be facebook free BUT headsets like the reverb g2 (and basically any other WMR headset) is an excellent choice for a usecase like this. Vr is actually relatively cheap if you’ve got a computer that can handle it.
I use immersedVR with Quest 2 connected through wifi to my lenovo thinkpad yoga(4 years old) running ubuntu, it works fine. multiple screens are not supported in linux yet.. you can also sideload termux amd use BT mouse / keyboard to run standalone.
actually handtracking and virtual keyboard is not horrible, it works, but slowly… Quest 2 is way better then GearVR (note 1 or s9), text looks fine, if you align the headset correctly. can not spot the pixels any more.
Autodesk : https://www.autodesk.com/solutions/virtual-reality
Dassault : https://www.youtube.com/watch?v=TkGJw2t6vIw
Neat idea though. Really need to see the data and it be close to 100% all of the time or it really isn’t going to work in a way that it is being portrayed. Change much of anything and it feels like the data is going to be thrown off enough to just not work at all or even if it drops to 98%, it is still going to be almost unusable. Curious to hear more about practical, real world use though.
Yeah, I doubt this is as good as wearing a sensor laden glove, be that the Polysense style ‘smart’ fabric or just a collection of IMU’s at the digits… Its a darn interesting concept though,
Word of caution: make sure the surface you are using for your virtual keyboard is capable of absorbing the impact of the fingers. I worked on a ‘laser-projected’ keyboard project some years back, and our test users reported joint and bone problems at a higher rate than the group using a regular keyboard.
Repeatedly impacting a rigid surface is not good for you, the force has to go somewhere and if the keyboard isn’t absorbing it, it gets reflected into your hands. Some of the people in our beta program showed evidence of micro-fractures in their metacarpals after extended use.
What’s a surface that would absorb the impact? Are you talking about something like a mouse mat?
I’m not at all surprised that the laser projected keyboards ended up hurting people, I didn’t know they’d work with a soft surface though. Seems ironic that you’d have to carry around something as large as a regular keyboard in order to use one safely.
We tested with Rubber mats and some foam core and they worked fairly well. For a test, we projected a keyboard onto board that was held up by a chest harness, and that also helped cut down injuries. We also tested using a human analog we had in the lab that helped the most (Essentially a medical skeleton that has been covered in ballistic gel with a skin-like texture covering the whole thing)
But, yeah, having to carry something around to not damage fingers was one of the biggest reasons why the project was ultimately cancelled. Ou test subjects were mostly college students in the early- / mid-20s, and since they were experiencing those levels of injury, we were afraid what would happen to someone with weakened bones.
It seems like gloves with rubber tips would solve the issue. Rubber thimbles?
We did consider something like that, but our testers weren’t fans of it because they maybe it difficult for them to pick up pieces of paper and write legibly. We were aiming to make devices for the medical industry, so were trying to make something that would integrate into the bottom of a monitor/thin client that would project the keyboard onto a table that could be cleaned with standard disinfectants. We did successfully build a monitor that would survive being dunked into a tub of bleach, so that was neat. I still have a finished prototype on my desk, I’m using it as part of a company-wide VDI deployment.
That is a really interesting idea. Can you rest your wrist on the surface? It seems as though that would mess up the detection. If you can’t I would find it uncomfortable.
“when monitors and real keyboards are more tactile and cheaper.”
Agreed on keyboards but monitors are only cheap up to a point. The bigger and fancier they get the more expensive they get, but they’ll get exponentially more expensive than their fancy VR headset equivalents as the area goes up due to manufacturing defects lowering yields.
I don’t know where the cutoff point is but it’s probably 30-50″ as a 49″ curved 5120×1440 monitor is more expensive than a good VR headset.
There’s already a commercial version of this technology: https://www.tapwithus.com/
Though their aim was at mobile use, I have used my tap keyboard in VR. You should even be able to use it while holding most controllers.
I have one of these, you won’t be surprised to learn that it’s in a drawer somewhere, the tap recognition was super touchy. And the comment about pounding your fingers on a table or hard surface above is definitely a thing.
Anything that requires detection of the vibration of your fingers on a surface gets a pretty heavy dose of skepticism from me after trying the tap.