TapType: AI-Assisted Hand Motion Tracking Using Only Accelerometers

The team from the Sensing, Interaction & Perception Lab at ETH Zürich, Switzerland have come up with TapType, an interesting text input method that relies purely on a pair of wrist-worn devices, that sense acceleration values when the wearer types on any old surface. By feeding the acceleration values from a pair of sensors on each wrist into a Bayesian inference classification type neural network which in turn feeds a traditional probabilistic language model (predictive text, to you and I) the resulting text can be input at up to 19 WPM with 0.6% average error. Expert TapTypers report speeds of up to 25 WPM, which could be quite usable.

Details are a little scarce (it is a research project, after all) but the actual hardware seems simple enough, based around the Dialog DA14695 which is a nice Cortex M33 based Bluetooth Low Energy SoC. This is an interesting device in its own right, containing a “sensor node controller” block, that is capable of handling sensor devices connected to its interfaces, independant from the main CPU. The sensor device used is the Bosch BMA456 3-axis accelerometer, which is notable for its low power consumption of a mere 150 μA.

User’s can “type” on any convenient surface.

The wristband units themselves appear to be a combination of a main PCB hosting the BLE chip and supporting circuit, connected to a flex PCB with a pair of the accelerometer devices at each end. The assembly was then slipped into a flexible wristband, likely constructed from 3D printed TPU, but we’re just guessing really, as the progression from the first embedded platform to the wearable prototype is unclear.

What is clear is that the wristband itself is just a dumb data-streaming device, and all the clever processing is performed on the connected device. Training of the system (and subsequent selection of the most accurate classifier architecture) was performed by recording volunteers “typing” on an A3 sized keyboard image, with finger movements tracked with a motion tracking camera, whilst recording the acceleration data streams from both wrists. There are a few more details in the published paper for those interested in digging into this research a little deeper.

The eagle-eyed may remember something similar from last year, from the same team, which correlated bone-conduction sensing with VR type hand tracking to generate input events inside a VR environment.

Continue reading “TapType: AI-Assisted Hand Motion Tracking Using Only Accelerometers”

Spooky USB Baby Types Out Messages From Beyond

You might think it’s a bit early for us to be running Halloween hacks, but don’t worry. While this microcontroller-equipped doll that mimics a USB keyboard to type out messages in the creepiest way possible might seem like a gag gift you’d get after attending somebody’s bone-chilling holiday bash, creator [Jonathan] actually put it together for a friend’s wedding. So not only is it an interesting piece of hacked together hardware, but it’s also a great reminder about the importance of having a wedding registry.

Even if this seems like a rather unusual wedding gift from an outsider’s perspective (for the record, pranks involving this “haunted doll” have been a running gag between them since their school days), we can’t help but be impressed with the way [Jonathan] implemented it. An ATtiny85-powered Digispark is hidden inside the doll, along with a simple USB 2.0 hub that supposedly eases some teething issues the diminutive development board has with newer USB 3.0 ports. Through the use of V-USB, this lets the baby type out messages once plugged into the recipient’s computer.

Soldering the Digispark to a cheap USB hub keeps newer computers happy.

Now he could have just stopped there, but [Jonathan] wanted this to be an interactive experience. Specifically, he wanted the baby to present the newlyweds with a personally test of sorts, and that meant taking user input. He came up with the clever user interface demonstrated in the video below, which responds to changes in the system’s “Caps Lock” state.

This platform-agnostic solution lets the user navigate the doll’s menu system by tapping a single key, although the Chromebook users out there will have to break out the Alt key to play along. It’s a neat trick for getting two-way communication going between a MCU and a computer without any client-side software, and worth filing away mentally for future non-haunted projects. It’s also worth checking out the effort [Jonathan] put into optimizing everything to fit into the chip’s paltry 6012 bytes of flash.

Incidentally, this is a good a time as any to remind readers that our Halloween Hackfest contest is live right now and taking entries until October 11th. If you’ve got any cursed bar mitzvah gifts you’ve been putting the finishing touches on, we’d love to see them.

Continue reading “Spooky USB Baby Types Out Messages From Beyond”

Touch-Typing On Fingertips? Prototype Says It Could Work

The fingertips are covered in touch sensors, each intended to be tapped by the thumbtip of the same hand.

Touch-typing with thumbs on a mobile phone keyboard is a pretty familiar way to input text, and that is part of what led to BiTipText, a method of allowing bimanual text input using fingertips. The idea is to treat the first segments of the index fingers as halves of a tiny keyboard, whose small imaginary keys are tapped with the thumbs. The prototype shown here was created to see how well the concept could work.

The prototype hardware uses touch sensors that can detect tap position with a high degree of accuracy, but the software side is where the real magic happens. Instead of hardcoding a QWERTY layout and training people to use it, the team instead ran tests to understand users’ natural expectations of which keys should be on which finger, and how exactly they should be laid out. This data led to an optimized layout, and when combined with predictive features, test participants could achieve an average text entry speed of 23.4 words per minute.

Judging by the prototype hardware, it’s understandable if one thinks the idea of fingertip keyboards may be a bit ahead of its time. But considering the increasingly “always on, always with you” nature of personal technology, the goal of the project was more about investigating ways for users to provide input in fast and subtle ways. It seems that the idea has some merit in principle. The project’s paper can be viewed online, and the video demonstration is embedded below.

Continue reading “Touch-Typing On Fingertips? Prototype Says It Could Work”