Welcome to the first installment of Inputs of Interest. In this column, we’re going to take a look at various input devices and methods, discuss their merits, give their downsides a rundown, and pontificate about the possibilities they present for hackers. I’ll leave it open to the possibility of spotlighting one particular device (because I already have one in mind), but most often the column will focus on input concepts.
Some inputs are built for having fun. Some are ultra-specific shortcuts designed to do work. Others are assistive devices for people with low mobility. And many inputs blur the lines between these three ideas. This time on Inputs of Interest, we’re going to chew on the idea of oral inputs — those driven by the user’s tongue, teeth, or both.
Unless you’ve recently bitten it, burned it, or had it pierced, you probably don’t think much about your tongue. But the tongue is a strong, multi-muscled organ that rarely gets tired. It’s connected to the brain by a cranial nerve, and usually remains undamaged in people who are paralyzed from the neck down. This makes it a viable input-driving option for almost everyone, regardless of ability. And yet, tongues and mouths in general seem to be under-utilized as input appendages.
Ideally, any input device should be affordable and/or open source, regardless of the driving appendage. Whether the user is otherwise able-bodied or isn’t, there’s no reason the device shouldn’t be as useful and beautiful as possible.
Meaningful Controls
Mouth-based interfaces could be incredibly useful for everyone, and not just as assistive technology for the alternatively-abled. But much like any other type of input device, their utility depends on a few factors, starting with the number of controls. Too little control, and there’s no point in teaching the tongue new tricks.
But too many controls will get out of hand fast. Wiring up a second 108-key board as a computer macro menu that’s large enough to rival the Cheesecake Factory’s is nice idea, but only if you can keep track of and find serious utility in that many shortcuts.
Whatever the controls are, they should ideally work across platforms and objects — i.e. the same device could be used to drive a wheelchair to a desk, and then switch over to interface with the computer sitting on the desk.
Old Stuff: Sip and Puff
Of course, too little control is better than none at all. I would be remiss if I didn’t acknowledge what is probably the oldest oral-based assistive device out there: the sip-and-puff interface. Using a single straw, users are able to control different inputs by sipping, or inhaling through the straw, and other inputs by puffing, or exhaling through the straw. Sip-and-puff has been around for many decades and is still used to drive everything from wheelchairs to sailboats to touch screens.
Depending on the device, sip-and-puff doesn’t have to be complicated or even wired. TubusOne is a low-tech implementation that acts like an extendable mouth finger. Huff a little puff of air, and a slim digit with a conductive fabric tip pops out to help you navigate any touch screen, anywhere.
The biggest downsides to commercial sip-and-puff devices is that they don’t offer much fine control, and they still cost hundreds of dollars after all this time. Aside from the expense, the interface is anything but discreet. Unless it looks super cool, we’d venture to guess that a fair number of able-bodied people who would use an oral input device would prefer it be hidden, if at all possible.
Is That a Mouse in Your Mouth?
You may remember Pallette from a few years back, because this Bluetooth mouth mouse won $1000 in the assistive technologies design challenge of the 2016 Hackaday Prize.
Pallette is built on a custom-molded retainer that sits against the upper teeth like a mouth guard. An infrared sensor reads the tongue’s position to move a cursor accordingly, and a microphone translates tongue taps into clicks.
I love that Pallette is so vehemently open source. The creators built a website that has detailed instructions for building your own for around $200.
Pallette ticks many boxen for me. It’s discreet — the thing fits entirely within your mouth, with no wires and no headset. The only downside is that it seems like a lot to pack inside of a small mouth.
Though it’s not open-source, here’s a controller that minimizes the mouth component BOM down to one item: a small magnetic tongue stud. The user wears a headset full of sensors that detect tongue movements through changes in the magnetic field of the stud. The big benefit here is that since there is so little hardware inside the mouth, it doesn’t get in the way of speaking. The system can also be switched off with a long press of the tongue, and left in place while eating. Here’s a video of it in action.
Safe, Sanitary, and Scalable
If these things are going to catch on in some form, they need to be safe, sanitary, and scalable. If ‘wireless’ means there’s a battery in your mouth, you want to be careful. Pallette uses a silver oxide battery that has many advantages over lithium-ion, not the least of which is a drastically lower flammability rating.
If we imagine these assistive inputs in the workplace, there are more things to consider. With Pallette, every worker would need to have a custom-fit mouth guard. Thermoplastic isn’t that expensive, but it would add up.
If the electronics live in a standardized enclosure, then you have a one-size-fits-all solution that’s scalable for mass production. The Bit, another Hackaday.io project, satisfies that condition nicely. All the electronic goodies are cleverly contained inside an adult pacifier. At first glance I thought it was molded acrylic, but this is an off-the-shelf pacifier. [oneohm] used a pair of hacked snap ring pliers to spread the opening wide, and then shoved the board in there.
The beta prototype runs on an ATmega32u4 breakout and uses an optical tracking sensor to keep tabs on the tongue. There’s also a time of flight sensor, because why not track the tongue in the z-direction, too? There’s a pair of conductive rubber force sensors to translate gentle tongue pressure into up and down scroll commands, and they register full-on chomps as clicks. As you can see in the picture, [oneohm] played around with triangulating the force sensors for joystick control.
Ease of Use
All of these devices have their merits, but there’s one thing they haven’t considered: the starting and stopping conditions. How are you gonna get this device in your mouth by yourself if you’re paralyzed from the neck down? The project that inspired this installment, Assistive Tongue-Operated Mouse (ATOM), handles that problem elegantly. When not in use, the retainer attaches magnetically to an acrylic stand, but it could just as easily link to something similar mounted on a wheelchair.
ATOM can be made for under $100, minus the base. The brain is an Arduino Pro Micro, and the main control is the pointing stick from a laptop keyboard that your tongue uses as a D-pad. I like that ATOM gets your teeth involved, too — there’s a pair of momentaries you can munch with your molars to register left and right clicks. [Ben Krasnow] made one kind of like it a few years back.
Something to Chew On
No matter who the user is, mouth inputs should be safe to use. Ideally, they should be cheap to make and easy to use from start to finish. In a perfect world, they would be completely hidden from view and small enough to allow speaking or even eating. That’s clearly a tall order, but it’s one worth working on. If you’ve got any favorite mouth inputs, let us know!
(Full disclosure: I’ve never used a tongue- or mouth-controlled input, but I totally would. And after giving them so much consideration, I may have to take a shot at making my own. It would be another way to do more while keeping my hands on the keyboard, and I’m all about that.)
I was just thinking about tongue input devices the other day. My goal would be a device that doesn’t prohibit or alter speech (to those capable of it). Most of the in-the-mouth devices would completely obstruct speech, but the tongue stud based project from Georgia Tech is really interesting!
And here I thought tongue studs were just for freaks! But really it is an interesting interface, minimally invasive and removable. I just wonder if the stud could be replaced with a small magnet held in place with some sort of strong but temporary biocompatible glue for people who don’t want to get a lingual lancing.
Depends on your definition of ‘freak’. Given that no one is average in all measures we’re all freaks.
I think glue would be a bad idea for people who have paralysis, they often struggle with swallowing so the risk of choking if it came loose would be pretty high.
The thing that gives me the heebie geebies is the very bad internal damage that can happen should magnets be swallowed, whether they come loose from a stud or unglued or get chewed out of rubber or whatever.
I would hope the magnet is not very strong, and on its own (with no other magnets or ferrous materials) it’s relatively innocuous.
“Depends on your definition of ‘freak’.” Why anyone who’s not just like me, of course! :)
I see your point with people with paralysis, I wonder if a tongue “paint” could be made magnetic?
Wireless shouldn’t be a primary requirement for assisting people that are, mishaply, by definition, bonded to a wheelchair or a bed. Non-intrusiveness should be a primary requirement.
Have a picture in my head of something that would be, while not the ideal interface, something that is adaptable to technology that is in place and readily available. It should also be fairly simple and cheapish. First we take something like a mouthguard, or snorestopper type dental device, a piece that sits firmly between the front teeth, bitten down on, but also maintains a distance between the teeth. Then we put a gimbal in it, or ball joint kind of deally, and on one end we have a cup/bucket kind of thing that the tongue can be pushed into, and on the other end, a tube, into which a commodity smartphone/tablet stylus can be inserted.
This is as you will recognise, a refinement of the simple rubber ended stick, that people with restricted movement have had to manage with for decades. However, putting in the balljoint/gimbal firmly held between the teeth, will allow more precision control with the tongue, using it as a reversed joystick as it were, to allow fine control without head movement. Thus it should be able to for one to operate a tablet, or touch screen ATM or payment terminal, almost normally.
A refinement would be, either to have a spring loaded central nipple (no sniggering) in the tongue cup, so that with extra pressure the tongue can push the stylus forward for the purposes of clicking, or pushing buttons, without having to move the head. This could be assisted electrically or electronically with a solenoid, motor or other actuator and a push button or pressure pad operating it.
A bit of thought with the design and I think you could have interchangeable implements on it, like a pickle grabber type arm for manipulation of small objects.
I was watching the Chiefs football game, in the bitter cold a player was swiping a tablet with his nose because he didn’t want to take off his gloves.
Are you sure he wasn’t just “wiping” his nose?
B^)
I often do that :o
Particularly when trying to wash up and the phone rings
The actor Tom Holland spoke of having to do that while he was in his lycra onesie while filming one of his ArachnidBoy movies.
I could do without the smears of face grease but it works. I need to just get a stylus-like thing for those times…
I wonder if a tiny camera could have a wide enough field of vision to do in-mouth tongue gesture tracking? An external camera doing the same might be functional, if not silly looking.
Smile, they are on Candid Camera
In my last job, I worked on an in-mouth wireless sensor product that included a Li-ion battery. The circuit was even placed in a premold and an overmold was shot over the back side of the circuit board with the battery encapsulated inside. It worked very well – the battery temperature never rose above 85C, which was well within the acceptable range for Li-poly. I always thought we might have to put a shield over the battery to protect it from compression in the mold, but it never ended up being needed. Also, regarding safety, we had hundreds of products in the field and never had an issue with a user being injured by the battery. Some people literally chewed the battery out of the product, and it still didn’t do anything. There just wasn’t all that much energy in one of the small batteries we used. You could short them without a protection circuit or even cut them open and nothing would happen.
Achieving acceptable durability was tough, because users would bend and chew it, but eventually we developed enough strategies that we mostly solved it. Then the company started its slow motion death, and I don’t work there anymore. I wish I knew where to apply some of the esoteric skills and techniques I learned there. Maybe the opportunity will present itself.
What about a device like a Hunter S Thompsonesque cigarette holder, where a slim battery is external the mouth, and a small “nub” projects into the mouth for manipulation with the tongue ? Hmm– gonna have to get on that.
At my institute we developed on a non invasive device to use the tongue as an input device. For that we measured the pressure in the ear channel which changes with every jaw or tongue movement. We were able to play a modified, automated version of the game “Looping Louie” in realtime or control a robot arm. one advantage of the solution is that you do not have to put anything in your mouth.
(If somebody is interested, just google for
“A human-machine interface based on tongue and jaw movements,” or “A jaw based human-machine interface with machine learning,”)
very neat.
When I used to work on asistive technology the oral devices were somewhat icky. Saliva can travel a long way…. it dries out leaves an unpleasant residue.
I’m surprised there’s no mention of LipSync here.
https://www.makersmakingchange.com/project/lipsync/
Most of the in-the-mouth devices would completely obstruct speech, but the tongue stud based project from Georgia Tech is really interesting!
In the novel Existence by David Brin, most people who can afford them have augmented reality implants (or glasses, for the old-fashioned), and a common way of giving input to them is tapping specific parts of specific teeth with other teeth, which is sensed by tooth implants. Generally, each such input seems to be mapped to a specific command rather than used for navigation (because, if something is “onscreen”, you can just look at it the right way to activate it). I tried this tooth-tapping (without having any hardware to detect it), and I found it reasonably easy; it would probably become second nature with practice, just like typing or using a mouse. This isn’t tongue input, but it’s related, and tooth input was already mentioned in one of the examples in the article.