Scratch Input

[youtube=http://www.youtube.com/watch?v=2E8vsQB4pug]

Scratch input allows us to use solid surfaces as an input devices by capturing the sounds they produce. Using a stethoscope and a high pass filter, they capture the unique sounds of specific gestures. Custom software then translates this to actions for applications. The video shows some really cool stuff, like turning an entire wall into an input device. It goes around corners and past doorways. They even talk about potential using your clothes to capture input.

[via Procrastineering]

29 thoughts on “Scratch Input

  1. This is very interesting. What about if 2 of these could be used to triangulate the position and make a table into a touch pad for a computer? It could be integrated to laptops with one on each side in the base.

  2. triangulation could allow actual writing, since a w and an m probably sound the same written.

    but even then, you’d need enough movement to make enough sound to get translated into mouse movement. I think it should be left for just gestures.

    ooh, you could get some multi-touch in there. if the two or 3 mics picked up a sound getting louder on both ends, it could be the user swiping two fingers away from each other, signaling to zoom in…

  3. Seconded… good thinking icefox

    with a triangulated system you would have to calibrate it… unlike the singular mic, but with the right code you could allow for more gestures (eg being able to decipher a V move from a > move…which to a single mic would probably appear the same)

    … another project now added to an already large list…DOH!

  4. After more thinking… if you could get a high enough sample rate, you could even base it on audio delay, as it would take more time for the sound to reach the furthest of the three mics. This would provide another set of data points, and might allow for better resolution.

  5. After more thought… if you could get a high enough sample rate, you could even base it on audio delay, as it would take more time for the sound to reach the furthest of the three mics. This would provide another set of data points, and might allow for a better resolution.

  6. @tuckie: no, triangulation means that you have 2 known points use those to create a triangle with the third unknown position using data gathered at the known positions.

    Really, though, I’m not sure this would work since each of the “sensors” is non-directional. You would probably need to use multiple mics with different phase orientations at each position in order to be able to determine an actual angle to the point at which the gesture was made. I’m not very clear on the physics here, but some experimentation would probably show whether this has potential.

    A related (but somewhat simpler) project created by my friend and fellow member of DorkbotPDX, Collin Oldham, is his Clapboard: http://homepage.mac.com/coldham/klang/clapboard.html

  7. Would it be possible to make a table capable of outputting delta X and delta Y data of any given scratch simply by having two sensors at the far top left and far bottom left and measuring the change in amplitude of each. Then using those changes in amplitude + some form of calibration to give something similar to the touch-pad on a laptop?
    Of course that assumes that the strength of your scratch is fairly constant, but only for the duration of each individual scratch.

    I can’t tell if this is a stupid idea. It’s too early.

  8. Trilateration is the term i think you guys are looking for, but that’s for obtaining a position in 3d space using intersecting spheres. For a 2d surface such as a flat wall, only two sensors would be required. To determine distance from the sensors, a time of arrival (phase sensitive detectors), or amplitude method could be used

    given sufficient training a machine-learning algorithm would be able to determine the location of a scratch.

  9. Since most of you seem to be debating my comment about triangulation, let me just throw what I was thinking about when I was thinking of. I do actually agree that 3 may work somewhat better.

    Do you know how the police (and other companies) can track cell phone locations? A call tower, or antennae, can determine the range from itself to a point, the phone. This forms a circle of where it could possibly be. A second antennae can form another “range circle” and narrow it down to 2 points in space. A third can be used to determine the exact point the cell phone, rather than 2 possible ones. My old math teacher actually was in the FBI.

    Now apply that logic to scratch input, but with much shorter ranges. It could be done with 2, but 3 would be more accurate. 1 however, just seems overcomplicated to me.

  10. It looks like my earlier comment isn’t going to come back. Anyway, I was pointing out this technology is used in Elo Touch’s acoustic pulse recognition touch panels. I worked with them at elo for a small contract job once.

    Basically it’s just two microphones on a sheet of glass. A machine taps the glass in a matrix and the stereo sound profile is recorded, maybe classified into an FFT fingerprint. Then the incoming audio is compared against a database of stored profiles. A normal sound card is enough to get good resolution, though of course Elo’s hardware is optimizing the signal and ADC so you don’t actually have to plug a line in to your computer. All the real work is in the software driver; optimizing the search for known patterns, rejecting noise, making it all fast enough for instant touch response.

    You can download whitepapers with more in-depth discussion of the technology on Elo’s website.

  11. I feel like you could make a surface that had micromachined groove patterns that were different from different directions, so that each direction (up, down, left, right) produced a unique sound when something scraped across it. Then you could use dead reconing to get an approximate shape of each individual movement, which would make gesture recognition better I wonder if a fresnel lens would be a good place to start? They already have directional grooves in them.
    -Taylor

  12. Look back at the recent post regarding the capacitive resistance touchpad using the metallic cloth and 4 points of contact. You could set up 4 scratch sensors and determine x/y location simply by assuming a matrix size they represent and using the strength of the scratch to determine to which side they are closest too.

    70% right 30% left = x+20 on a 100/100 grid.

  13. 4 sensors would be needed to figure out position. Also a calibration would have to be needed, because each time you do something, even if you tapped the same place, the volume isn’t going to be the same. Maybe tap each corner 3 times. Light, medium and hard.. Just an idea..

  14. I agree. I think for quality of sample and accuracy 4 sensors could be placed on the 4 corners of the surface you are using. I think that as few as 3 could be successfully used, but for a technology like that to be implemented successfully you would need to have >99.999% recognition, or higher. Even 1 false input in 10,000 would cause problems.

    That movie in French was good, but I think that you can see there are inherent problems with such things as on surface keyboards, because it would be nearly impossible for the technology to recognize when a button is held, making many hot keys and short cuts unusable.

  15. @mike – you only need a “hotkey”, i.e. two short taps, followed by your command.

    @ellisgl – volume may not be the same, but the sound should always have a relative speed difference among the 4 inputs. Volume “don’t enter into it”, the same thing is in this case – it’s not the volume but the waveform that’s measured/analyzed.

    This is a really cool idea… although I must confess I’m thinking along the lines of a security device – could you imagine miking floors, in combination with camera/thermal/etc? You could probably fairly “easily” get the exact location of a person… combine that with a fast turret/airsoft gun combination… w00t! (or real gun, if that happened to be your thing.)

  16. Fantastic video and project. Well done to all those involved.

    Would love to try this out myself. Any idea if the software is available to try?

    Thanks to Johnny Lee and his web-site for pointing me in the direction of this project.

    Anyone with any details about where to find resources/software for this please post them.

    Thanks!

  17. Few things, first off every blog repeating this is saying the same thing “High Pass Filter”

    You can easily make this yourself. A First order (6dB per Octave) High Pass filter is simply a Capacitor! Look online to determine how many farads to get the desired filter you wish.

    I will be installing one as soon as i can find some sort of software for linux (ubuntu)

    I have a complete home audio system connected to one computer running Linux. It would be WONDERFUL to have sound control when we can’t find the Logitech Harmony or the RF keyboard/mouse combo.

    Any links to any sort of software?

Leave a Reply to NelsonCancel reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.