Scratch input

Scratch input allows us to use solid surfaces as an input devices by capturing the sounds they produce. Using a stethoscope and a high pass filter, they capture the unique sounds of specific gestures. Custom software then translates this to actions for applications. The video shows some really cool stuff, like turning an entire wall into an input device. It goes around corners and past doorways. They even talk about potential using your clothes to capture input.

[via Procrastineering]

Comments

  1. icefox says:

    This is very interesting. What about if 2 of these could be used to triangulate the position and make a table into a touch pad for a computer? It could be integrated to laptops with one on each side in the base.

  2. kyle says:

    could it possible be used to pick up shapes and recognise them as letter. so you could use a wall as a white board

  3. Tuckie says:

    Doesn’t “TRI”angulation generally require 3 sensors ;) (but I really like that idea icefox, it has me thinking…)

  4. _matt says:

    triangulation could allow actual writing, since a w and an m probably sound the same written.

    but even then, you’d need enough movement to make enough sound to get translated into mouse movement. I think it should be left for just gestures.

    ooh, you could get some multi-touch in there. if the two or 3 mics picked up a sound getting louder on both ends, it could be the user swiping two fingers away from each other, signaling to zoom in…

  5. Tim says:

    Seconded… good thinking icefox

    with a triangulated system you would have to calibrate it… unlike the singular mic, but with the right code you could allow for more gestures (eg being able to decipher a V move from a > move…which to a single mic would probably appear the same)

    … another project now added to an already large list…DOH!

  6. Tuckie says:

    After more thinking… if you could get a high enough sample rate, you could even base it on audio delay, as it would take more time for the sound to reach the furthest of the three mics. This would provide another set of data points, and might allow for better resolution.

  7. Edward Nardella says:

    Triangulation only requires three sensors in 3D space, in 2D you can use only 2 sensors.

  8. Tuckie says:

    After more thought… if you could get a high enough sample rate, you could even base it on audio delay, as it would take more time for the sound to reach the furthest of the three mics. This would provide another set of data points, and might allow for a better resolution.

  9. Tuckie says:

    (sorry for the double post, my connection bugged out on me, and didn’t show that it had gone through)

  10. thomas says:

    @tuckie: no, triangulation means that you have 2 known points use those to create a triangle with the third unknown position using data gathered at the known positions.

    Really, though, I’m not sure this would work since each of the “sensors” is non-directional. You would probably need to use multiple mics with different phase orientations at each position in order to be able to determine an actual angle to the point at which the gesture was made. I’m not very clear on the physics here, but some experimentation would probably show whether this has potential.

    A related (but somewhat simpler) project created by my friend and fellow member of DorkbotPDX, Collin Oldham, is his Clapboard: http://homepage.mac.com/coldham/klang/clapboard.html

  11. Nelson says:

    Would it be possible to make a table capable of outputting delta X and delta Y data of any given scratch simply by having two sensors at the far top left and far bottom left and measuring the change in amplitude of each. Then using those changes in amplitude + some form of calibration to give something similar to the touch-pad on a laptop?
    Of course that assumes that the strength of your scratch is fairly constant, but only for the duration of each individual scratch.

    I can’t tell if this is a stupid idea. It’s too early.

  12. Nelson says:

    (Given that x and y are arbitrary units of distance)

  13. Ryan says:

    Trilateration is the term i think you guys are looking for, but that’s for obtaining a position in 3d space using intersecting spheres. For a 2d surface such as a flat wall, only two sensors would be required. To determine distance from the sensors, a time of arrival (phase sensitive detectors), or amplitude method could be used

    given sufficient training a machine-learning algorithm would be able to determine the location of a scratch.

  14. macegr says:

    what’s going on with the comment system, i posted a while ago and it never showed up…i tried to post the same thing and wordpress showed a duplicate comment. ????

  15. icefox says:

    Since most of you seem to be debating my comment about triangulation, let me just throw what I was thinking about when I was thinking of. I do actually agree that 3 may work somewhat better.

    Do you know how the police (and other companies) can track cell phone locations? A call tower, or antennae, can determine the range from itself to a point, the phone. This forms a circle of where it could possibly be. A second antennae can form another “range circle” and narrow it down to 2 points in space. A third can be used to determine the exact point the cell phone, rather than 2 possible ones. My old math teacher actually was in the FBI.

    Now apply that logic to scratch input, but with much shorter ranges. It could be done with 2, but 3 would be more accurate. 1 however, just seems overcomplicated to me.

  16. macegr says:

    It looks like my earlier comment isn’t going to come back. Anyway, I was pointing out this technology is used in Elo Touch’s acoustic pulse recognition touch panels. I worked with them at elo for a small contract job once.

    Basically it’s just two microphones on a sheet of glass. A machine taps the glass in a matrix and the stereo sound profile is recorded, maybe classified into an FFT fingerprint. Then the incoming audio is compared against a database of stored profiles. A normal sound card is enough to get good resolution, though of course Elo’s hardware is optimizing the signal and ADC so you don’t actually have to plug a line in to your computer. All the real work is in the software driver; optimizing the search for known patterns, rejecting noise, making it all fast enough for instant touch response.

    You can download whitepapers with more in-depth discussion of the technology on Elo’s website.

  17. Satiagraha says:

    I have to admit, this is a fantastically detailed report for what seems like a weekend experiment. Kudos to the authors, procrastineering, for an innovative little project.

  18. I feel like you could make a surface that had micromachined groove patterns that were different from different directions, so that each direction (up, down, left, right) produced a unique sound when something scraped across it. Then you could use dead reconing to get an approximate shape of each individual movement, which would make gesture recognition better I wonder if a fresnel lens would be a good place to start? They already have directional grooves in them.
    -Taylor

  19. morden says:

    Look back at the recent post regarding the capacitive resistance touchpad using the metallic cloth and 4 points of contact. You could set up 4 scratch sensors and determine x/y location simply by assuming a matrix size they represent and using the strength of the scratch to determine to which side they are closest too.

    70% right 30% left = x+20 on a 100/100 grid.

  20. john says:

    yay im putting microphones in every wall

  21. polymath says:

    if they get this to work in clothing I’ll really be impressed. if nothing else it will give men the world over an excuse to scratch themselves in public.

  22. speps says:

    I already saw that applied here :

    http://www.journaldunet.com/solutions/systemes-reseaux/reportage/08/0208-surfaces-intelligentes-reversys.shtml

    It’s in French sorry but the video should be enough.

  23. Adam Ziegler says:

    piezo

  24. bountyx says:

    very innovative, i can think of a million different uses for this. How about using a unique gesture to on your car window to open the door?

  25. EllisGL says:

    4 sensors would be needed to figure out position. Also a calibration would have to be needed, because each time you do something, even if you tapped the same place, the volume isn’t going to be the same. Maybe tap each corner 3 times. Light, medium and hard.. Just an idea..

  26. Mike says:

    I agree. I think for quality of sample and accuracy 4 sensors could be placed on the 4 corners of the surface you are using. I think that as few as 3 could be successfully used, but for a technology like that to be implemented successfully you would need to have >99.999% recognition, or higher. Even 1 false input in 10,000 would cause problems.

    That movie in French was good, but I think that you can see there are inherent problems with such things as on surface keyboards, because it would be nearly impossible for the technology to recognize when a button is held, making many hot keys and short cuts unusable.

  27. srilyk says:

    @mike – you only need a “hotkey”, i.e. two short taps, followed by your command.

    @ellisgl – volume may not be the same, but the sound should always have a relative speed difference among the 4 inputs. Volume “don’t enter into it”, the same thing is in this case – it’s not the volume but the waveform that’s measured/analyzed.

    This is a really cool idea… although I must confess I’m thinking along the lines of a security device – could you imagine miking floors, in combination with camera/thermal/etc? You could probably fairly “easily” get the exact location of a person… combine that with a fast turret/airsoft gun combination… w00t! (or real gun, if that happened to be your thing.)

  28. Ben says:

    Fantastic video and project. Well done to all those involved.

    Would love to try this out myself. Any idea if the software is available to try?

    Thanks to Johnny Lee and his web-site for pointing me in the direction of this project.

    Anyone with any details about where to find resources/software for this please post them.

    Thanks!

  29. cassetti says:

    Few things, first off every blog repeating this is saying the same thing “High Pass Filter”

    You can easily make this yourself. A First order (6dB per Octave) High Pass filter is simply a Capacitor! Look online to determine how many farads to get the desired filter you wish.

    I will be installing one as soon as i can find some sort of software for linux (ubuntu)

    I have a complete home audio system connected to one computer running Linux. It would be WONDERFUL to have sound control when we can’t find the Logitech Harmony or the RF keyboard/mouse combo.

    Any links to any sort of software?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 94,628 other followers