From sign language to spoken language

As part of a senior design project for a biomedical engineering class [Kendall Lowrey] worked in a team to develop a device that translates American Sign Language into spoken English. Wanting to eclipse glove-based devices that came before them, the team set out to move away from strictly spelling words, to combining sign with common gesture. The project is based around an Arduino Mega and is limited to the alphabet and about ten words because of the initial programming space restraints. When the five flex sensors and three accelerometer values register an at-rest state for two seconds the device takes a reading and looks up the most likely word or letter in a table. It then outputs that to a voicebox shield to translate the words or letters into phonetic sounds.

Comments

  1. fluidic says:

    Now that’s how you use an Arduino.

    Can it be ARM time now? Triple bonus score given the application.

  2. grenadier says:

    Kick ass!

  3. Erik Johnson says:

    Now if/when they remake “Congo”, the gorilla won’t have a huge backpack housing the electronics

  4. zool says:

    no video?
    only the alphabet and ten words?
    not too useful

    if you’re going to have this huge thing on, why not just use something that does text to speech in a handheld device

    i bet she got an A though
    it’s a pretty good use of all those parts

  5. Eirinn says:

    @zool do you know what a prototype is? :) although i’ll admit, video please.

  6. grenadier says:

    @zool

    Not too useful? Give this to koko the gorilla!

  7. klowrey says:

    @zool @Eirinn we did have a video for the class, but it encompassed the engineering design process rather than the device’s function–biomed class, not ece. and it was embarrassing.

    best i can offer for now is http://www.flickr.com/photos/45901891@N07/5046901595/in/photostream/ at least until i can get v2 up

  8. jzoe says:

    @zool- why not just text to speech? Because ASL is not the same thing as spoken or written english. The grammar and sentence structure is not the same- it is its own independent language. In the same way that Babelfish could take French or German text and make an approximation in English, this is a first step toward such a thing for ASL.

  9. Whatnot says:

    The alphabet in sign is done with the hand alone AFAIK, but words are often done with the whole arm and both hands and arms together, so obviously the easiest initial attempts would be to do the alphabet.
    And you can use the accelerometers to track the arms but not only does it get more complex but also not everybody moves the same, and you have to discard the movements not part of the message between the words.
    It’s truly an interesting challenge that in the end might also give you something useful.

  10. NatureTM says:

    I like projects with a conscience, very nice.

  11. tjb says:

    ASL also uses facial expression and situation awareness to communicate. I think these kinds of things are cool.

    I can attest that a knowing even a few signs can make communication much better. I did an event this weekend with a deaf group. I only know yes,no, and a couple of other signs. We got the job done in short order. After all was said and done my wife shows up (who is fluent in ASL) and then we could really communicate.

  12. Hirudinea says:

    Could this be adapted to using signing over then internet, for sign practice, etc.

  13. Spork says:

    Having this as a starting platform someone could develop some very useful hardware.

    I’ve just been learning ASL the last few months and it comes very naturally. I think a device like this could be useful for learning ASL as well as allowing ASL speakers to communicate with non-ASL speakers. Though the device has limitations, it is a big step in bridging the signed/spoken-language gap.

  14. rasz says:

    “Now that’s how you use an Arduino.”

    Yes, you use it so your project has to be limited to 10 words …

  15. fluidic says:

    @rasz It’s called a prototype for a reason, aye?

  16. crzy says:

    Pretty cool, man. Good luck building upon it!

    I have to say that one of the most rewarding projects I did a few years ago was creating a campus-wide visual communications system for a Deaf school…

  17. blue carbuncle says:

    huh huh uh huh huh “Ya could use it on your thingy”
    “Yeah Yeah” Heheheheheh

  18. I’m just finishing my Master Thesis project about translating hand gestures into sound using computer vision:

    http://code.google.com/p/sonic-gesture/

    maybe you think it is interesting

  19. gordon says:

    @rasz
    “Now that’s how you use an Arduino.”

    Thats right you use a cheap and simple system to work out some of the wrinkles and then you expand the project as you work out what you need.
    Then you dont waste your money or time on an all dancing and singing over specced system that you end up discarding because its not suitable after all.

  20. pff says:

    Its a good project, but its not really the best solution. deafs are all ready born with the disability, why should they have to buy this thing just to interact with the world. teach sign language in schools instead of making deafs even more of a curiosity. Stephen hawking gets enough jollies poked at him for his voice box, i cant imagine this would be any better than that.

  21. Kaity says:

    In theory this is cool, I do love the movie Congo. But ASL is so complex you would need the machine to have facial sensors, pretty much a whole body suit to understand grammar. Plus it would need to be so sensitive to understand classifiers and context. I don’t think it’s very logical but it still looks cool and I’m sure was fun to work on.

  22. xtreker15 says:

    “Bad gorilla…. Bad gorilla” please record a video for youtube. thanks

  23. J. Swift says:

    I came here to note that the deaf will probably start complaining about it, and lo and behold I see that pff has already beaten me to the punch by bitching about it.

    There is no reason the entire nation should be forced to learn that curiously terrible language called ASL. Rather, we should focus on creating prosthesis devices such as this and “deaf culture” be damned.

  24. pff says:

    There is far more reason for people to learn sign language than any other language. Think about it logically, if we could reduce the number of languages we could get it down to 2, a spoken language for talkers, and a sign language for deafs.
    if we should be getting rid of any languages, it should be one of the rubbish languages, like that one African one where they click *click click bop*.
    sign language is necessary, other languages are not.

  25. ???? says:

    Where can you find those flex sensors?? I tried ebaying and amazoning with no results!

  26. bluewraith says:

    to ????:
    http://www.sparkfun.com has a few.

  27. rasz says:

    btw those flex sensors are so frickin simple in design its almost amazing. I didnt even had to read the patent (noticed “patented technology blablabla” all over sparkfun shop). Looks like flexible resistive band with glued metal pads, when you bend it metal pads touch less and less and make the band more resistive, when its straight metal pads conduct most of the current and you only get resistance that is between them.

  28. J. Swift says:

    @pff: Ok, you try telling all the deafies of the world “Ok, we’re all going to ASL, so you foreign people had better drop whatever sign language you know now and learn ASL”. There are plenty of deaf people who refuse to teach their children lip-reading because DEAF CULTURE. Can you really just magic away all the other sign languages? If you manage that, for my 2nd wish I want every country to speak English, and for my 3rd wish I want a toilet seat made of solid gold.

    Sign language is only necessary until we figure out how to fix deafness. I’d rather fix deafness and have my kids learn Spanish, or Chinese, or German, or any number of languages that have actual grammar and tense and don’t require twisting your face up like a drunken gorilla.

  29. PlastBox says:

    A for effort, but the idea itself is a complete fail. As someone who knows sign language quite well, I can attest that signing isn’t all that much about the signs at all. You have to speak, gesture and sign at the same time. Sooo many signs have a plethora of different meanings, it’d be completely impossible to use a glove like this to translate anything other than the alphabet (where each character has it’s own, distinct sign).

    Also, most deaf people can in fact make themselves well understood using their voice (most get special training for this) or by writing on a cell phone etc. The “problem” is communicating the other way around, so a device that translated voice into signs would be far more useful, not to mention possible to create.

  30. ???? says:

    You think I could disassemble a Nintendo power glove to get at the 5 flex sensors?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 93,810 other followers