From sign language to spoken language

As part of a senior design project for a biomedical engineering class [Kendall Lowrey] worked in a team to develop a device that translates American Sign Language into spoken English. Wanting to eclipse glove-based devices that came before them, the team set out to move away from strictly spelling words, to combining sign with common gesture. The project is based around an Arduino Mega and is limited to the alphabet and about ten words because of the initial programming space restraints. When the five flex sensors and three accelerometer values register an at-rest state for two seconds the device takes a reading and looks up the most likely word or letter in a table. It then outputs that to a voicebox shield to translate the words or letters into phonetic sounds.

30 thoughts on “From sign language to spoken language

  1. no video?
    only the alphabet and ten words?
    not too useful

    if you’re going to have this huge thing on, why not just use something that does text to speech in a handheld device

    i bet she got an A though
    it’s a pretty good use of all those parts

  2. @zool- why not just text to speech? Because ASL is not the same thing as spoken or written english. The grammar and sentence structure is not the same- it is its own independent language. In the same way that Babelfish could take French or German text and make an approximation in English, this is a first step toward such a thing for ASL.

  3. The alphabet in sign is done with the hand alone AFAIK, but words are often done with the whole arm and both hands and arms together, so obviously the easiest initial attempts would be to do the alphabet.
    And you can use the accelerometers to track the arms but not only does it get more complex but also not everybody moves the same, and you have to discard the movements not part of the message between the words.
    It’s truly an interesting challenge that in the end might also give you something useful.

  4. ASL also uses facial expression and situation awareness to communicate. I think these kinds of things are cool.

    I can attest that a knowing even a few signs can make communication much better. I did an event this weekend with a deaf group. I only know yes,no, and a couple of other signs. We got the job done in short order. After all was said and done my wife shows up (who is fluent in ASL) and then we could really communicate.

  5. Having this as a starting platform someone could develop some very useful hardware.

    I’ve just been learning ASL the last few months and it comes very naturally. I think a device like this could be useful for learning ASL as well as allowing ASL speakers to communicate with non-ASL speakers. Though the device has limitations, it is a big step in bridging the signed/spoken-language gap.

  6. Pretty cool, man. Good luck building upon it!

    I have to say that one of the most rewarding projects I did a few years ago was creating a campus-wide visual communications system for a Deaf school…

  7. @rasz
    “Now that’s how you use an Arduino.”

    Thats right you use a cheap and simple system to work out some of the wrinkles and then you expand the project as you work out what you need.
    Then you dont waste your money or time on an all dancing and singing over specced system that you end up discarding because its not suitable after all.

  8. Its a good project, but its not really the best solution. deafs are all ready born with the disability, why should they have to buy this thing just to interact with the world. teach sign language in schools instead of making deafs even more of a curiosity. Stephen hawking gets enough jollies poked at him for his voice box, i cant imagine this would be any better than that.

  9. In theory this is cool, I do love the movie Congo. But ASL is so complex you would need the machine to have facial sensors, pretty much a whole body suit to understand grammar. Plus it would need to be so sensitive to understand classifiers and context. I don’t think it’s very logical but it still looks cool and I’m sure was fun to work on.

  10. I came here to note that the deaf will probably start complaining about it, and lo and behold I see that pff has already beaten me to the punch by bitching about it.

    There is no reason the entire nation should be forced to learn that curiously terrible language called ASL. Rather, we should focus on creating prosthesis devices such as this and “deaf culture” be damned.

  11. There is far more reason for people to learn sign language than any other language. Think about it logically, if we could reduce the number of languages we could get it down to 2, a spoken language for talkers, and a sign language for deafs.
    if we should be getting rid of any languages, it should be one of the rubbish languages, like that one African one where they click *click click bop*.
    sign language is necessary, other languages are not.

  12. btw those flex sensors are so frickin simple in design its almost amazing. I didnt even had to read the patent (noticed “patented technology blablabla” all over sparkfun shop). Looks like flexible resistive band with glued metal pads, when you bend it metal pads touch less and less and make the band more resistive, when its straight metal pads conduct most of the current and you only get resistance that is between them.

  13. @pff: Ok, you try telling all the deafies of the world “Ok, we’re all going to ASL, so you foreign people had better drop whatever sign language you know now and learn ASL”. There are plenty of deaf people who refuse to teach their children lip-reading because DEAF CULTURE. Can you really just magic away all the other sign languages? If you manage that, for my 2nd wish I want every country to speak English, and for my 3rd wish I want a toilet seat made of solid gold.

    Sign language is only necessary until we figure out how to fix deafness. I’d rather fix deafness and have my kids learn Spanish, or Chinese, or German, or any number of languages that have actual grammar and tense and don’t require twisting your face up like a drunken gorilla.

  14. A for effort, but the idea itself is a complete fail. As someone who knows sign language quite well, I can attest that signing isn’t all that much about the signs at all. You have to speak, gesture and sign at the same time. Sooo many signs have a plethora of different meanings, it’d be completely impossible to use a glove like this to translate anything other than the alphabet (where each character has it’s own, distinct sign).

    Also, most deaf people can in fact make themselves well understood using their voice (most get special training for this) or by writing on a cell phone etc. The “problem” is communicating the other way around, so a device that translated voice into signs would be far more useful, not to mention possible to create.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s