A cool motion device called Copy Cat promises to help the deaf be heard and people who do not know American Sign Language (ASL) to be able to interpret the motions made by a person signing.

ASL is the standard for communication among deaf and mute persons in the United States. Over many years, ASL has been refined into a fairly standard set of motions made by the speaker.

Copy Cat, and its related game called Castle Quest, can read and interpret those signs, explains Thad Starner, director of the Contextual Computing Group at Georgia Institute of Technology’s College of Computing and School of Interactive Computing.

Unlike hearing children of English-speaking parents or deaf children of signing parents, deaf children often lack access to language at home which is necessary in developing linguistic skills during the critical period of language development. In many cases, these children’s only exposure to language is from signing at school. Castle Quest and CopyCat are games with a serious intent. They use a sign language recognition system to augment early classroom teaching for developing ASL skills in young deaf children. Castle Quest is the iterated version of CopyCat, Starner explains.

Brightly colored gloves, equipped with accelerometers, are worn by the child. These typically are red gloves. Each finger is colored near the fingertip: green for the thumb, then blue, yellow, purple and orange.

“CopyCat is designed both as a platform to collect gesture data for our ASL recognition system and as a practical application which helps deaf children acquire language skills while they play the game,” Starner says. CopyCat is based on a video camera and wrist-mounted accelerometers. The wrist-mounted accelerometers are the primary sensors. In CopyCat, the user and the character of the game, Iris the Cat, communicate with the player via ASL. With the help of ASL linguists and educators, the game is designed with a limited, age-appropriate phrase set. For example, the child will sign to Iris, “you go play balloon” in ASL. If the child signs poorly, Iris looks puzzled, and the child is encouraged to attempt the phrase again. If the child signs clearly, Iris then frolics and plays with a red balloon.

If the child cannot remember the correct phrase to direct Iris, she can click on a button bearing the picture of the object with which she would like Iris to play. The system shows a short video with a teacher demonstrating the correct ASL phrase, Starner explains. The child can then mimic the teacher to communicate with Iris.
“Gesture-based interaction expands the possibilities for deaf educational technology by allowing children to interact with the computer in their native language,” Starner says. The system was suggested by Georgia Tech’s partners at the Atlanta Area School for the Deaf.

Right now, the current game has eight phrases per level. The player has to get each phrase signed properly before moving on to the next level.

However, the future of the project is a bit up in the air.

VIDEO DEMO

A good video of the project from the Conference of Educational Administrators of Schools and Programs for the Deaf can be viewed here.