Understanding gestures with systematic variations in movement dynamics

Abstract

Sign language communication includes not only lexical sign gestures but also grammatical processes which represent inflections through systematic variations in sign appearance. We present a new approach to analyse these inflections by modelling the systematic variations as parallel channels of information with independent feature sets. A Bayesian network framework is used to combine the channel outputs and infer both the basic lexical meaning and inflection categories. Experiments using a simulated vocabulary of six basic signs and five different inflections (a total of 20 distinct gestures) obtained from multiple test subjects yielded 85.0% recognition accuracy. We also propose an adaptation scheme to extend a trained system to recognize gestures from a new person by using only a small set of data from the new person. This scheme yielded 88.5% recognition accuracy for the new person while the unadapted system yielded only 52.6% accuracy.