ASL Trigger Recognition in Mixed Activity/Signing Sequences for RF Sensor-Based User Interfaces

The past decade has seen great advancements in speech recognition for control of interactive devices, personal assistants, and computer interfaces. However, Deaf and hard-ofhearing (HoH) individuals, whose primary mode of communication is sign language, cannot use voice-controlled interfaces. Although there has been significant work in video-based sign language recognition, video is not effective in the dark and has raised privacy concerns in the Deaf community when used in the context of human ambient intelligence. RF sensors have been recently proposed as a new modality that can be effective under the circumstances where video is not. This paper considers the problem of recognizing a trigger sign (wake word) in the context of daily living, where gross motor activities are interwoven with signing sequences. The proposed approach exploits multiple RF data domain representations (time-frequency, range-Doppler, and range-angle) for sequential classification of mixed motion data streams. The recognition accuracy of signs with varying kinematic properties is compared and used to make recommendations on appropriate trigger sign selection for RFsensor based user interfaces. The proposed approach achieves a trigger sign detection rate of 98.9% and a classification accuracy of 92% for 15 ASL words and 3 gross motor activities.

Work in Progress: Enabling robot device discovery through robot device descriptions

There is no dearth of new robots that provide both generalized and customized platforms for learning and research. Unfortunately as we attempt to adapt existing software components, we are faced with an explosion of device drivers that interface each hardware platform with existing frameworks. We certainly gain the efficiencies of reusing algorithms and tools developed across platforms but only once the device driver is created. We propose a domain specific language that describes the development and runtime interface of a robot and defines its link to existing frameworks. The Robot Device Interface Specification (RDIS) takes advantage of the internal firmware present on many existing devices by defining the communication mechanism, syntax and semantics in such a way to enable the generation of automatic interface links and resource discovery. We present the current domain model as it relates to differential drive robots as a mechanism to use the RDIS to link described robots to HTML5 via web sockets and ROS (Robot Operating System).