Multi-Point Gesture Recognition Leveraging Acoustic Signals and CNN

Abstract

With the emergence of smart home appliances and AR/VR content, the demand for a better user interface is constantly increasing. Traditional user interface, touch-based interface is no longer applied to AR/VR applications and vision or RF-based gesture recognition requires camera and sensors, resulting in additional cost. The accuracy of above-mentioned methods highly depends on brightness and surrounding envi- ronment, therefore they fail to guarantee robustness. There are several researches on acoustic-based gesture recognition, but are limited to single point movement such as straight line, circle and triangle. In this paper, we design and implement multi-point gesture recognition system utilizing both the acoustic signals and machine learning technique. We use channel impulse response (CIR) containing multi-path information to recognize multi-point gestures, and construct a CNN model to learn its features. In addition, we present a method of constructing a CNN model suitable for gesture recognition. Evaluation results show that our system successfully recognizes multi-point gestures and demonstrate its efficacy.