Development of New Myoelectric Input Interfaces for Immersive and Convenient Virtual Reality

Abstract

In recent years, the virtual reality (VR) and augmented reality (AR) accomplished remarkable advances in both technologically and market. The VR and AR, that had been only applied in some professional fields due to the limited technology, now entered into the commercial market and expanding to various field with the release of new portable head-mounted display devices such as HoloLens and Oculus rift. Especially the progression of VR interface incorporated with the social media and creating new immersive communication platform which provides user realistic communicate with others in the virtual environment. On the other hand, the AR expands toward numerous fields such as medical, industrial and military applications with providing supportive information on the users’ sight. Although the VR and AR are developing fast, the inconvenient input interfaces and difficult of facial gesture recognition in HMD environment are disturbing the future progression. To address this issue, the author proposes three alternative interfaces which are facial electromyogram (fEMG)-based facial expression recognition (FER) system, FER-based control interface system and the silent speech recognition (SSR) system. fEMG is a bioelectric signal generated from facial muscle activation that can be recorded by electrodes attached on the face. By analyzing the unique fEMG pattens elicited by users’ facial expression and / or speech, it is able to predict the users’ facial expression and / or intension of speech so that it could be used to develop the fEMG-based human-machine interface (HMI). In particular, the electrodes for recording fEMG can be easily embedded on the face pad of HMD which makes easy to implement the fEMG-based interface with HMD. Considering that the HMD device occludes the face of user disturbing conventional FER modalities, and the VR and AR applications often encounter a situation that user’s both hands are occupied so that the manipulation of HMD system is unable, the fEMG-based interface has considerable advantage because it does not need hand and it can be implemented in HMD device easily. In the first study, the author developed facial expression capture system based on fEMG recorded only around the eye area. This study has unique advantages in two perspectives which are 1) relies on fEMG recorded from limited region to purpose of easy implementation int the HMD, 2) ability to track the continuous changing facial gesture variable, not a simple classification. The two advantages let the user enjoy more immersive and interactive virtual social activity by presenting the users naturalistic facial gestures on the virtual avatar. In addition, the study employed electrooculogram (EOG)-based eyeblink detection and gaze movement system to achieve more realistic facial gestures. It was difficult to accomplish the continuous facial gesture tracking with limited fEMG location. To compensate it, the author suggested two stage algorithms consisted of the first stage; classify the facial gestures into several rough category and second stage; predict the intensity of facial gesture according to the amplitude of fEMG. The proposed algorithm successfully tracks the continuous change of facial expression with high accuracy. The reported correlation coefficient value between users’ face and predicted face on avatar achieved 0.88. In second study, the author proposed a fEMG-based HMI which can recognize various facial gestures of user and utilize it as a control interface. The system also used limited fEMG locations. Furthermore, the study explored the optimal facial gesture set which can be easily distinguished so that maximize the performance of control interface. For three to 15 commands, the optimal facial sets promising the higher performance were discovered and the result reported the optimal facial gesture set achieved considerably better performance compared to the randomly selected facial gestures. With the result of eight optimal facial gesture commands, the fEMG-based video player control interface was developed. It is expected that the optimal facial gesture set found in the study will compensate the inevitable performance degradation due to the limited fEMG electrodes’ location for practical usage. In the last study, the author suggested the SSR system-based on fEMG recorded only around the eye region, same as previous studies. To circumvent the aforementioned problems, the author suggested to simplify the sequential change of lip motions while speaking into a series of six different lip motions. in addition, the user’s intended word was predicted by comparing the sequence of lip motions with the pre-determined template using the dynamic time warping method. The suggested system showed average accuracy of 93.49%. In this thesis, the author proposed immersive and convenient fEMG-based interfaces which potentially solve the problems that VR and AR interfaces have. In further research, the author would like to discover the method to increase the robustness of the system or minimizing the personal training of the system to develop more popular practical interface and it is expected that the study would be helpful to development and popularization of HMD-based applications.