Explainable Smile Detection Trough Semi-Supervised Learning with Generative Networks

Abstract

Facial expression is the most natural and direct way to convey people’s inner emotions and thoughts. Among basic facial expressions, smile is one of the most important facial expressions and smile is a potent social tool. Although facial expressions and smiles are essential to human life, there are people who feel it difficult to make facial expressions or smile for various reasons. They practice making facial expressions look natural, usually by practicing on their own without any guide or by using a clinic to improve their facial expressions. In these situations, it is important to know the current state of facial expression, find out which facial parts to be moved, and figure out how to move the facial parts. Therefore, it is necessary to provide a guide to instruct which facial parts to be moved and how to move the parts. To this end, first, the current facial expression needs to be detected. And then, it is necessary to localize which part of the face needs to be changed to become the target expression. That way, it is possible to generate a more precise guide on how to move on that part. In the computer vision domain, facial expression recognition has been studied a lot and its performance is also excellent. However, there have been no studies considering visual guides to help practice facial expressions. Thus, in this paper, we propose a new approach for smile detection and localization to guide which facial parts to be moved to make smile. Our proposed method is based on a basic theme of reconstruction-based anomaly detection using generative model and used convolutional variational autoencoder(cVAE) for reconstruction. And we used the difference between the reconstructed image and the input to detect the smile. Also, if the prediction is not a smile, the facial area that must move to be a smile is visualized in the form of a heat map, which called smile localization. Unlike previous supervised learning-based smile detection studies, our proposed method can figure out which facial parts to be moved to make smile. Experimental results demonstrate that our approach achieves a promising performance in smile detection. Not that all, our smile localization approach is explainable and deliberate. Also, there is the potential to generate a guidance on how to move the facial part to make the current facial expression smile.