A Deep-Learning-Based Multimodal Screening Model for Elderly Neurological Diseases

Abstract

In this study, we suggest a process for screening rapidly growing neurological diseases. We propose various examination protocols for the screening of neurological diseases and collect data by videotaping the person performing these examination protocols. We converted video data into human landmarks that can capture action information with a much smaller data dimension. We also used voice data because voice patterns are effective indicators of neurological disorders. We designed a subnetwork for each protocol to extract features from landmarks or voice, and a feature aggregator that combines all the information extracted from the protocols to make a final decision. To capture meaningful information about these human landmarks and voices, we conducted transfer learning in various ways. The spatiotemporal characteristics of landmarks are extracted using a pre-trained graph neural network, and the semantic information of speech is extracted using a pre-trained time-delay neural network. These extracted high-level features are then passed onto the subnetworks and a feature aggregator that are simultaneously trained. We also used various data augmentation techniques to overcome the shortage of data. By using a frame-length staticizer that considers the characteristics of the data, we can capture momentary tremors without wasting information. Finally, we quantified the effects of using various protocols, different body parts, and voices through extensive experiments, where the proposed method achieved high performance.