Ethically-driven multimodal emotion detection for children with autism

Emotion detection (ED) aims to identify people's emotions automatically. However, most ED applications do not consider individuals who express emotions differently, such as people with autism. Although studies have already focused on creating ED models tailored for children with ASD, this application of ED suffers from a scarcity of resources and remains underperforming compared to the state-of-the-art ED models for the general population. This thesis addresses the gap in automatic ED between the general population and autistic children while ensuring an ethically driven approach, i.e., having the well-being of participants as the main priority during the whole research process. To meet our research objectives, we created a data collection framework that minimises emotional disruption to the participants, respects their privacy and rights according to GDPR, and provides a dataset that can be shared with the research community. We created CALMED, a multimodal annotated dataset for ED featuring children with autism that includes privacy-preserving features, novel target emotion classes, annotations provided by the participants' parents and a researcher specialist who works with children with ASD. Using the CALMED dataset, we created hundreds of models with unique configurations and analysed them to explore the effectiveness of various methods for multimodal ED in autism. Then, utilising the knowledge acquired in this analysis, we proposed a multimodal ED model that outperformed the previous state-of-the-art, reaching 81.56% and 75.47% for accuracy and balanced accuracy, respectively. Finally, we created and shared many systems to support the data acquisition process and data experiments creation and analysis. We placed great importance on ensuring reproducibility, reusability, and ethical conduct. This research has made significant contributions to the field of ED applied to ASD. It has provided a valuable dataset, analytical insights, a state-of-the-art model, and many computer systems that can serve as a groundwork for future work.

Publisher

University of Galway

Rights

Attribution-NonCommercial-NoDerivatives 4.0 International

cbn

Collections

University of Galway Theses (PhD Theses)