Stefan R. Schweinberger: "Multimodal Social Interaction: Tools for Basic Science, Assessment, and Intervention in Face and Voice Communication Science"

Univ. Prof. Dr. Stefan R. Schweinberger, Friedrich Schiller University, Jena, Germany:

"Multimodal Social Interaction: Tools for Basic Science, Assessment, and Intervention in Face and Voice Communication Science"

Abstract
Social interaction is regarded a main driving factor to shape human brain development. Emotions may be fundamentally multimodal in nature, with strong overlap between the neural systems mediating production and perception (in nonverbal vocal communication: Frühholz & Schweinberger, 2021). We systematically use parameter-specific facial and vocal morphing and caricaturing in experiments – but also in assessment of face and voice perception, and in interventions for people with challenges to social interaction. My first example regards vocal emotion perception in cochlear implant (CI) users. Extensive research exists on speech comprehension with a CI, but research on abilities to perceive nonverbal emotional prosody or speaker identity is largely missing – despite findings that nonverbal abilities for communicating socio-emotional cues can be as important for quality of life (QoL) with a CI. I will show how parameter-specific voice morphing can help assess acoustic voice processing with a CI, and how perceptual training with digital caricatures of emotions can help CI users to enhance emotion perception skills. My second example regards established benefits in facial emotion recognition that are elicited by motor imitation of a perceived emotion. Working with automatic emotion recognition algorithms to quantify imitation, we show that emotion recognition by adults with autism can benefit from imitation in facial emotion recognition – and in vocal emotion recognition as well. My third example regards people pre-screened for their poor face recognition skills. I present a study which suggests that parameter-specific facial caricature training is a promising way to improve performance in selected domains of facial processing. Specifically, our initial results suggest that whereas shape caricature training can improve face matching, texture caricature training promotes marked benefits in face memory tasks. Overall, we propose that these tools are promising both for precision diagnostics of nonverbal social skills and for interventions that benefit social communication and, ultimately, QoL.

Keywords: Cochlear Implant; Autism; Diagnostics; Intervention; Socio-Emotional Communication; Quality of Life