Building and analysing multimodal corpora

Language is used in a variety of environments. Going beyond text, it is used in a wide range of spoken utterances revealing what is hardly captured in written form: the personality and mood of the speaker, his / her attitudes and intentions, the dynamics of a conversation with changing roles of the speakers, and the success or incompleteness of an interaction. In order to convey all or any of these kinds of information the sound does not go alone: it is accompanied by the flow of gestures, those of the eyes, head, hand or posture.

The workshop will focus on this broad understanding of spoken language and offer an insight into, as well as hands-on experience with the multimodal complex of integrating sounds, words and gestures. As a result, the acquired knowledge can be used in better understanding speech and communication related issues in areas related to psycholinguistics, studies in emotions, speech and communication disorders and therapy, speaker verification in forensic linguistics, in human-machine interaction, communication in general, as well as wherever spoken utterances and interactions are subject to any kind of further investigation.

The first part of the workshop will focus on spoken language based on the audio channel of recorded interactions. Beyond the verbal content of utterances, we will extract temporal sequences of turn-taking, silences and other prosodic features as important and meaningful factors to characterize a conversation and human behavior. From the methodological aspect, the participants will learn about computational ways of speech processing including data extraction (spectral and prosodic analysis), annotation and visualization technics using the software environment and scripting language of the Praat program.

The second part will focus on multimodality and integrate physical and interpretive parameters of sounds with those associated with gestures. Real-life experience will be gained from on-site video and audio recordings, their annotation with the ELAN annotation tool, the analysis of the data both within ELAN and by the use of the statistical framework offered by Theme to capture hidden multimodal patterns of behavior.