Multimodal Corpora: How to build and how to understand them

This workshop aims at studying the interaction between verbal and nonverbal communication and applying this knowledge to both human–human and human–machine interaction. It consists of two parts:

• Part 1: Introduction to the basic concepts of multimodality and to the descriptive and functional aspects of how spoken language and gestures interact. As a hands-on, elementary course, students will design and build their own sample multimodal corpus starting from video recording to annotation to structured corpus analysis.
• Part 2: Understanding multimodality: as an instance of formal and functional data mining, students will learn how to derive both speaker dependent and speaker independent functional, pragmatic and emotional interpretation from a sequence of aligning formal markers (sounds, words, as well as nonverbal gestures) in a communicative event, based on an existing multimodal corpus. It is a hands-on, medium level course.

Part 1: Introduction to multimodality in communication

MULTIMODALITY

• Basic concepts of multimodality from sound to words to gestures
• Descriptive and functional aspects of multimodality

BUILDING A MULTIMODAL CORPUS

• Corpus design: its conceptualisation and technical settings. Recording of a dialogue
• Annotation 1: setting up of the ELAN annotation environment. Automatic and/or manual annotation of the video part of the recorded dialogue for gestures and emotions
• Annotation 2: Automatic and/or manual annotation of the audio part of the recorded dialogue for prosody and emotions
• Descriptive data analysis for the alignment of video and audio markers

Part 2: Understanding multimodality in communication

THE MULTIMODAL BUILDING BLOCKS OF COMMUNICATION

• The smallest verbal and nonverbal markers across modalities and their temporal organisation
• Combination and optionality of markers based on individual variation, scenario and ontology

UNDERSTANDING A MULTIMODAL CORPUS

• Corpus analysis within the ELAN annotation environment. Text analysis. Sequential and hierarchical multimodal pattern analysis
• Analysis 1: Descriptive analysis of the multimodal alignment of verbal and nonverbal markers (prosody, syntax and gestures)
• Analysis 2: Interpretative (functional, pragmatic) analysis of the multimodal alignment of verbal and nonverbal markers (prosody, syntax and gestures)
• Discussion: towards a more effective human–computer interaction (theoretical and technological challenges)