Probral Workshops and Symposium, 21-22 May and 1-2 July 2024: Methodological aspects of analyzing motion and video-mediated interaction

Audiovisual speech communication consists of both acoustic and visual components. The acoustic component is the speech signal, which is a one-dimensional pressure signal. To ensure accurate reproduction or computational analysis, the speech signal is typically sampled at ten or twenty thousand samples per second. For computational analysis, it is usually segmented into twenty-millisecond windows, each characterized by parameters such as formant frequencies or LPC coefficients. These parameters are represented as vectors within the acoustic space. In turn, the visual component comprises facial expressions, head movements, and body gestures. These movements are commonly represented by the coordinates of various points tracked in the speaker's image, forming vectors within the visual space.

In this presentation, we will demonstrate how to represent both speech acoustics and speech gestures in coordinate systems that align with the inherent degrees of freedom in speech communication. Additionally, we will explore methods for aligning acoustic and visual coordinate systems to investigate their relationship. These techniques are applicable for analyzing both individual speakers and dialogues.

This talk presents a methodology for assessing the coupling between different modalities of speech (acoustic, visual) during spoken communication. The techniques presented can be equally applied to both intra- and inter-speaker scenarios. The talk focuses on two main points: i) the use of the FlowAnalyzer software to measure motion from video and ii) the use of CMA (Correlation Map Analysis) as a means of assessing the time-varying coupling between two domains of interest. We first show how to use FlowAnalyzer to extract motion signals from pre-recorded video sequences. We demonstrate the utility of the tool by showing how it allows the experimenter to rely only on ordinary video cameras to measure motion in a completely non-invasive way during speech production experiments. We then show how to use the CMA technique to quantify the time-varying coupling between the visual and acoustic components of speech. In our demonstration, the visual domain will be represented by the motion signals produced by FlowAnalyzer, whereas the acoustic domain will be represented by some parameterization of the acoustic waveform, such as the root mean square (RMS) value or Line Spectrum Pair (LSP) coefficients. We finish by discussing possible directions for future development of the tools and techniques presented.

In this mini-course we will show how to use tools and techniques outlined in our talk "Using optical flow and correlation maps to assess coordination during communicative interaction" to analyze data acquired during actual interactive communication sessions. More specifically, we will show how to:

- install FlowAnalyzer
- use FlowAnalyzer to extract motion data from regions of interest in a video file
- plot the extracted motion signals
- export the extracted motion signals to different file formats so that they can be imported into other frameworks for later analysis
- import the motion files saved by FlowAnalyzer into Python and Matlab
- apply Correlation Map Analysis (CMA) to the motion signals extracted by FlowAnalyzer in order to quantify intra- and inter-speaker coordination during communicative interaction
- extract parameters of interest from the correlation maps with the aim of summarizing the main aspects of the coordination

We also look forward to discussing with colleagues how these tools could be applied to their particular research efforts and how they could be improved in order to make them more useful to researchers.

Programme