MMASCS – Multi-Modal Annotated Synchronous Corpus of Speech

Intro

The MMASCS multi-modal annotated synchronous corpus of audio, video, facial motion and tongue motion data of normal, fast and slow speech is a collection of speech recordings for research purposes. It consists of a total of 770 sentences spoken by a male Austrian German speaker, recorded at slow, normal and fast speaking rates, in in the following modalities:

Optical facial motion tracking
Electromagnetic tongue motion tracking
Frontal control video from optical system (gray scale)
Camcorder video from slightly slanted view (color)
Studio-quality audio

The following video shows an example sentence:

Related Publications

A detailed description of the MMASCS corpus was presented at the 9th edition of the Language Resources and Evaluation Conference (LREC) in May 2014:

Dietmar Schabus, Michael Pucher, and Phil Hoole. "The MMASCS multi-modal annotated synchronous corpus of audio, video, facial motion and tongue motion data of normal, fast and slow speech". In Proceedings of the 9th Language Resources and Evaluation Conference (LREC), pp. 3411–3416, Reykjavik, Iceland, May 2014.

This paper can be downloaded from the LREC website, a preprint version of it can be downloaded here, and BibTeX information can be obtained here. Please cite this paper if you use the corpus for your research.

For each of the recorded utterances (320 at normal, 320 at fast and 130 at slow speaking rate), the corpus contains:

A RIFF wave audio file (mono, 44100 Hz, 16-bit signed integer)
Facial marker data in the form of a matrix stored as a text file (41 markers, 3 spatial coordinates per marker, 100 Hz)
Articulograph data in the form of a matrix stored as a text file (8 markers, 3 spatial coordinates per marker, 100 Hz)
A Python Pickle file combining the above three, suitable for playback with the player listed below
A gray scale video from the optical system (H264/AAC MPEG-4, 640x480 pixels, 100 fps)
A color video from a separate camcorder (H264/AAC MPEG-4, 960x540 pixels, 50 fps)
A combined video showing a frontal view rendering of the marker data, a side view rendering of the marker data, the gray scale video, the color video, and a plot of the waveform with a time indicator (H264/AAC MPEG-4, 1200x900 pixels, 100 fps)
A text file listing the phones spoken in the utterance including begin and end times of all phones
A HTK quin-phone full-context label file

Furthermore, the package contains:

The 320 sentences in plain text
A software which can play back pickle files as mentioned above (Python, OpenGL).
This video shows an example:

Obtaining the Data

The data can be obtained free of charge for scientific research purposes (see the license). Contact Michael Pucher (michael.pucher@oeaw.ac.at) if you would like to obtain a copy.

Acknowledgements

The MMASCS corpus was created in the research project Adaptive Audio-Visual Dialect Speech Synthesis (AVDS), funded by the Austrian Science Fund (FWF) under the project number P22890-N23. This project was coordinated by the Telecommunications Research Center Vienna (FTW). The Competence Center "FTW Forschungszentrum Telekommunikation Wien GmbH" is funded within the program COMET Competence Centers for Excellent Technologies by BMVIT, BMWFW and the City of Vienna. The COMET program is managed by the FFG.

The recordings were carried out in the premises of the Institute of Phonetics and Speech Processing of the Ludwig-Maximilians Universität Munich.

Contact

Michael Pucher (michael.pucher@tugraz.at)