Predicting Head Pose from Speech with a Conditional Variational Autoencoder

Greenwood, David, Laycock, Stephen and Matthews, Iain (2017) Predicting Head Pose from Speech with a Conditional Variational Autoencoder. In: Proceedings of Interspeech 2017. ISCA, pp. 3991-3995.

Preview

PDF (Accepted manuscript) - Accepted Version Download (716kB) | Preview

Abstract

Natural movement plays a significant role in realistic speech animation. Numerous studies have demonstrated the contribution visual cues make to the degree we, as human observers, find an animation acceptable. Rigid head motion is one visual mode that universally co-occurs with speech, and so it is a reasonable strategy to seek a transformation from the speech mode to predict the head pose. Several previous authors have shown that prediction is possible, but experiments are typically confined to rigidly produced dialogue. Natural, expressive, emotive and prosodic speech exhibit motion patterns that are far more difficult to predict with considerable variation in expected head pose. Recently, Long Short Term Memory (LSTM) networks have become an important tool for modelling speech and natural language tasks. We employ Deep Bi-Directional LSTMs (BLSTM) capable of learning long-term structure in language, to model the relationship that speech has with rigid head motion. We then extend our model by conditioning with prior motion. Finally, we introduce a generative head motion model, conditioned on audio features using a Conditional Variational Autoencoder (CVAE). Each approach mitigates the problems of the one to many mapping that a speech to head pose model must accommodate

Item Type:	Book Section
Faculty \ School:	Faculty of Science > School of Computing Sciences
UEA Research Groups:	Faculty of Science > Research Groups > Interactive Graphics and Audio
Related URLs:	http://www.isca-speech.org/archive/Inter...
Depositing User:	Pure Connector
Date Deposited:	13 Sep 2017 05:08
Last Modified:	02 Jun 2023 01:13
URI:	https://ueaeprints.uea.ac.uk/id/eprint/64846
DOI:	10.21437/Interspeech.2017-894

Downloads

Actions (login required)

View Item