Arm motion symmetry in conversation

Windle, Jonathan, Taylor, Sarah, Greenwood, David and Matthews, Iain (2022) Arm motion symmetry in conversation. Speech Communication, 144. pp. 75-88. ISSN 0167-6393

Preview

PDF (1-s2.0-S0167639322001054-main) - Accepted Version Available under License Creative Commons Attribution. Download (8MB) | Preview

Abstract

Data-driven synthesis of human motion during conversational speech is an active research area with applications that include character animation, computer gaming and conversational agents. Natural looking motion is key to both perceived realism and understanding of any synthesised animation. Multi-modal speech and body-motion data is scarce and limited, so it is common to augment real motion data by mirroring the body pose to double the number of training samples. This augmentation is based on the assumption that a person’s gesturing is not affected by handedness and that the reflected pose is plausible. In this study, we explore the validity of this assumption by evaluating the reflective symmetry of a speaker’s arms during conversational exchanges. We analyse the left and right arm motion of 36 subjects during dyadic conversation and present the per-frame symmetry of the arm gestures. To identify temporal offsets caused by the presence of a leading hand, we compute the time lag between movements of the left and right arms. We perform a nearest neighbour search to test the validity of any mirrored pose. We also consider information theory to examine the information gain from mirroring the data. We implement a speech-to-gesture generative model to determine the efficacy of lateral mirroring techniques for data augmentation. Our findings suggest that both positional symmetry and left–right motion offsets vary from speaker to speaker. We conclude that data augmentation by mirroring is valid in certain cases when considering the mirrored pose as a new virtual identity, but that it should be carefully considered as a generic approach if the gesturing style and handedness of the original speaker is to be maintained.

Item Type:	Article
Additional Information:	Acknowledgements: Sarah Taylor was supported by the Engineering and Physical Research Council (Grant number EP/S001816/1).
Uncontrolled Keywords:	speech-driven conversational agents,motion symmetry,conversational gesture analysis
Faculty \ School:	Faculty of Science > School of Computing Sciences
Depositing User:	LivePure Connector
Date Deposited:	08 Sep 2022 15:30
Last Modified:	01 Jun 2023 13:30
URI:	https://ueaeprints.uea.ac.uk/id/eprint/87977
DOI:	10.1016/j.specom.2022.08.001

Downloads

Actions (login required)

View Item