LLAniMAtion: LLAMA driven gesture animation

Windle, J., Matthews, I. and Taylor, S. (2024) LLAniMAtion: LLAMA driven gesture animation. Computer Graphics Forum, 43 (8). ISSN 0167-7055

Preview

PDF (Windle_etal_2024_ComputerGraphicsForum) - Published Version Available under License Creative Commons Attribution. Download (26MB) | Preview

Abstract

Co-speech gesturing is an important modality in conversation, providing context and social cues. In character animation, appropriate and synchronised gestures add realism, and can make interactive agents more engaging. Historically, methods for automatically generating gestures were predominantly audio-driven, exploiting the prosodic and speech-related content that is encoded in the audio signal. In this paper we instead experiment with using Large-Language Model (LLM) features for gesture generation that are extracted from text using Llama2. We compare against audio features, and explore combining the two modalities in both objective tests and a user study. Surprisingly, our results show that Llama2 features on their own perform significantly better than audio features and that including both modalities yields no significant difference to using Llama2 features in isolation. We demonstrate that the Llama2 based model can generate both beat and semantic gestures without any audio input, suggesting LLMs can provide rich encodings that are well suited for gesture generation.

Item Type:	Article
Additional Information:	Publisher Copyright: © 2024 The Author(s). Computer Graphics Forum published by Eurographics - The European Association for Computer Graphics and John Wiley & Sons Ltd.
Uncontrolled Keywords:	animation,ccs concepts,• computing methodologies → machine learning algorithms,computer graphics and computer-aided design ,/dk/atira/pure/subjectarea/asjc/1700/1704
Faculty \ School:	Faculty of Science > School of Computing Sciences
Related URLs:	http://www.scopus.com/inward/record.url?...
Depositing User:	LivePure Connector
Date Deposited:	28 Nov 2024 01:37
Last Modified:	07 Dec 2024 01:40
URI:	https://ueaeprints.uea.ac.uk/id/eprint/97795
DOI:	10.1111/cgf.15167

Downloads

Actions (login required)

View Item