How much is a cow like a meow? A novel database of human judgements of audiovisual semantic relatedness

Wegner-Clemens, Kira, Malcolm, George ORCID: and Shomstein, Sarah (2022) How much is a cow like a meow? A novel database of human judgements of audiovisual semantic relatedness. Attention, Perception & Psychophysics, 84 (4). 1317–1327. ISSN 1943-3921

[thumbnail of Wegner-Clemens.APP.2022]
PDF (Wegner-Clemens.APP.2022) - Accepted Version
Download (1MB) | Preview


Semantic information about objects, events, and scenes influences how humans perceive, interact with, and navigate the world. The semantic information about any object or event can be highly complex and frequently draws on multiple sensory modalities, which makes it difficult to quantify. Past studies have primarily relied on either a simplified binary classification of semantic relatedness based on category or on algorithmic values based on text corpora rather than human perceptual experience and judgement. With the aim to further accelerate research into multisensory semantics, we created a constrained audiovisual stimulus set and derived similarity ratings between items within three categories (animals, instruments, household items). A set of 140 participants provided similarity judgments between sounds and images. Participants either heard a sound (e.g., a meow) and judged which of two pictures of objects (e.g., a picture of a dog and a duck) it was more similar to, or saw a picture (e.g., a picture of a duck) and selected which of two sounds it was more similar to (e.g., a bark or a meow). Judgements were then used to calculate similarity values of any given cross-modal pair. An additional 140 participants provided word judgement to calculate similarity of word-word pairs. The derived and reported similarity judgements reflect a range of semantic similarities across three categories and items, and highlight similarities and differences among similarity judgments between modalities. We make the derived similarity values available in a database format to the research community to be used as a measure of semantic relatedness in cognitive psychology experiments, enabling more robust studies of semantics in audiovisual environments.

Item Type: Article
Additional Information: Funding Information: This research was supported by NSF BCS-1921415 and NSF BCS-2022572 to SS.
Uncontrolled Keywords: audiovisual,multisensory,naturalistic stimulus set,semantics,experimental and cognitive psychology,language and linguistics,sensory systems,linguistics and language ,/dk/atira/pure/subjectarea/asjc/3200/3205
Faculty \ School: Faculty of Social Sciences > School of Psychology
UEA Research Groups: Faculty of Social Sciences > Research Groups > Cognition, Action and Perception
Faculty of Social Sciences > Research Groups > Developmental Science
Related URLs:
Depositing User: LivePure Connector
Date Deposited: 29 Mar 2022 13:30
Last Modified: 21 Apr 2023 01:38
DOI: 10.3758/s13414-022-02488-1


Downloads per month over past year

Actions (login required)

View Item View Item