Evaluating the influence of audiovisual unity in cross-modal temporal binding of musical stimuli

Abstract

An observer’s inference that multimodal signals come from a common underlying source can facilitate cross-modal binding in the temporal domain. This ‘unity assumption’ can cause asynchronous audiovisual speech streams to seem simultaneous (Vatakis & Spence, 2007), but follow-up work has been unable to replicate this effect for non-speech, musical events (Vatakis & Spence, 2008). Given that amplitude envelope (the changes in energy of a sound over time) has been shown to affect audiovisual integration, the current study investigates whether previous null findings with musical stimuli can be explained by the similarity in amplitude envelope between audiovisual conditions. To test whether amplitude envelope affects temporal cross-modal binding, Experiment 1 contrasted events with clearly differentiated envelopes: cello and marimba audiovisual stimuli. Participants performed an un-speeded temporal order judgments task; they viewed audio-visually matched (e.g. marimba auditory with marimba video) and mismatched (e.g. cello auditory with marimba video) versions of stimuli at various stimulus onset asynchronies and indicated which modality was presented first. As predicted, participants were less sensitive to temporal order (greater JNDs) in matched conditions, suggesting that the unity assumption facilitates synchrony perception outside of non-speech stimuli. Results from Experiments 2 and 3 revealed that when spectral information was removed, amplitude envelope alone could not facilitate the influence of audiovisual unity on temporal binding. We propose that both amplitude and spectral cues affect the percept of audiovisual ‘unity’, likely working in concert to help an observer determine the causal source of an auditory event.