1. The purpose of the project and development during the project period

We live in a world that contains objects that we can see, hear, smell and touch, and our perceptual systems reflect this reality. For example, when a basketball hits the wooden floor of a sports arena, visual, tactile, auditory and other sensory signals are seamlessly fused into a unitary multisensory percept. Everyday experiences suggest to us that events and objects we perceive in the physical world can be clearly separated from events and objects that we conjure up in our minds. However, is the distinction between perception and imagination really as clear-cut as is generally assumed? Is it possible for imagined sounds to change what one sees or for imagined sights to change what one hears? In this project, we used psychophysics and experimental psychology approaches to show that classical multisensory illusions can be elicited by simply instructing participants to imagine a sensory stimulus. Collectively, our results demonstrate that imagined stimuli in one sensory modality can be integrated with real stimuli in a different sensory modality to produce robust multisensory perceptions. This finding represents a major advance in our understanding of the nature of imagery, the basic features of top-down influences on multisensory integration, and the ways in which imagery can change our perception of the world. Because imagination is the hallmark of human society and culture, our results should have a bearing on future research within many areas of the humanities.

Our specific aims as specified in the original research plan are as follows:

(i). To provide evidence that the ‘ventriloquism illusion’ – that a visual stimulus leads to the translocation of sound toward the visual stimulus – can be elicited by replacing a real visual stimulus with an imaged visual stimulus.

(ii). To demonstrate that the imagery-induced ventriloquism illusion produces a sensory adaption aftereffect.

(iii). To demonstrate that the rubber hand illusion – a body illusion that involves the integration of visual and somatosensory signals – can be elicited by replacing a tactile or visual stimulus with an imagined stimulus in the corresponding modality.

To address the first aim – whether what you imagine in one sensory modality changes what you perceive in another sensory modality – we not only used the ventriloquist illusion (Berger and Ehrsson 2018) but also the so called ‘cross-bounce’ illusion (Berger and Ehrsson, 2013), in which an auditory stimulus presented at the moment two passing objects coincide promotes the illusory perception that the objects bounce off, rather than cross by, one-another. By using a psychophysics approach and signal detection theory analysis, the results from this study revealed that imagining a sound at the movement the two visual objects coincide elicits the bounce precept just like a real sound (Berger and Ehrsson 2017). These results address our aim of providing evidence that imagined sensory stimuli can integrate with real cross-modal sensory stimuli to produce multisensory perception.

We also successfully addressed the second aim of our research project and demonstrated that mental imagery can produce multisensory illusion aftereffects (Berger and Ehrsson 2018). In this study we conducted a series of well-controlled experiments and presented conclusive evidence that the imagery-induced ventriloquist illusion produced an aftereffect.

With respect to the third aim, employed motor imagery - the imagery of moving the body with the associated sensations of movement – which is a robust form of imagery for most participants to perform. As a stepping stone, we first conducted an experiment where we showed that motor imagery generates internal sensory predictions about the sensory consequences of the imagined movement (Kilteni et al. 2018), which suggested that motor imagery recruits the same sensory-motor mechanisms and leads to the same sensation of movement as real movement. We then conducted the study with the rubber hand illusion and found that motor imagery can elicit the “moving rubber hand illusion”, which is a version of the rubber hand illusion that is elicited by finger movement rather than brushstrokes (Kalckert and Ehrsson 2012). We are currently in the process of finalizing the manuscript for submission to a leading peer-reviewed international journal (Berger and Ehrsson, in preparation). In sum, the project has worked out very well and we have successfully addressed all the original aims.

2. The project's three most important results

The three most important results are as follows: (i) mental imagery changes the perception of the world; (ii) mental imagery induces central plasticity and changes the future perception of the world; and (iii) mental imagery changes the perception of one’s own body.

To demonstrate that imagined sensory stimuli are integrated with external sensory stimuli to shape our perception of the world, we made use of the cross-bounce illusion in which an auditory stimulus presented at the moment two passing objects meet promotes the perception that the objects bounce off rather than cross by one another. In our study (Berger and Ehrsson 2017), we show that the content of imagined sound changes visual motion perception in a manner that is consistent with multisensory integration. Specifically, we found that auditory imagery of a sound with acoustic properties typical of a collision (i.e., damped sound) promoted the bounce percept, but auditory imagery of the same sound played backwards (i.e., ramped sound) did not. Moreover, the vividness of the participants' auditory imagery predicted the strength of this imagery-induced illusion. Collectively, these findings suggest that a imagery-induced multisensory illusion reflects the successful integration of real and imagined cross-modal sensory stimuli, and more generally, that what we imagine hearing can change what we see.

Our second major discovery was that what we imagine in our minds can change how we perceive the world in the future. Stable perception of the environment depends on a continuous process of multisensory integration and recalibration that is responsible for maintaining a correspondence between the senses. This process depends on the plasticity of our central sensory systems. The so-called ventriloquism aftereffect - a shift in the perceived localization of sounds presented alone after repeated exposure to spatially mismatched auditory and visual stimuli - is a well-established example of this type of plasticity in the audiovisual domain. In a published study (Berger and Ehrsson 2018), we found that it is possible to elicit an imagery-induced ventriloquism aftereffect in which imagining a visual stimulus elicits the same specific auditory aftereffect as actually seeing one. This result demonstrated that mental imagery recalibrates the senses and induced the same cross-modal sensory plasticity as real sensory stimuli.

Finally, we found that mental imagery can change the multisensory perception of our own body. Specifically, we found that mental imagery of finger movement elicits a robust illusion of owning a rubber hand that moves its finger. In this experiment, we systematically varied the relative timing of the imagined and seen finger movements (synchronous versus asynchronous). Only the synchronous condition elicits the rubber hand illusion. The illusion was quantified subjectively with questionnaires, and objectively in the form of misreaching in a pointing task when asked to localize the position of the real (hidden) hand (“proprioceptive drift”). We also measured activity in the participants' first dorsal interosseous muscle using electromyography to ensure that the participants did not move their finger during the motor imagery conditions. The results suggest that the moving rubber hand illusion can be elicited by imagined finger movements. This result is important because it shows that our conclusions regarding mental imagery and multisensory perception generalize to different combinations of sensory modalities, including the integration of visual information and somatosensory information from the body. Thus, mental imagery not only changes how we perceive the outside world but also how we perceive our physical selves.

3. Dissemination of the results

The results of the project have been communicated to other scientists at international conferences and workshops, and via the principal investigator’s invited talks at various international conferences. We have also communicated our results in peer-reviewed international journals; all our articles are open access. The American Psychology Association issued a press release from our study published in Psychological Science (Berger and Ehrsson 2018) and described it in an online report (12 April: https://www.psychologicalscience.org/news/releases/imagining-an-object-can-change-how-we-hear-sounds-later.html).