In everyday life, we are often faced with noisy situations like a “cocktail party” environment (Cherry, 1953), talking with friends while music is playing, or dining in a crowded cafe, where noise makes it difficult to understand speech and participate in conversation (Miller, 1947). The world is also multisensory, and we constantly receive information from all sensory modalities simultaneously, including auditory and visual information. Under such high-load situations, the mechanism of selective attention, which focuses on the necessary information and ignores unnecessary information, plays an important role in efficient perceptual cognition (Broadbent, 1954; Forte et al., 2017; Fritz et al., 2007; Lavie, 2005; Lavie, 2010; Shinn-Cunningham, 2008; Talsma et al., 2010). Previous electroencephalography (EEG) and magnetoencephalography (MEG) studies have shown that event-related potentials (ERPs) generated by the same auditory stimulus are modified by attentional modulation (Astheimer and Sanders, 2009; Hillyard et al., 1973; Mesgarani and Chang, 2012; Näätänen et al., 2011; Picton and Hillyard, 1974; Snyder et al., 2006). For example, the brain responses to attended sounds were greater than those to unattended sounds.

Additionally, some studies have examined selective attention during dichotic listening using ERPs such as N100 and N100m (Bertoli et al., 2005; Ding and Simon, 2012; Hari and Mäkelä, 1988; Kawase et al., 2012; Shirakura et al., 2021). In the dichotic listening test, subjects were simultaneously presented with two different auditory stimuli that was, directed into different ears over headphones or earphones. Generally, subjects were instructed to pay attention to one stimulus and ignore distractor stimuli, such as contralateral or masking sounds. Previous reports have shown that N100 and N100m responses were not significantly affected when the contralateral distractor stimuli sounds were continuous white or speech bubble noises (Bertoli et al., 2005; Hari and Mäkelä, 1988; Kawase et al., 2012), whereas the responses were significantly suppressed when the contralateral sound was presented as speech, music, or intermittent noise (Hari and Mäkelä, 1988; Shirakura et al., 2021). The effects of contralateral sounds on ERPs, such as N100m, vary with the type of contralateral sound, even if the sound pressure level is the same. It has been reported that this may be due to differences in the saliency of each stimulus (Shirakura et al., 2021).

Several studies have examined the effects of different types of contralateral sounds and attentional modulation on ERPs during dichotic listening, when competing information is presented with the same sensory modality, such as auditory stimuli (Bertoli et al., 2005; Ding and Simon, 2012; Hari and Mäkelä, 1988; Kawase et al., 2012; Shirakura et al., 2021). However, despite the multisensory nature of the world, little is known about whether ERPs are modulated under cross-modal conditions in which visual information, a different sensory modality, is presented during dichotic listening.

Understanding whether the presentation of visual information simultaneously with distractor sounds has a cross-modal effect on the perceptual processing of the target sound is particularly important because we are constantly presented with information involving all our sensory modalities simultaneously in our daily lives. For example, compared to listening to a person's conversation while listening to the radio with audio-only information, or listening to a person's conversation while watching television with audio-visual information, the question remains as to whether audio-visual distractors make it more difficult to hear a person's conversation.

The present study aimed to clarify whether presenting visual information related to contralateral sound during a dichotic listening task has an additive effect on contralateral sound suppression. We modified the contra-sound effects paradigm of Shirakura et al. (2021) to investigate the effects of visual information related to contralateral sound on ERPs such as N100m responses. The subjects were instructed to selectively direct their attention to the target sound and press a button. In previous studies, passive listening conditions were set according to instructions to pay attention to a silent movie (Kong et al., 2014; Vanthornhout et al., 2019; Zhang et al., 2016). In the present study, we manipulated the perceptual load by adjusting the conditions to either audio-only, visual-only, or audio-visual, using movies as the distractor stimuli.

We hypothesized that the presentation of visual information for the contralateral sound modulates a perceptually loaded distractor that requires more attentional resources and has an additive effect on the contralateral sound effect.