Introduction

The perception of the world around us is a multisensory experience, relying on sensory modalities like vision, touch, and hearing. A crucial aspect of this multisensory perception is the understanding of the surface texture of objects [1,2]. In research on tactile perception, tactile and visual cues are emphasized due to their role in the general recognition of objects and in distinguishing different textures [3–5]. However, auditory cues are essential for more delicate texture analyses, as the sounds produced during interaction with textures can influence both perception and discrimination [6,7]. Moreover, previous neuroimaging studies have reported significant interactions and shared neural mechanisms between auditory and tactile sensory processing, highlighting their neurological interconnections [8–10].

The texture perception, especially the surface stickiness, remains relatively understudied, despite its substantial implications for daily interactions with various materials and objects. To date, the study of surface stickiness has been challenging due to the difficulty in quantifying the sticky sensation, relying on mechanical sensations such as the skin’s friction against a surface or its stretching. In our prior research [11], we investigated the psychophysical bases of multisensory surface stickiness perception. This work revealed that auditory cues provide better discrimination sensitivity than tactile and visual cues, highlighting the importance of auditory perception in discerning stickiness. However, while these findings have helped clarify the role of individual modalities, the comprehensive understanding of their combined impact on stickiness intensity perception is still unclear. In this behavioral and functional MRI (fMRI) study, we aimed to explore how the human brain perceives surface stickiness, particularly when there are discrepancies in intensity cues.

Methods Participants and ethics approval

A total of 22 participants (9 females, undergraduate students, mean age = 22.6 ± 2.9 years) took part in the experiment. However, the data of one participant who had palmar hyperhidrosis was excluded from the analyses due to the potential influence of this condition on tactile perception. All participants were right-handed and had no known impairments in auditory or tactile processing. Synesthetes were not included in the study. Experimental procedures were approved by the ethical committee of Sungkyunkwan University (IRB# 2018-05-001) and the study was conducted in accordance with the Declaration of Helsinki. All participants were informed about the experimental procedure, and provided written informed consent prior to their participation.

Stimuli

In the same manner as previous research [11], we utilized repositionable tapes (3M Center, St. Paul, MN, USA) for the sticky stimuli. Specifically, we selected two different tapes, ‘9415pc’ and ‘9425’, each with a unique physical stickiness intensity measured using the ‘probe tack test’ [12]. This test assesses the peak adhesive force, reflecting the tape’s instantaneous adhesion property. The ‘9415pc’ tape had a stickiness intensity of 22.9 gf (gram-force), and the ‘9425’ tape had a stickiness intensity of 131.2 gf. Using these tapes, we created tactile and auditory stimuli. For the tactile stimulus, a tape measuring 5 × 1.9 cm was attached to an acrylic plate measuring 5 × 9 cm. For the auditory stimulus, the sound generated while touching and detaching with the right index fingertip was recorded using a condenser microphone. Each audio clip was 3.5 s long and consisted of two parts, that is, touching period for the first 2 s, and detaching period for the last 1.5 s.

Experimental design

In this study, we conducted both behavioral and fMRI experiments. In the behavioral experiment, participants completed a total of 72 trials, wherein the congruent and incongruent conditions were randomly presented, with each condition comprising 36 trials. In the congruent condition, both tactile and auditory stimuli were presented with the same stickiness intensity, whereas in the incongruent condition, they were presented with different stickiness intensities. For the congruent condition, participants encountered two pairs of stimuli with matching intensities, repeated 18 times (2 intensities × 18 repetitions = 36 trials). In the incongruent condition, there were two combinations of different intensities, repeated 18 times, also totaling 36 trials. Participants were required to assess whether the stickiness intensities presented through tactile and auditory stimuli were congruent or incongruent during each trial.

During the fMRI data acquisition, participants were positioned comfortably with their right arm along the magnet bore and a response box in their left hand. Participants watched a computer screen via a surface-mirror, while auditory stimuli and ear protection were provided through an MRI-compatible Headphone (OptoACTIVE Slim Optical ANC Headphones, Optoacoustics Ltd., Israel) that offered both passive attenuation of 25 dB and active noise canceling. The experiment consisted of four runs, encompassing both tactile and auditory sensory modalities in each run. Each run comprised 36 trials, beginning with a fixation cross displayed for 6 to 8 s, followed by a stimulation period involving touching for 2 s and detaching for 1.5 s. During the runs, participants were exposed to both tactile and auditory stimuli. They touched the sticky stimulus when visually instructed and listened to the audio clips at the same time. In each run, participants were presented with 18 congruent and 18 incongruent stimuli, ensuring a balanced exposure to both types of sensory matches across the trials. After each stimulus presentation, participants pressed a button not to evaluate the stimulus, but simply to confirm their attention to the experiment.

fMRI data acquisition and preprocessing

Functional MRI experiments were performed using a 3T MRI scanner (Magnetom TrioTim, Siemens Medical Systems, Erlangen, Germany) with a standard 24-channel head coil. Functional images were acquired using BOLD sensitive gradient-echo-based echo planar image (GE-EPI; TR = 2000 ms, TE = 35 ms, Flip angle = 90°, FOV = 200 mm, slice thickness = 2 mm, and in-plain resolution = 2 × 2 mm) with 72 slices that cover the whole cerebrum. To obtain T1-weighted anatomical images from each participant, a 3D magnetization-prepared gradient-echo (MPRAGR) sequence was used (TR = 2300 ms, TE = 2.28 ms, Flip angle = 8°, FOV = 256 mm, Slice thickness = 1 mm, and in-plain resolution = 1 × 1 mm). The preprocessing and statistical analysis of fMRI data were performed using SPM12 (Wellcome Department of Imaging Neuroscience, UCL, London, UK) and a high-pass filter of 128 s was used to eliminate low-frequency noise. The EPI data were realigned for motion correction, coregistered to the individual T1-weighted images, normalized into the Montreal Neurological Institute (MNI) space, and spatially smoothed by a 4 mm full-width-half-maximum (FWHM) Gaussian Kernel.

fMRI searchlight analysis

First, we extracted parameter estimates of voxel responses using a General Linear Model. We considered the moments of ‘Detach’ as events, which were convolved with the canonical hemodynamic response function from SPM12. To increase the number of examples, we modeled each trial as an individual regressor, resulting in 144 event-related regressors (congruent/incongruent conditions × 18 repetitions × 4 fMRI runs) for predicting voxel responses in each condition. These trial-specific parameter estimates were normalized for centering relative to the mean and achieving unit variance. Subsequently, they were used as input features for a multivoxel pattern analysis conducted using the SearchMight Toolbox [13].

For the searchlight analysis, we used a 5 × 5 × 5 voxel cube that scanned the entire brain. In each searchlight, a Gaussian Naïve Bayes classifier was used to decode the congruent and incongruent conditions from the neural activity patterns. Each experimental run was considered as one-fold and a 4-fold cross-validation procedure was employed. The classification performances from each fold were averaged and assigned to the central voxel of each searchlight. To obtain deviations from chance, we subtracted the chance-level accuracy (0.5 in this case) from the values in each voxel. The P-value for each voxel was then calculated by comparing the classification accuracy against this null distribution. These individual accuracy maps were then used in a random-effects group analysis. To identify significant clusters while correcting for multiple comparisons, we established an empirical cluster size threshold based on searchlight accuracy maps from a randomly chosen sub-group of participants [14].

Results

The results of the behavioral experiment showed that the accuracy was 97.1 ± 3.2%, presented as mean ± SD, under congruent conditions and 95.0 ± 4.4% under incongruent conditions (Fig. 1). As a result of the t-test, there was no difference in accuracy according to congruency (P > 0.05). Regarding the effect size, Cohen’s d values were computed between the congruency. The effect sizes were interpreted as follows: 0.20 = small, 0.50 = medium, and 0.80 = large [15]. The effect size for the difference in accuracy between the congruent and incongruent conditions, as measured by Cohen’s d, was 0.0185, indicating a negligible difference between the two conditions. Response time was 576.6 ± 54.6 ms under congruent condition and 517.8 ± 42.5 ms under incongruent condition. As a result of the t-test, there was no difference in response time according to congruency (P > 0.05). The effect size for the difference in reaction time between the congruent and incongruent conditions, as measured by Cohen’s d, was 0.0658.

Fig. 1:

Behavioral experiment results. (a) displays the accuracies for congruent and incongruent conditions, and bars are with error bars indicating standard deviations. (b) shows the response time in milliseconds.

The purpose of the searchlight analysis was to identify brain regions exhibiting the most distinct voxel activation patterns in response to congruent and incongruent conditions. Our searchlight analysis found a significant cluster in the superior temporal gyrus (STG) in the right hemisphere (P < 0.01 uncorrected, size > 50, Fig. 2). Decoding accuracies obtained from the identified cluster were 67.8 ± 6.1%. It should be noted that the likelihood of identifying voxel clusters through our searchlight analysis by chance is extremely low [14]. According to a permutation procedure, the chance of randomly obtaining clusters of the same size as ours is less than 5%.

Fig. 2:

Searchlight analysis identified a brain region exhibiting distinct voxel activation patterns in response to congruent and incongruent stickiness intensities (P < 0.01 uncorrected, size >50). Entry in italics indicates sub-peak within the cluster.

Discussion

This study investigated both behavioral and neural responses when stickiness intensity information were perceived simultaneously through tactile and auditory channels. The behavioral results indicated that participants accurately perceived the congruency in stickiness intensities, and no statistical difference was observed between the conditions. This implies that participants were able to accurately distinguish between the congruent and incongruent conditions, regardless of the sensory channel through which the tactile and auditory stimuli were presented [16,17].

The searchlight analysis aimed at identifying brain regions exhibiting distinct voxel activation patterns in response to congruent and incongruent stickiness intensities. As a result, we observed distinctive multivoxel patterns in the right STG. Known primarily as an auditory processing area, the STG has been implicated in handling various auditory stimuli such as music and speech [18,19]. Previous research indicates that the STG plays a key role in multisensory information processing, especially in synchronizing audio and visual stimuli [20,21]. This function of the STG affects both the superior temporal sulcus and the primary sensory areas of the brain. A notable point in our findings is the absence of significant activation in regions typically associated with somatosensory or motor functions, despite stickiness being a surface texture property. This raises the question: Why is the STG, not the somatosensory or motor-related areas, involved in processing the multisensory information of stickiness intensity? One possible explanation could be that we are not commonly accustomed to perceiving stickiness through auditory cues [7]. Since we do not frequently experience stickiness through sound, this might be a relatively unfamiliar experience. This unfamiliarity could result in more salient activation in the auditory cortex, as the brain may allocate more resources to process this uncommon sensory information. Another possible explanation could be inline with the findings of our previous behavioral study [11]. This study reported that the discriminability of stickiness intensity is best represented through auditory cues and that the processing of texture information in the auditory domain is distinctive from other modalities. Therefore, our present results suggest that auditory cues play a crucial role in discerning stickiness intensity, and the neural representation of texture information through sound is more distinctive than other sensory modalities [22].

The use of standard EPI sequences, rather than those specifically adapted for auditory processing, may be a potential limitation. The inherent loudness of these sequences could potentially interfere with auditory perception, impacting the accuracy of our findings. Acknowledging this, our future research would consider employing specialized EPI sequences, as investigated by earlier literature [23,24]. This approach could enhance the overall reliability of findings in auditory perception studies.

Combining the results from the behavioral and fMRI experiments, it can be concluded that the perception of stickiness involves complex multisensory integration. Behavioral results reported high accuracy in perceiving stickiness intensity, regardless of congruency between tactile and auditory stimuli, indicating robust sensory processing. The fMRI results highlighted the significant role of the STG, even for a surface texture property like congruency of stickiness intensity. This suggests that the brain’s interpretation of sensory information is not limited to traditional modality-specific areas but includes significant cross-modal interactions, enhancing our understanding of sensory perception and neural processing.

Acknowledgements

The present research has been conducted by the Research Grant of Kwangwoon University in 2023 and by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2021R1F1A1055814, RS-2023-00279315).

I-S.L. and J.K. performed experiments, analyzed data, and wrote the original draft of the manuscript. J-H.K. revised the manuscript.

Conflicts of interest

There are no conflicts of interest.

References 1. Lederman SJ, Klatzky RL. Haptic perception: a tutorial. Atten Percept Psychophys 2009; 71:1439–1459. 2. Ujitoko Y, Ban Y, Hirota K. Modulating fine roughness perception of vibrotactile textured surface using pseudo-haptic effect. IEEE Trans Vis Comput Graph 2019; 25:1981–1990. 3. Kappers AML, Tiest WMB. Visual and haptic perception of roughness. Perception 2005; 34:45–46. 4. Tiest WMB, Kappers AML. Haptic and visual perception of roughness. Acta Psychol 2007; 124:177–189. 5. Heller MA. Visual and tactual texture perception: intersensory cooperation. Percept Psychophys 1982; 31:339–344. 6. Bresciani JP, Ernst MO, Drewing K, Bouyer G, Maury V, Kheddar A. Feeling what you hear: auditory signals can modulate tactile tap perception. Exp Brain Res 2005; 162:172–180. 7. Lederman SJ. Auditory texture perception. Perception 1979; 8:93–103. 8. Kayser C, Petkov CI, Augath M, Logothetis NK. Integration of touch and sound in auditory cortex. Neuron 2005; 48:373–384. 9. Schurmann M, Caetano G, Hlushchuk Y, Jousmaki V, Hari R. Touch activates human auditory cortex. Neuroimage 2006; 30:1325–1331. 10. Roder B, Rosler F, Neville HJ. Auditory memory in congenitally blind adults: a behavioral-electrophysiological investigation. Brain Res Cogn Brain Res 2001; 11:289–303. 11. Lee H, Lee E, Jung J, Kim J. Surface stickiness perception by auditory, tactile, and visual cues. Front Psychol 2019; 10:2135. 12. Mith S, Carpentier L, Monteil G. Correlation between the sensorial perception and the descriptive instrumental analysis of the tackiness of silicone elastomers. Surf Interface Anal 2008; 40:142–145. 13. Pereira F, Botvinick M. Information mapping with pattern classifiers: a comparative study. Neuroimage 2011; 56:476–496. 14. Oosterhof NN, Wiggett AJ, Diedrichsen J, Tipper SP, Downing PE. Surface-based information mapping reveals crossmodal vision-action representations in human parietal and occipitotemporal cortex. J Neurophysiol 2010; 104:1077–1089. 15. Cohen J. Statistical power analysis for the behavioral sciences. Academic Press; 2013. 16. Kitagawa N, Spence C. Audiotactile multisensory interactions in human information processing. Jpn Psychol Res 2006; 48:158–173. 17. Guest S, Catmur C, Lloyd D, Spence C. Audiotactile interactions in roughness perception. Exp Brain Res 2002; 146:161–171. 18. Okada K, Venezia JH, Matchin W, Saberi K, Hickok G. An fMRI study of audiovisual speech perception reveals multisensory interactions in auditory cortex. PLoS One 2013; 8:e68959. 19. Ghinst MV, Bourguignon M, de Beeck MO, Wens V, Marty B, Hassid S, et al. Left superior temporal gyrus is coupled to attended speech in a cocktail-party auditory scene. J Neurosci 2016; 36:1596–1606. 20. Schnell K, Heekeren K, Schnitker R, Daumann J, Weber J, Hesselmann V, et al. An fMRI approach to particularize the frontoparietal network for visuomotor action monitoring: detection of incongruence between test subjects’ actions and resulting perceptions. Neuroimage 2007; 34:332–341. 21. Noesselt T, Rieger JW, Schoenfeld MA, Kanowski M, Hinrichs H, Heinze HJ, et al. Audiovisual temporal correspondence modulates human multisensory superior temporal sulcus plus primary sensory cortices. J Neurosci 2007; 27:11431–11441. 22. Bensmaia S. Tactile intensity and population codes. Behav Brain Res 2008; 190:165–173. 23. Shah NJ, Steinhoff S, Mirzazade S, Zafiris O, Grosse-Ruyken ML, Jancke L, et al. The effect of sequence repeat time on auditory cortex stimulation during phonetic discrimination. Neuroimage 2000; 12:100–108. 24. Bunzeck N, Wuestenberg T, Lutz K, Heinze HJ, Jancke L. Scanning silence: mental imagery of complex sounds. Neuroimage 2005; 26:1119–1127.