Brain-inspired multisensory integration neural network for cross-modal recognition through spatiotemporal dynamics and deep learning

Alais D, Newell F, Mamassian P (2010) Multisensory processing in review: from physiology to behaviour. See Perceiv 23(1):3–38. https://doi.org/10.1163/187847510X488603

Article  Google Scholar 

Alvarado JC, Vaughan JW, Stanford TR, Stein BE (2007) Multisensory versus unisensory integration: contrasting modes in the superior colliculus. J Neurophysiol 97(5):3193–3205. https://doi.org/10.1152/jn.00018.2007

Article  Google Scholar 

Aponte DA, Handy G, Kline AM, Tsukano H, Doiron B, Kato HK (2021) Recurrent network dynamics shape direction selectivity in primary auditory cortex. Nat Commun 12(1):314. https://doi.org/10.1038/s41467-020-20590-6

Article  CAS  Google Scholar 

Barak O (2017) Recurrent neural networks as versatile tools of neuroscience research. Curr Opin Neurobiol 46:1–6. https://doi.org/10.1016/j.conb.2017.06.003

Article  CAS  Google Scholar 

Bi Z, Zhou C (2020) Understanding the computation of time using neural network models. Proc Natl Acad Sci 117(19):10530–10540. https://doi.org/10.1073/pnas.1921609117

Article  CAS  Google Scholar 

Bolognini N, Maravita A (2007) Proprioceptive alignment of visual and somatosensory maps in the posterior parietal cortex. Curr Biol 17(21):1890–1895. https://doi.org/10.1016/j.cub.2007.09.057

Article  CAS  Google Scholar 

Christian S, Wei L, Yangqing J, Pierre S, Scott R, Dragomir A, Dumitru E, Vincent V, Andrew R (2015) Going deeper with convolutions. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), 1–9. https://doi.org/10.1109/CVPR.2015.7298594

Driver J, Noesselt T (2008) Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron 57(1):11–23. https://doi.org/10.1016/j.neuron.2007.12.013

Article  CAS  Google Scholar 

Erkaymaza O, Ozerb M, Perc M (2017) Performance of small-world feedforward neural networks for the diagnosis of diabetes. Appl Math Comput 311:22–28. https://doi.org/10.1016/j.amc.2017.05.010

Article  Google Scholar 

Ghazizadeh E, Ching S (2021) Slow manifolds within network dynamics encode working memory efficiently and robustly. Plos Comput Biol 17(9):e1009366. https://doi.org/10.1371/journal.pcbi.1009366

Article  CAS  Google Scholar 

Goudar V, Buonomano DV (2018) Encoding sensory and motor patterns as time-invariant trajectories in recurrent neural networks. Elife. https://doi.org/10.7554/eLife.31134

Article  Google Scholar 

Guclu U, van Gerven MAJ (2015) Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J Neurosci 35(27):10005–10014. https://doi.org/10.1523/JNEUROSCI.5023-14.2015

Article  CAS  Google Scholar 

Holmes NP, Spence C (2005) Multisensory integration: space, time and superadditivity. Curr Biol 15(18):R762–R764. https://doi.org/10.1016/j.cub.2005.08.058

Article  CAS  Google Scholar 

Hubel DH, Wiesel TN (1962) Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J Physiol 160(1):106–154. https://doi.org/10.1113/jphysiol.1962.sp006837

Article  CAS  Google Scholar 

Kell AJE, Yamins DLK, Shook EN, Norman-Haignere SV, McDermott JH (2018) A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron 98(3):630–644. https://doi.org/10.1016/j.neuron.2018.03.044

Article  CAS  Google Scholar 

Knöpfel T, Sweeney Y, Radulescu CI, Zabouri N, Doostdar N, Clopath C, Barnes SJ (2019) Audio-visual experience strengthens multisensory assemblies in adult mouse visual cortex. Nat Commun 10(1):5684. https://doi.org/10.1038/s41467-019-13607-2

Article  CAS  Google Scholar 

Kravitz DJ, Saleem KS, Baker CI, Ungerleider LG, Mishkin M (2013) The ventral visual pathway: an expanded neural framework for the processing of object quality. Trends Cogn Sci 17(1):26–49. https://doi.org/10.1016/j.tics.2012.10.011

Article  Google Scholar 

Kriegeskorte N (2015) Deep neural networks: a new framework for modeling biological vision and brain information processing. Annu Rev vis Sci 1(1):417–446. https://doi.org/10.1146/annurev-vision-082114-035447

Article  Google Scholar 

Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun Acm 60(6):84–90. https://doi.org/10.1145/3065386

Article  Google Scholar 

Lou J, Zuo D, Zhang Z, Liu H (2021) Violence recognition based on auditory-visual fusion of autoencoder mapping. Electronics 10(21):2654. https://doi.org/10.3390/electronics10212654

Article  Google Scholar 

Ma C, Yang J, Wang Q, Liu H, Xu H, Ding T, Yang J (2022) A method of feature fusion and dimension reduction for knee joint pathology screening and separability evaluation criteria. Comput Meth Prog Bio 224:106992. https://doi.org/10.1016/j.cmpb.2022.106992

Article  Google Scholar 

Mante V, Sussillo D, Shenoy KV, Newsome WT (2013) Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature 503(7474):78–84. https://doi.org/10.1038/nature12742

Article  CAS  Google Scholar 

Mastrogiuseppe F, Ostojic S (2018) Linking connectivity, dynamics, and computations in low-rank recurrent neural networks. Neuron 99(3):609–623. https://doi.org/10.1016/j.neuron.2018.07.003

Article  CAS  Google Scholar 

Miller P, Brody CD, Romo R, Wang X (2005) A recurrent network model of somatosensory parametric working memory in the prefrontal cortex. Cereb Cortex 15(5):679. https://doi.org/10.1093/cercor/bhi102

Article  Google Scholar 

Orhan AE, Ma WJ (2019) A diverse range of factors affect the nature of neural representations underlying short-term memory. Nat Neurosci 22(2):275–283. https://doi.org/10.1038/s41593-018-0314-y

Article  CAS  Google Scholar 

Pollock E, Jazayeri M (2020) Engineering recurrent neural networks from task-relevant manifolds and dynamics. Plos Comput Biol 16(8):e1008128. https://doi.org/10.1371/journal.pcbi.1008128

Article  CAS  Google Scholar 

Rao AR (2018) An oscillatory neural network model that demonstrates the benefits of multisensory learning. Cogn Neurodynamics 12(5):481–499. https://doi.org/10.1007/s11571-018-9489-x

Article  Google Scholar 

Remington ED, Narain D, Hosseini EA, Jazayeri M (2018) Flexible sensorimotor computations through rapid reconfiguration of cortical dynamics. Neuron 98(5):1005–1019. https://doi.org/10.1016/j.neuron.2018.05.020

Article  CAS  Google Scholar 

Riesenhuber M, Poggio T (1999) Hierarchical models of object recognition in cortex. Nat Neurosci 2(11):1019–1025. https://doi.org/10.1038/14819

Article  CAS  Google Scholar 

Senkowski D, Talsma D, Grigutsch M, Herrmann CS, Woldorff MG (2007) Good times for multisensory integration: effects of the precision of temporal synchrony as revealed by gamma-band oscillations. Neuropsychologia 45(3):561–571. https://doi.org/10.1016/j.neuropsychologia.2006.01.013

Article  Google Scholar 

Serre T, Oliva A, Poggio T (2007) A feedforward architecture accounts for rapid categorization. Proc Natl Acad Sci 104(15):6424–6429. https://doi.org/10.1073/pnas.0700622104

Article  CAS  Google Scholar 

Shi J, Tripp B, Shea-Brown E, Mihalas S, Buice MA (2022) Mousenet: a biologically constrained convolutional neural network model for the mouse visual cortex. Plos Comput Biol 18(9):e1010427. https://doi.org/10.1371/journal.pcbi.1010427

Article  CAS  Google Scholar 

Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International conference on learning representations.1409–1556. https://doi.org/10.48550/arXiv.1409.1556.

Song HF, Yang GR, Wang X (2016) Training excitatory-inhibitory recurrent neural networks for cognitive tasks: a simple and flexible framework. Plos Comput Biol 12(2):e1004792. https://doi.org/10.1371/journal.pcbi.1004792

Article  CAS  Google Scholar 

Spoerer CJ, McClure P, Kriegeskorte N (2017) Recurrent convolutional neural networks: a better model of biological object recognition. Front Psychol 8:1551. https://doi.org/10.3389/fpsyg.2017.01551

Article  Google Scholar 

Stein BE, Stanford TR, Rowland BA (2009) The neural basis of multisensory integration in the midbrain: its organization and maturation. Hearing Res 258(1):4–15. https://doi.org/10.1016/j.heares.2009.03.012

Article  Google Scholar 

Surucu M, Isler Y, Perc M, Kara R (2021) Convolutional neural networks predict the onset of paroxysmal atrial fibrillation: theory and applications. Chaos 31(11):113119. https://doi.org/10.1063/5.0069272

Article  CAS  Google Scholar 

Sussillo D, Barak O (2013) Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks. Neural Comput 25(3):626–649. https://doi.org/10.1162/NECO_a_00409

Article  Google Scholar 

Sussillo D, Churchland MM, Kaufman MT, Shenoy KV (2015) A neural network that finds a naturalistic solution for the production of muscle activity. Nat Neurosci 18(7):1025–1033. https://doi.org/10.1038/nn.4042

Article  CAS  Google Scholar 

Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), 2818–2826. https://doi.org/10.1109/CVPR.2016.308

Tan H, Zhou Y, Tao Q, Rosen J, van Dijken S (2021) Bioinspired multisensory neural network with crossmodal integration and recognition. Nat Commun. https://doi.org/10.1038/s41467-021-21404-z

Article  Google Scholar 

Tripp B (2019) Approximating the architecture of visual cortex in a convolutional network. Neural Comput 31(8):1551–1591. https://doi.org/10.1162/neco_a_01211

Article  Google Scholar 

Wada Y, Kitagawa N, Noguchi K (2003) Audio–visual integration in temporal perception. Int J Psychophysiol 50(1):117–124. https://doi.org/10.1016/S0167-8760(03)00128-4

Article  Google Scholar 

Wang WY, Hu L, Valentini E, Xie XB, Cui HY, Hu Y (2012) Dynamic characteristics of multisensory facilitation and inhibition. Cogn Neurodyn 6(5):409–419. https://doi.org/10.1007/s11571-012-9197-x

Article  CAS  Google Scholar 

Wang J, Narain D, Hosseini EA, Jazayeri M (2018) Flexible timing by temporal scaling of cortical responses. Nat Neurosci 21(1):102–110. https://doi.org/10.1038/s41593-017-0028-6

Article  CAS  Google Scholar 

Wyatte D, Curran T, O’Reilly R (2012) The limits of feedforward vision: recurrent processing promotes robust object recognition when objects are degraded. J Cognitive Neurosci 24(11):2248–2261. https://doi.org/10.1162/jocn_a_00282

Article  Google Scholar 

Yang C, Lin C (2013) Coherent activity between auditory and visual modalities during the induction of peacefulness. Cogn Neurodynamics 7(4):301–309. https://doi.org/10.1007/s11571-012-9234-9

Article  Google Scholar 

留言 (0)

沒有登入
gif