1 Introduction

Imagination is the ability to generate patterns that differ from reality, utilizing representational elements that correspond to elements in the environment acquired through experience. For intellectual systems, imagination is crucial, as it is necessary for generating hypotheses for situations that cannot be directly experienced. Furthermore, imagination forms the foundation of all meaning, understanding, and inference (Johnson, 2013). It is believed that the imaginative capabilities present in modern humans were acquired during the “Great Leap Forward,” approximately 70,000 years ago (Diamond, 1992). Given that imagination is a core function in intelligence, designing an architecture for imagination is highly beneficial for developing brain-like artificial intelligence and for the computational understanding of the brain. Here, the term “architecture” refers to a description of a system in which multiple computationally meaningful components are organically connected, thereby forming the design information for the software.

Since the beginning of this century, cognitive pathology research has progressed, particularly concerning imagination. For example, the lack of creativity in children with autism and Asperger's syndrome has been studied (Craig and Baron-Cohen, 1999). Additionally, a comprehensive analysis of pathological findings suggests that social imagination, particularly as embodied in the default mode network of the human brain, may mediate representations along the opposite dimensions of autism and risk for psychosis/emotional disorders (Crespi et al., 2016). Several types of studies measuring brain neural activity have been conducted. EEG analysis has shown that brain waves appear to be specific to visual imagination and perception, especially in the alpha frequency band (Xie et al., 2020). An fMRI study capturing the default mode network of human imagination of the future demonstrated that the medial temporal lobe responds to the vividness of the imagined event, whereas the dorsal or core Default Mode Network responds to its semantic depth (Lee et al., 2021). When a motor schema is established by motor imagery learning, interregional connectivity in the occipital lobe may be significantly reduced, as demonstrated by fMRI studies (Zhang et al., 2012). While such studies do not directly propose an architecture, they can provide insight into its design. From a human-focused cognitive neuroscience approach, some studies conceptualize the function of imagination by estimating the role of brain organs associated with mental disorders. Imagination has been conceptualized as an important mediator between acquired knowledge and creative insight, constraining potential solutions through mental simulation or “incubation” (Duch, 2007). Moreover, visual imagery has been hypothesized to involve a network of brain regions from the frontal to the sensory cortex, and imagination has been proposed to function similarly to a weaker version of afferent perception (Pearson, 2019). However, these studies offer primarily phenomenological explanations and are somewhat distant from architectural design; moreover, they did not consider brain organs associated with imagination other than the neocortex and hippocampus.

Agnati et al. (2013) proposed a hypothesis on the mechanisms involved in the executive function of imagination; however, this study focused only on neocortical mechanisms. Based on their considerations, Marques and Holland (2009) defined an architecture that realizes the conditions for functions performed by the imagination and implemented it in a humanoid simulated robot and a real robot. This study focuses primarily on the neocortex and briefly describes the hippocampus. As described above, studies of architectures of imagination tend to focus solely on the neocortex, with little consideration of other connected brain organs and their functional roles.

Conversely, a research area related to imagination is the study of consciousness in computational neuroscience, as described below. Many studies relate phenomenal features of neural activity to the anatomical structure of brain regions, as follows: neural field theories (Kinsbourne, 1988) assume that consciousness is a characteristic of a specific pattern of neuron firing. The field model (Pockett, 2000; McFadden, 2002) assumes that consciousness is related to the electromagnetic field of the brain. The thalamocortical rhythms model (Llinás et al., 1998) assumes that neural activity phenomena in the thalamocortical loop create the state of consciousness. Interconnected neural networks (Crick and Koch, 2003) assume that “coalitions” of neurons determine the content of consciousness. In theories such as the information integration theory (Tononi, 2004), information closure theory (Chang et al., 2019), and higher-order thinking theory (Rosenthal, 1986), the aforementioned core function is considered to be realized by the coordination of the neocortex and thalamus. Global workspace theory describes consciousness as a process that leads information, selected by attention from the realm of unconsciousness, into the global workspace and flexibly processes and retains it (Baars, 1993). Moreover, the global workspace dynamics theory (Baars et al., 2013) incorporates the idea of the dynamic core hypothesis (Edelman and Tononi, 2000), which corresponds to the functions of the neocortex and thalamus. Additionally, Francis Crick posited that the claustrum is the neural foundation of consciousness and plays the role of a conductor in the orchestra of the neocortex (Crick and Koch, 2005; Stevens, 2005).

As noted earlier, the research on imagination and consciousness has been centered on the phenomena and computational functions associated with the neocortex. The neocortex, widely regarded as a repository for a diverse array of representations, is theorized to play a pivotal role in the execution of imagination through the integration of these representations. Nonetheless, drawing from previous studies, other brain structures, including the hippocampus, thalamus, and prefrontal cortex, also contribute to these processes. Therefore, designing a comprehensive architecture for imagination incorporating these peripheral brain components is necessary.

A notable approach for designing architecture that emphasizes consistency with the brain's anatomical structure based on neuroscience findings is the Structure-Constrained Interface Decomposition (SCID) method (Yamakawa, 2021). The SCID method is a reverse engineering process, focusing primarily on the brain's anatomical structure by identifying the Regions of Interest (ROI) in the brain related to the function of interest. However, a current challenge of the SCID method is its inapplicability to broad brain regions, due to the requirement of a detailed understanding of the anatomical structure of the ROI down to the mesoscopic level. Imagination is a complex ability that involves a wide range of brain regions; the current SCID method cannot be used to design architecture that adequately supports it.

Therefore, in this study, we aimed to propose a new function-oriented SCID method, comprising four distinct steps, that enables the reverse engineering of a broad spectrum of brain regions. We have also applied this innovative method to elucidate the computational architecture underpinning the imaginative capability of the brain, resulting in a proposed architecture involving five key brain structures.

2 Methods

This section first describes the Brain Reference Architecture (BRA)-driven development approach. Brain Reference Architecture is abbreviated here as BRA. Subsequently, the SCID methodology, which implements the design portion of BRA-driven development, is detailed. Based on this, we propose a function-oriented SCID method that can improve the weaknesses of the conventional SCID method.

2.1 BRA-driven development

Brain Reference Architecture (BRA)-driven development is a methodology conceived to construct brain-like software. This methodology emerged as a response to three challenges previously encountered in the development of brain-inspired software: (Challenge 1) Professionals adept in both neuroscience and software development are extremely rare, and cultivating such interdisciplinary expertise is challenging and complex. (Challenge 2) The vast body of neuroscience knowledge covering the entire brain makes it impractical to rely on an individual's cognitive capacity for designing software that seamlessly integrates all brain functions. (Challenge 3) An appropriate level of granularity has to be selected in the referenced brain descriptions to faithfully represent the brain's cognitive functions in software.

BRA-driven development addresses Challenge 1 by dividing the process into two distinct phases: design of the BRA, which serves as the specification information for brain-inspired software, and implementation of the software according to this BRA. Moreover, this process overcomes Challenge 2 by promoting collaborative work, facilitated through the standardization of the BRA data (Figure 1).

Figure 1. Brain reference architecture-driven development.

BRA is a standardized data format that functions as the specification information for brain-like software. It encompasses data representing the brain's information flow and a hypothetical component diagram articulated in conjunction with this data.

• Brain Information Flow (BIF) data: This data refers to the information flow of the brain's neural circuits at a mesoscopic anatomical level. It is formatted as a directed graph, with diverse granularity “circuits” within the brain represented as nodes and “connections,” corresponding to axon projections between them, depicted as links. Notably, the origins of these connections are constrained to Uniform Circuits.

• Hypothetical Component Diagram (HCD) data: This constitutes a hypothetical component diagram that aligns functions with the anatomical structure of the Top Level Function (TLF) executed by the Region of Interest (ROI) within the BIF. Importantly, an HCD can correspond to any circuit within the BIF, and multiple HCDs can overlap and be associated with a specific circuit in the BIF.

Challenge 3 was resolved by defining the smallest circuit unit to be portrayed in the Binary Interchange Format (BIF) as the Uniform Circuit. The Uniform Circuit is, fundamentally, an ensemble of neurons composed of a specific cell type situated within a designated brain region.

The creation of the BRA, depicted on the left side of Figure 1, adheres to a precise procedure known as the SCID method. This reverse engineering method entails an in-depth analysis of the anatomical structures within an ROI and hypothesizes a functional system that could realize the top-level function that the ROI serves in alignment with its structure. The BRA was evaluated in a phased manner, as illustrated on the right side of Figure 1. The evaluation includes a fidelity assessment, which evaluates the alignment of the software with the HCD; a consistency evaluation, which verifies the inclusion of the structure utilized in the HCD within the BIF structure; and an authenticity evaluation, which verifies whether neuroscience findings substantiate the BIF.

2.2 Conventional SCID method

The SCID method is performed by creating BRA data.

The SCID method involves three steps:

• Step 1. BIF construction: Investigating brain anatomy and neural circuits in the ROI to define BIF.

• Step 2. Alignment of ROI and TLF: Identifying the Top-Level Function (TLF).

• Step 3. HCD design: Candidate component diagrams describing computational functions and interfaces are enumerated from the HCDs, and inappropriate ones are eliminated based on neuroscientific evidence and logical consistency. The remaining ones are HCDs.

As neuroscience has now accumulated anatomical knowledge of a relatively wide range of regions, the SCID method uses this knowledge to design functions, resulting in broad applicability.

This SCID method eases the process of HCD design by detailing the BIF within the ROI of interest to a uniform circuit granularity. However, this approach presents challenges when utilized to reverse engineer extensive regions of the brain's computational functionality. Particularly, generating a BIF for a broad region in the initial step is a complex task. Moreover, even if a BIF is successfully build, deconstructing a high-level abstract TLF into uniform circuit granularity functionalities for a wide range of ROI in the third step is also challenging.

2.3 Proposing the function-oriented SCID method

Based on the aforementioned technical background, a new methodology called the “function-oriented SCID method” was proposed to determine the TLFs of multiple small ROIs from highly abstract functions in a broad ROI.

The proposed methodology consists of the following four steps.

• Step 1. Capability and requirements definition: Define a Broad TLF at a high level of abstraction in an expansive brain region (e.g., whole brain) and decompose it into a set of requirements to realize it.

• Step 2. Function decomposition: Build a set of independent functions that can satisfy all requirements and significant biological constraints.

• Step 3. Architecture design: Design a mechanism to realize each function as a component that defines input/output signals with semantics. Ensure that the created components work cohesively to meet all requirements.

• Step 4. Brain region mapping: Map the above components to reasonable brain regions concerning input/output signal semantics and internal processing feasibility. However, if no appropriate mapping is found, return to Step 2 and revisit the decomposition of the function.

To facilitate Step 2, TLF specifications and biological constraints are considered to narrow down the candidate functional decompositions.

In step 4, the input and output signals required for each component are considered during the solution to see if the inputs and outputs can be mapped to candidate brain regions, considering the semantics of the signals. The feasibility of internal processing of the component in the candidate brain regions is also assessed if possible. If, after these considerations, a consistent mapping is possible, the brain region is mapped to the component.

As mentioned above, the process advances through Steps 2, 3, and 4. If mapping to brain structures fails during Step 4, the process reverts to Step 2 (see Figure 2). To optimize efficiency, it's crucial to minimize such revisions. Consequently, it's advisable to perform functional decomposition in Step 2 while anticipating the subsequent mapping to brain structures as much as possible.

Figure 2. Function-oriented SCID method and its application for imagination capability.

Therefore, for each specific component, the TLF is defined in Step 2, and the ROI is identified in Step 4. By applying the conventional SCID method to these ROI/TLF pairs, a hypothesis of the computational function of the brain can be constructed at a more detailed level within the ROI.

3 Applying the function-oriented SCID method for imagination

In this section, we apply the functionally oriented SCID method to the brain's imagination according to the four steps of this method.

A comprehensive illustration of the approach is shown in Figure 1. In Section 3.1, as the first step, we set the TLF in imagination while considering the treatment of modalities and decompose it into six requirements. In Section 3.2, as the second step, we organize the TLFs into six functionalities based on the requirement specifications, considering the biological constraints for the neocortex as distributed memory. In Section 3.3, as the third step, we construct an architecture for imagination by designing five components to realize the functions. In Section 3.4, as the fourth step, we propose an architecture for creativity in the brain by mapping each component to a brain region.

3.1 Step 1. Capability and requirements definition: define imagination capability 3.1.1 Modes that govern imagination

Representational patterns that humans can obtain directly from the environment involve the contexts wherein their own experience occurs. In contrast, various imaginary contexts deviate from them in some respect. For instance, recalling a previous memory is associated with the context of “the past.” In the case of a newly generated pattern that is made from a combination of representations, its context is not self-explanatory. However, if this pattern represents a prediction, it assumes the context of “the future.” If it guides subsequent actions, it is regarded as an “intention.” Here, “intention” refers to the mental process of planning a specific course of action (Bratman and Others, 1987).

Modal logic (Garson, 2018) is a well-known theoretical background for understanding how contexts can modify the patterns they encompass. Based on modal logic, we defined a context that modifies the entirety of an imagined pattern as a “mode.” Individuals frequently engage in social reasoning by adopting the perspectives of others, although modal logic in the realm of artificial intelligence does not usually account for this. Therefore, in this study, the mental states of others other than the self shall also be treated as “modes” of some form.

The treatment of modals in imagination has received much attention in philosophy. Still, much of the work seems to focus on epistemological arguments regarding the range of beliefs that are legitimate to imagine (Kung, 2010; Ichikawa and Jarvis, 2012).

For higher animals, including humans, the ability to infer others' beliefs and intentions is a critical requirement for survival in a society consisting of members of the same species. As noted in the study of the mind theory, humans acquire the ability to estimate the beliefs and intentions of others in early childhood. It is also known that the brain contains mirror neurons (di Pellegrino et al., 1992) which react to the actions of others as if they were one's own actions.

In order to exert imaginative abilities, modes are indispensable for distinguishing the present, the past and the future, the self and the others, beliefs, and intentions. This idea has been debated in the field of philosophy (Nichols, 2006; Kind, 2016), but has been largely overlooked in both neural processing technologies and computational neuroscience to date.

From our perspective, the following provides a more specific explanation of the necessity for apprehending and managing modes regarding imagined patterns.

First, the process of imagining differs depending on the type of mode. For instance, future prediction would manipulate representations from the present to the future, inference about the past would manipulate representations by reversing causation from the present to the past, and performing obligatory actions would manipulate the future image with the goal of realizing a state wherein the obligation has been fulfilled. Second, the manner of utilizing patterns depends on their mode. For instance, if an agent conceives a pattern in the mode of intention about its own future, that pattern should be acted on when an appropriate circumstance is found. Conversely, if the agent conceives a pattern that involves the mode of prohibition, which is a kind of obligation that dictates not to do something, that pattern should deter action, regardless of circumstances. Third, when verbalizing imagined patterns, their modes affect the conjugation of auxiliary verbs and main verbs. Fourth, storing a pattern must be paired with the current mode state. A pattern being restored at a later time with an incorrect mode would cause a situation corresponding to the symptom of memory confusion.

3.1.2 Definition of imagination

Imagination has been studied across various fields, including neuroscience, philosophy, psychology, and artificial intelligence. Each discipline has approached the concept of imagination from its unique perspective, leading to a diverse array of definitions. These definitions often reflect the specific focus and methodologies employed within each field. Neuroscientists, for instance, may emphasize the neural mechanisms underlying the formation of mental images, while philosophers might explore the representational nature of imagination and its relation to other cognitive processes. Psychologists, on the other hand, could focus on the role of imagination in mental simulation and its impact on behavior and decision-making. Researchers in artificial intelligence may view imagination as a crucial component in creating intelligent systems capable of generating novel ideas and solutions. The following list presents a selection of definitions that showcase the breadth of perspectives on imagination:

• To imagine is to represent without aiming at things as they actually, presently, and subjectively are. One can use imagination to represent possibilities other than the actual, to represent times other than the present, and to represent perspectives other than one's own. Unlike perceiving and believing, imagining something does not require one to consider that something to be the case. Unlike desiring or anticipating, imagining something does not require one to wish or expect that something to be the case (Liao and Gendler, 2020).

• Imagination is the production or simulation of novel objects, sensations, and ideas in the mind without any immediate input of the senses (Wikipedia contributors, 2023).

• Imagination, considered as “forming new ideas, mental images, or concepts,” represents a central facet of human social evolution and cognition (Crespi, 2016).

• Imagination may depend on separate neural networks involved in the construction and evaluation of imagined future events (Lee et al., 2021).

• What is the human imagination? What is this amazing ability, which most of us have, that allows us to travel through space and time, testing out different virtual worlds, objects, foods, fears, and pleasures (Pearson, 2019)?

• Imagination: the act or power of forming mental images of what is not actually present or has never been actually directly experienced. Notably, imagination not only has the potential to enrich the meaning of an experience and deepen understanding by multiplying and expanding the perspectives from which a phenomenon can be considered, but it also allows anticipating the outcome of an action without actually performing it via a “simulation” process. At its peak, imagination is the very mental faculty underlying visionary and creative thought (Agnati et al., 2013).

• Imagination can be defined broadly as the manipulation of information that is not directly available to an agent's sensors (Marques and Holland, 2009).

• Imagination involves episodic memory retrieval, visualization, mental simulation, spatial navigation, and future thinking, making it a complex cognitive construct (Jung et al., 2016).

Based on the above definitions, the common denominator of imagination is the “capability to generate new patterns by manipulating the representation,” which is also used in this paper. However, the manipulation must be mode-dependent, as previously described (Section 3.1.1); therefore, “depending on various modes” is added. Furthermore, as this study deals with the architecture of the imagination, it is necessary to consider the peripheral capabilities along with the core capability of the imagination. The peripheral capability of imagination includes the capability to store and reuse the results of imagination and reflect them in real-time actions. For this reason, we added “allowing them to influence real-time actions and save and reuse them” to the definition.

Based the previous considerations, this paper defines imagination as follows.

Definition of Human Imagination:

The capability to generate new patterns by manipulating the representation depending on various modes, allowing them to influence real-time actions and save and reuse them.

3.1.3 Requirements of imagination

First, we took the possible requirement specifications from the above definition and organized them into six categories.

• Intervention requirement:

The requirement to make the outcome of an execution (patterns) affect real-time execution (behavioral output to the environment and internal attention control). For example, a plan that is built by imagination has to be put into action at the right time.

• Real-execution requirement:

Apart from imaginative abilities, continuously performing real-time processing grounded in the environment using representations in the distributed memory is essential for maintaining activities as an agent. This requirement is primarily related to the ideomotor theory in psychology (James, 1890; Shin et al., 2010), which posits that unconscious mental images or thoughts can automatically trigger corresponding motor actions or behaviors.

• Flexible imagination requirement:

The requirement to combine representations widely dispersed in the distributed memory to create various patterns flexibly.

• Manipulation requirement:

The requirement to imagine new patterns by manipulating representations in the distributed memory, depending on the types of their mode. Note that the imagined patterns do not directly affect the actions of the intelligent system.

• Modal control requirement:

The requirement to generate a series of imagined patterns under a particular mode.

• Reusability requirement:

The requirement to preserve the outcome of imaginative generation (patterns) and its modes together in order to retrieve and reuse them at a later time.

The previously defined human imaginative capacity is realized through multiple above requirements that operate in concert. Firstly, the aspect of “manipulating the representation depending on various modes” is augmented by two requirements: Mode Control and Manipulation. Mode Control offers the ability to produce a sequence of imaginative patterns under specific conditions or aspects, whereas Manipulation bestows the capability to create new patterns by manipulating representations. Secondly, the notion of “generating new patterns” is underpinned by the requirement for Flexibility of Imagination. This necessitates flexibly combining a wide range of representations stored in distributed memory, thereby generating diverse patterns. Thirdly, the facet of “influencing real-time actions” is embodied by two requirements: Intervention and Real Execution. An intervention allows the generated imaginative patterns to be reflected in real-time external behavior or internal attention control. Real Execution fortifies this aspect by enabling continuous real-time processing as an agent, thereby facilitating interaction with the external world. Lastly, the aspect of “save and reuse them” is complemented by the requirement for Reusability. This involves storing the generated imaginative patterns and their mode information for future reuse.

3.2 Step 2. Functional decomposition: functions for imagination capability

The second step of the function-oriented SCID method involves organizing the above requirement specifications into five functions. In doing so, we introduced a dynamic zone segmentation function, considering the significant biological constraints of the neocortex, which functions as a distributed memory.

3.2.1 The necessity of dynamic zone partitioning function in the neocortex

The neocortex corresponds to distributed memory in the brain and is classified into numerous areas such as Broadman's brain map. Although the neocortex has an almost common mechanism, each area handles different information related to the senses, such as vision and hearing, motor control, language, and thought. In the brain, the ability to flexibly combine information stored in different neocortical areas, as described below in Section 3.4.3, is the key to rich human intellectual abilities. In addition, its function is the executive function of both Real and Virtual, as described above.

To proceed with the discussion, we first point out that the neocortex, as a distributed memory mechanism, has two biological properties that differ from those of distributed memory on the computer.

• Uniqueness of representation :

It may be postulated that individual distributed representations within the neocortex exhibit unique responsiveness characteristics. This phenomenon is attributable to the intrinsic capacity of neurons to modify their response properties through learning processes. The absence of a neurological mechanism for replicating identical response properties across disparate representations further substantiates this assertion. Although neurons situated in proximal local clusters may manifest similar responsiveness when exposed to analogous stimuli, it is advisable to regard representations that are geographically distant from one another as inherently distinct.

• Singleness of modality:

It appears that, to a single neuron or local neural circuit, only one state of mode can be associated at a given point in time. This supposition is based on the intuition that, if there were a group of 100 neurons representing a “cat,” they would be challenging to code, as 80 of them represent a cat that is currently and visibly present, while the remaining 20 neurons represent a cat that was seen yesterday.

The representations in the distributed memory are obtained by Representation and Execution (RAE) and utilized for execution. Since each of these representations in the distributed memory exhibits uniqueness as a characteristic, exerting imaginative abilities requires a mechanism that enables the flexible use of various representations while satisfying the flexible imagination requirement to divert representations in distributed memory. However, since a representation can be associated with only one mode, the only way of diverting its use is to use it at a different time. For this reason, a function that dynamically sets an area for imagination (hereinafter called the imaginary zone) has to be introduced in a distributed memory with properties similar to the neocortex.

The RAE and Manipulation functions are required in the distributed memory's real and imaginary zones. These two areas would correspond to System 1 and System 2 in the double-process theory (Stanovich, 2009) and to the immediate response and contemplation in cognitive architecture.

Patterns on a distributed memory should be treated differently according to their modes (see Section 3.1.1). For this reason, the imaginary zone, to which various modes can be given, needs to be clearly separated from the real zone. The dynamic zone partitioning function accomplishes this separation, satisfying the requirement of mode maintenance, and enabling the imagination of a series of patterns in the imaginary zone under a particular mode.

The following scientific findings also enhance the potential of the proposed idea. There is some evidence that imagination is like a weak form of perception in the neurophysiology involved in vision (Ishai and Sagi, 1995; Pearson et al., 2008; Pearson, 2019). Imagination is also thought to be the capability to reuse the neural structures originally involved in the execution of a function for an image of the virtual execution of that function (Agnati et al., 2013).

Moreover, similar ideas to the dynamic zone partitioning function have been proposed, as seen in the dynamic core hypothesis (Edelman and Tononi, 2000), and in the global workspace dynamics inspired by the former theory (Baars et al., 2013). This function is sometimes referred to as a “cognitive decoupling” in a cognitive science field (Perner, 1991; Tooby and Cosmides, 2000; Nichols and Stich, 2003).

3.2.2 Functions derived from requirements

The functions that realize the six requirements, enumerated above, are identified while taking into account a biological constraint of the neocortex as a distributed memory to reveal that the five functions shown in Figure 1 are necessary.

• Representation and execution (RAE) function:

Functions to represent different types of information. There are sensory representations to execute perception, motor representations to execute control, and higher-order representations to execute thought. These functions can be divided into those that output actions in real-time to the environment and those that are imaginative and do not contribute directly to actions. However, functions that reflect the results of imagination in real-time behavior must also be included to meet intervention requirements.

• Dynamic zone partitioning function:

This function dynamically specifies an imaginary zone in distributed memory and maintains it for a necessary period. An imaginary zone is a subset of the representation for executing imagination on distributed memory. This functionality allows for manipulation and flexible imagination requirements Note that the complementary set of the imaginary zone will be referred to as the real zone.

• Manipulation function:

This function imagines new patterns by manipulating representations on the distributed memory according to a policy appropriate to its type of mode and context to meet the requirements of manipulation and modal control.

• Mode maintenance function:

This function maintains the current mode that affects the sequence control of patterns generated or replayed on the distributed memory to meet the requirements of mode control.

• History management function:

This function records the pattern of activity in distributed memory at a given time as an episode. It restores patterns of activity in distributed memory from episodes as needed. Patterns include mode, context, routing, and designation control.

3.2.3 Requirements realized by functions

Next, it is necessary to verify that all requirements are met by fulfilling the functions derived above. In the following, we will review this point for each of the six requirements.

• Intervention requirement:

This requirement is fulfilled by the “function to reflect the results of imagination in real-time behavior,” which is included in the RAE function.

• Real-execution requirement:

This requirement is achieved by outputting the results calculated by the Manipulation function in the real zone on Distributed Memory, which has a Representation and execution function, partitioned by the Dynamic-zone partitioning function, as the results of real-time processing.

• Flexible imagination requirement:

This requirement is fulfilled by flexibly setting the area on Distributed Memory used for imagination through the role ofthe Dynamic-zone partitioning function.

• Manipulation requirement:

Distributed Memory, designated by the Dynamic-zone partitioning function, through the Manipulation function.

• Modal control requirement:

This requirement is accomplished by generating a pattern by the Manipulation function based on the mode maintained by the Mode maintenance function.

• Reusability requirement:

This requirement is accomplished by the History management function, which stores an index of activity patterns in distributed memory and returns the results to reproduce those patterns in distributed memory as needed.

3.3 Step 3. Architecture design: architecture for imagination

In the third step of our design process, we devised an architecture comprising five key components: routing conductor, mode memory, imaginary zone conductor, recorder, and distributed memory-which corresponds to the neocortex. These components were integrated into a hypothetical architecture, represented as a component diagram that defines the meaning of the signals exchanged between them (see Figure 2). This section substantiates that each function is realized through coordination among these components. The meaning of the signals exchanged between the components will be explained in the received components.

• Distributed memory (and its role in RAE function):

The distributed memory component is a highly versatile, distributed associative storage system that excels at representing and manipulating various patterns. This component outputs real-time action signals in response to environmental sensing signals. The various internal pattern information is distributed to the surrounding components. This component dynamically sets imaginary zones based on area designation signals, changes associative behavior according to transmission channel signals, and reproduces patterns according to pattern index signals.

• Imaginary zone maker (and its role in dynamic zone partitioning function):

The imaginary zone maker is a component that realizes the dynamic zone partitioning function. The designation policy generates area designation signals that specify areas of imaginary zones on distributed memory based on the received designation control signals.

• Routing conductor component (and its role in the manipulation function):

The Routing Conductor component is the mechanism responsible for the manipulation function. The route selection policy chooses and generates the appropriate transmission channel signals from this component's received routing pattern signals. This selection is controlled according to the type of mode obtained from the current mode signal, depending on the context information by the received context signal.

• Mode memory (and its role in mode maintenance):

The mode memory holds the specific mode pattern signal obtained from one of the distributed memory as mode information and transmits it continuously as a current mode signal. This mechanism enables the mode maintenance function.

• Recorder component (and its role in history management):

The recorder component serves as the backbone of the history management function. This component records the episodes as indexes of acquired patterns. These episodes restore and send to distributed memory when needed. Here, pattern signals encompass mode, context, routing, and designation control.

The architecture proposed above shows that the five functions discussed in the previous subsection can be realized mainly by the components corresponding to each of them.

3.4 Step 4. Brain region mapping: imagination architecture in the brain

As the fourth step, we propose a brain-imagination architecture as a finishing touch to the “function-oriented SCID method” by mapping each component to a part or component of the brain. In this study, we envision assigning the neocortex to distributed memory. Therefore, we align appropriate brain regions to Imaginary zone makers, routing conductors, mode memory, and recorder. Assigning a brain region to each component would set the TLF at a relatively small ROI. Therefore, the results can be used for reverse engineering through the conventional SCID method based on the TLFs of each brain region.

Figure 3 shows a schematic representation of the brain regions (mainly the neocortex) treated in this study.

Figure 3. Architecture proposed for imagination composed with six components.

3.4.1 Distributed memory is allocated to a neocortex

The distributed memory allocated to the neocortex as a precondition has the following mechanisms, as described in Section 3.2.1. First, imaginary zones can be allocated according to the obtained area designation signals to realize the dynamic zone partitioning function. In this case, the remaining areas are used as a real zone. The RAE function receives the sensing signals in the real zone, processes them in real-time, and outputs them as actions. In contrast, the manipulation function determines the combination of expressions based on the transmission line signals in the imaginary zone. It also has an intervention function that allows patterns imaged in the imaginary zone to intervene in processing the real zone. To further realize the reuse function, it receives a pattern index signal and reproduces the pattern on expression.

In the following, the mechanism of distributed memory, which corresponds to the structure of the neocortex, will be explained, focusing on the functions it performs.

Dynamic zone partitioning function is realized by a mechanism in the neocortex that blocks the bottom-up observation signals in units smaller than the area. This mechanism involves interneurons exerting an inhibitory effect on layer 4, which receives bottom-up signals. In the Figure 4, the area where the bottom-up signal passes through is indicated by a blue circle, and a red X indicates where it is blocked. In this study, we assumed that the area designation signal to specify the area to be blocked originates from the Claustrum. We hypothesized that when the bottom-up signal is properly blocked in the neocortex, an imaginary zone isolated from observation is formed.

Figure 4. Neocortical schemes and related brain organs: the symbols in parentheses for each brain region are brain region identifiers as defined in the DHBA (Developing Human Brain Atlas) ontology. This study assumes that distributed memory corresponds to the neocortex (ref. Section 3.2.1). The neocortex is located at the center of Figure 3 and consists of many interconnected regions. There are two types of connections between areas: bottom-up pathways that transmit observed signals from the environment to the inside and top-down pathways that transmit signals in the opposite direction.

The RAE function is a neocortical mechanism involving a set of areas where the influx of observation signals other than those formed by the imaginary zone above is not blocked; thus, we refer to this area as the real zone. The real zone is a mechanism that processes Sensing signals from the environment in real-time and outputs the results as Actions to the environment. When performing this processing, the routing conductor mechanism can be used to bind different types of information.

The manipulation function is a mechanism that receives coherent low-frequency oscillation signals, which are transmission channel signals, from multiple domains, and transmits high-frequency oscillations modulated by Phase-Amplitude Coupling between those domains by Cortical Dynamic Routing. This mechanism allows it to imagine various images.

Intervening functions require mechanisms by which imagined patterns in imaginary zones affect processing in real zones. The anatomical projection patterns between laminar structures across domains have not yet been fully elucidated in corticocortical connections. However, there is a large body of knowledge regarding top-down corticocortical connections from higher-order to lower-order domains (Rockland and Pandya, 1979; Felleman and Van Essen, 1991; Barbas et al., 2005; Medalla and Barbas, 2006; Shipp, 2007; Solari and Stoner, 2011; Markov et al., 2014; Barbas, 2015). Thus, assuming that the intervention function is realized through those top-down connections is not unnatural.

The history management function seems to be realized in the hippocampus, as discussed in Section 3.4.5 below. The hippocampus can consolidate relatively higher-order activity patterns from the widely dispersed memories of the neocortex. Moreover, episodic memories stored in the hippocampus are known to be reproduced in the neocortex; however, the specific details of this mechanism are unknown. Therefore, it is reasonable to assume that the neocortex also has a mechanism for reproducing patterns in its representation based on received pattern-index signals.

3.4.2 Imaginary zone maker is allocated to claustrum

The brain ROI corresponding to the imaginary zone maker mechanism can be considered a claustrum. This is because the inputs (designation control signals) and outputs (area designation signals) of the imaginary zone-maker mechanism can be mapped to the inputs and outputs of the claustrum. The output of area designation signals may be able to control the blockage of the aforementioned bottom-up input by projection from pyramidal cells within the claustrum to inhibitory neurons in layer 4 of the neocortex; this occurs in both the sensory and motor cortex. This has been termed the Gain Control Signal of Sensory Processing in previous studies (Goll et al., 2015). It is reasonable to assume that the designated control signal that controls the behavior of the imaginary zone maker is a projection from some higher-order areas. In fact, there is a reciprocal projection from the frontal lobes (Goll et al., 2015), which supports this validity. As a result, because the frontal disabilities correspond to components of the imaginary zone maker, their function is dynamic zone partitioning.

3.4.3 Routing conductor is allocated to the complex of thalamus and basal ganglia

The brain ROI responsible for the routing conductor mechanism is a complex consisting of the basal ganglia and thalamus. This is because the inputs (current mode signals, context signals, and routing pattern signals) and outputs (transfer path signals) of the routing conductor mechanism can be naturally mapped to the inputs and outputs of that complex.

The inputs of the current mode and Context signals correspond to the striatum of the basal ganglia, which can receive information from various areas of the neocortex. The input of the Routing Pattern signal can correspond to the thalamus (matrix-type relay cells), which receives information from various areas of the neocortex. As the core-type relay cells in the thalamus are thought not to be controlled by the basal ganglia, we focused on the matrix-type relay cells. The output of transmission channel signals corresponds to a mechanism in which the thalamus (matrix-type relay cells) sends the same low-frequency oscillatory signal to L1 in the neocortex for the transmission channel it wishes to include in the exchange of information between representations, establishing interregional transmission channels L2/3/4.

The route selection policy within the complex can be considered to be realized by a mechanism in which the internal globus pallidus (GPi)/substantia nigra pars reticulata (SNr) nuclei, the output of reinforcement learning in the basal ganglia, disinhibit the thalamus (matrix relay cells).

From the above, the input/output signals required for the routing conductor mechanism can be semantically mapped to the input/output of a complex consisting of the basal ganglia and thalamus. Therefore, this complex can be considered as the ROI of the brain circuitry responsible for the routing conductor mechanism. The TLF corresponding to this ROI is the manipulation function defined in Step 2.

3.4.4 Mode memory is allocated to a specific agranular neocortical area

In this section, we explain that the ROI responsible for the Mode memory mechanism in the brain is likely to be one of the agranular higher neocortical areas.

The mode memory mechanism refers to the mode pattern signals from the distributed memory to generate, maintain, and send out the current mode signals used to adjust the routing conductor's route selection policy. As previously mentioned in Section 2.1, modes are diverse, including present/past/future, self/other, and belief/intention. In neuroscience, prospective memory, including temporal modes (Owen, 1997; Okuda et al., 1998, 2007; Tanji and Hoshi, 2008; Reynolds et al., 2009) associated with activity in the rostral PFC (A10). Moreover, studies of facial apraxia have shown temporal lobe involvement (Gainotti and Marra, 2011). Given these examples, the modal pattern signals representing the mode are likely to reside in multiple distributed locations in the neocortex. However, the Routing conductor must operate based on a specific mode at a given time, and sustain that operation for a certain period. Thus, the mode storage mechanism focuses on specific mode patterns in the distributed memory and maintains the mode to be processed.

At this stage, identifying the specific neocortical areas corresponding to the memory mode is challenging. However, the area is assumed to be located in a higher-order region unrelated to any sensor or behavior. Furthermore, the modal memory itself is real and does not need to distinguish between imagination. Therefore, an agranular neocortical area with no granular layer to block input would be appropriate. From this point of view, areas such as the anterior cingulate cortex (MFC) are promising areas for mode memory.

3.4.5 Recorder is allocated to hippocampal formation

The ROI responsible for the Recorder mechanism in the brain corresponds to the hippocampal formation (HPF), which is composed of the hippocampus and the entorhinal cortex (EC).

First, as an internal mechanism of the Recorder, it acquires pattern signals from various parts of distributed memory, records them as pattern indexes, and outputs pattern index signals to reproduce patterns in the allocated memory. A pattern signal is an overall representation in distributed memory that includes various patterns such as mode, context, routing, designated control, etc. In the hippocampus, the dentate gyrus, which receives input from EC, isolates patterns (Kesner, 2007), and the recurrent circuits of CA3 and CA2 that receive this information can rapidly encode the sequence of those patterns (Rolls et al., 2005; Kesner, 2007; Kubik et al., 2007; Rolls, 2007; Behrendt, 2013; Okamoto and Ikegaya, 2019). This configuration is thought to allow the hippocampus to associate and encode various information conveyed by input from EC to form temporally linked episodic memories (Fortin et al., 2004; Eichenbaum and Fortin, 2005) and participate in episodic replay along with information about time and place (Ergorul and Eichenbaum, 2004). Thus, the internal mechanisms of the recorder would be consistent with HPF.

Furthermore, for the Recorder to perform the function of history management concerning distributed memory, it must be signal interconnected with distributed memory, acquire pattern signals over a wide area, and output a pattern index. The hippocampus presents structures that interconnect with extensive areas of the neocortex via EC, and this structure is conserved across species (Eichenbaum, 2000; Vincent et al., 2006; Kahn et al., 2008; Behrendt, 2013). Therefore, the connection form of recorders and distributed memory may correspond to the form of HPF and neocortex.

From the above discussion, it seems reasonable to map the Recorder mechanism to HPF regarding both internal mechanisms and connection form.

3.5 Summary of the hypothesis of imagination architecture

The results provide the hypothesis that imaginary zone makers map to the claustrum, routing conductors to a complex consisting of the basal ganglia and thalamus, mode memory to a specific agranular neocortical area, and recorders to HPF. The above finding of multifaceted concordance between the internal processing of the components and the meaning of the signals exchanged would seem to validate this hypothesis. The observed multifaceted concordance between the internal functioning of these components and the semantic nature of the exchanged signals lends considerable support to this hypothesis.

Table 1 shows the function of each component and the corresponding brain organ, including the mapping of distributed memory to the neocortex, which is the premise of this study. Table 2 then shows the signals exchanged between those components and their functional meaning.

Table 1. Components of imagination architecture.