A neuromorphic event data interpretation approach with hardware reservoir

1 Introduction

Event cameras are bio-inspired vision sensors that operate differently from conventional frame-based sensors. They are sensitive to detect the brightness change based on scene dynamics rather than capturing image's frame at a fixed rate for frame-based camera (Gallego et al., 2020). The event behavior in individual pixel is triggered only when the brightness intensity change exceeds the threshold without waiting for the command of the global shutter (Furmonas et al., 2022). Unlike the conventional camera that outputs an analog value in each pixel, the dynamic vision camera (DVS) produces a serials of events or spikes, which weigh only 1 bit (Lichtsteiner et al., 2008). The unique properties of event camera offer high temporal resolution and high dynamic range with better energy efficiency and latency compared with traditional cameras. With faster motion of objects, more events are generated as each pixel adjusts the delta modulator sampling rate according to the change of the logarithm of optical intensity. This makes event cameras exceptionally fast and efficient for edge applications, such as surveillance or monitoring, where only motion or change relevant (Delbruck and Lang, 2013; Glover and Bartolozzi, 2016).

However, traditional computer vision algorithms cannot be directly used to process event data as it only contains binary information of asynchronous brightness intensity change. How to effectively process event data still remains an issue. These event data are asynchronous spike behavior with microsecond-level resolution, which requires special algorithm or specialized hardware for good representation or prepossessing. Many studies have been proposed to realize good event data representation. Gehrig et al. (2019) proposed differentiable operation method to convert event streams into grid-based representations. Sironi et al. (2018) introduced a local memory sharing method with time surface, which converts event stream into an image with function of the motion history at that location. The other common method is to integrate event behaviors to frames by slicing a constant time window or fixed number of event packets (Everding and Conradt, 2018). Nevertheless, these methods are mainly based on the software level optimization without the implementation of hardware.

The memristor is an emerging non-volatile device, which shows the dynamic conductance range with programming ability (Khalid, 2019). It is a two terminal device, which retains a state of resistance with flexible response to varying electrical inputs. Many memristor devices exhibit the short-term-memory (STM) behavior, which has the non-linearity and memory decay characteristics in response to a stimulus as optical or electrical input (Kumar et al., 2023; Hu et al., 2021; Wu et al., 2023; Jo et al., 2010). It allows the devices to map temporal input patterns into collective memristor resistance states, which can be viewed as a dynamic reservoir node. Many research studies have been proposed to implement hardware reservoir computing based on memristor (Zhong et al., 2021, 2022; Sun et al., 2021). Such properties can efficiently process event data and extract information through the natural behavior of the device. The memristor reservoir encoder takes advantage of both spike neural network (SNN) and reservoir computing, which is effective to capture features in high dimensional space and resembles the processes of visual cognition (Chen et al., 2024). This approach provides a solution for event representation at a low cost, which is suitable for edge applications including robotic systems, real-time monitoring, and complex signal processing.

In this study, we propose a neuromorphic approach for event-based data interpretation (NEIR), which utilizes memristor-based circuit as reservoir encoder to encode event data. The stochastic nature and non-linear property enables the reservoir layer to effectively extract the event-based information at a low cost. We also propose a simplified memristor model based on the VTEAM model with time-surface behavior to fit the non-linear property. The state of memristor in reservoir circuit changes according to the input of event streams, and the result of extracted information is represented by the generated current. The reservoir layer allows the effective feature extraction for the temporal information in all prior spike inputs produced by the event camera without any use of dedicated memory units and logic circuits for complex preprocessing algorithms. Our design is triggered by the digital signal with ON and OFF events, which eliminates the need for Digital-to-Analog Converter (DAC) requirements. It is similar as retina cells that can directly sense and encode raw, asynchronous visual inputs at low cost. This scheme introduces a novel approach for event-data encoding with memristor-based reservoir node, which highlights the potential application in this field.

The contributions of this study are as follows:

1. We introduce a hardware system called NEIR which utilizes a memristor-based circuit to encode event data. The memristor array can effectively simulate the reservoir state as hardware implementation with non-linear property.

2. We propose a simplified memristor model specifically designed for the pulse behavior, which is suitable for non-linear behavior modeling and event-based data representation. It is an intuitive and easy-to-use framework derived by the VTEAM model to capture the pulse stimulus with less parameters.

3. We compare our memristor encoding method with other event representation methods with four event-based datasets. Our evaluations demonstrate that our encoding method can achieve the highest accuracy across the same bottleneck structure on both spike and non-spike models.

The remainder of this study is organized as follows. We introduce in Section 2 previous studies in both reservoir computing and memristor areas. In Section 3, we provide a detailed description of our proposed method. The performance of this approach is evaluated in Section 4. We then provide a discussion on experimental results and performance in Section 5. Finally, the study is concluded in Section 6.

1.1 Related work

Reservoir computing is a widely used machine learning framework known for its non-linear and dynamic characteristics (Yan et al., 2024). This approach facilitates the handling of complex temporal patterns without the need for extensive retraining or parameter adjustments inherent in traditional models (Cucchi et al., 2022). By leveraging a fixed number of randomly generated neurons in the reservoir to extract information in high dimensions, it allows for efficient computation and robust performance across a variety of temporal-spatial tasks (Tanaka et al., 2019). For hardware implementation, the memristor is a popular candidate to work as the physical reservoir due to its non-linear and dynamic programming properties (Cao et al., 2022). The memristor has shown its unique short-term or long-term memory with both optical and electrical programming abilities. It can work as the reservoir nodes to perform non-linear information transformation (encoding) of the temporal input data into the stored reservoir states (Zhang et al., 2023). In such systems, the reservoir node encodes the spatiotemporal information naturally by device dynamics, which eliminates the need for external memory or arithmetic and logic units (ALUs). By implementing physical reservoir with memristors, it can achieve outstanding energy efficiency and power consumption as the number of nodes is fixed.

The STM characteristic of the memristor brings its non-linear and time-dependent properties, which can serve as a suitable hardware platform for the dynamics required for reservoir computing. Many studies have shown the potential ability to utilize memristor-based circuit as the reservoir node and take advantage of the storage ability (Liang et al., 2024; Yan et al., 2024; Yang et al., 2024). Wu et al. (2023) integrated organic light-sensing materials into memristors to simulate an optical reservoir system with a spike neuromorphic network for pattern recognition. Yang et al. (2024) proposed a mask reservoir computing circuit using memristor arrays to process analog biomedical signals. Zhou et al. (2023) proposed light-responsive vision sensors that convert dynamic motion into event signals and extract features for efficient motion recognition. These studies have integrated memristors into reservoir computing platforms, offering potential opportunities for hardware-based reservoir computing applications.

However, most of the existing studies primarily concentrate on device characteristics and software simulations. In this study, we provide a general event representation method via memristor modeling and circuit design for reservoir computing, which enhances the event-based feature extraction with low energy consumption.

2 Memristor model and reservoir computing

Figure 1 shows the reservoir computing system implemented for event data processing. For event data processing, the memristor-based reservoir layer has the non-linear dynamic property to respond to stimuli, where both optical and electrical pulse can be viewed as potential stimuli (Pereira et al., 2023). The asynchronous events generated by event camera can be directly sensed and processed by the reservoir layer without any configuration of external logic circuits. The bipolar terminal in memristor enables the flexibility and reconfigurability of programming to different conductance values with effective feature extraction ability. The STM property of the memristor device offers the superior ability to react with temporal information and decay with time surface, which could work as the encoder for event data to extract information (Moon et al., 2019). The feature map extracted by the reservoir layer is sent to a classification bottleneck network for identification. The network including convolutional layer and fully connected layer can be easily implemented by memristor crossbar with high energy efficiency and low power consumption.

Figure 1. NEIR structure: asynchronous spike inputs are directly sent to hardware reservoir layer to extract the feature. The neural network can be implemented for further classification.

2.1 Memristor-based circuit unit

The memristor circuit unit plays as the basic element of reservoir node for event data interpretation. It is used to extract high dimension information and behave as non-linear state of reservoir. The DVS camera measures output with event behavior through the brightness change (Lichtsteiner et al., 2008). The pixel circuit is shown in Figure 2A, where the brightness change over threshold value of comparator is generated as ON and OFF event. The asynchronous event behaviors can be directly fed and processed by the proposed reservoir circuit as shown in Figure 2B. The ON and OFF signal represents the digital control signal of event data, while the Vin is the constant voltage supply for memristor programming. By definition of event data, the ON and OFF signal cannot be activated simultaneously as brightness change can be either positive or negative. The schematic diagram is shown in Figures 2C, D when ON and OFF signal is activated at a high level. The event data can drive the switch control of voltage, where the memristor state changes accordingly with the event type and stores the previous integrated information. The bipolar programming of the memristor device provides the flexibility of reservoir state modification with non-linear dynamics. According to the memristor readout, the multiplexer with clock control is utilized to read the conductance of each memristor simultaneously. During the read phase, a small voltage (< 0.15 V) is applied across the memristor to prevent any alteration of its conductance state. A transimpedance amplifier (TIA) can be utilized to convert the small input current into a proportional voltage signal for further processing.

Figure 2. (A) The circuit diagram of DVS sensor. (B) The circuit diagram of memristor-based reservoir node. (C) The circuit diagram when ON signal is activated. (D) The circuit diagram when OFF signal is activated.

2.2 A simplified memristor model

The VTEAM model is a popular voltage-controlled model considering the threshold voltage phenomenon, which is compatible with many window functions (Kvatinsky et al., 2015). In addition, the VTEAM model has great flexibility to simulate the non-linear dopant drift phenomenon. Considering the STM behavior in many solid-state memory devices, we improve the memristor model by adding a decay term below the threshold value. The optimized VTEAM model is shown below:

where xi and yi are the positions of an active pixel, pi represents the polarity of an event, ti is the time step, and ei is the ith event in the stream. For the reservoir encoding method, the memristor array updates its conductance state according to the event streams. We partition the event recordings into smaller chunks and perform feature extraction by reading the current periodically at each segment. These patterns in an analog state can be processed by subsequent DNN/SNN blocks. The proposed encoding method is shown in Algorithm 1.

Algorithm 1. The proposed encoding method.

2.4 Event-data representation methods and result visualization

Event data are processed and transformed into alternative representations to extract significant information and facilitate specified algorithm processing. These representation methods focus on converting raw event data into a format that is more amenable to algorithmic processing, enabling efficient information extraction and decision-making (Lakshmi et al., 2019). Our reservoir layer serves as the encoder with the same purpose but applying a memristor-based hardware framework with non-linear dynamics. This is distinct from traditional methods and aligns with our objective to optimize both performance and hardware compatibility. In this study, we make a comparison of existing popular representation methods for event data in Table 1. This comparison presents the characteristics and storage feature of each method, which highlight their suitability for different computational scenarios. Our proposed memristor-based method can extract both spatial and temporal features with the advantage of being hardware-friendly and energy-efficient.

Table 1. Comparison of event representation methods.

Figure 4A displays the visualization result of different event representation methods for digit “9” in NMNIST dataset. The visualization result contains four different methods, namely, raw event, frame, VoxelGrid, and our method. The raw event data consist of individual events plotted in space and time, resulting in a sparse and scattered visualization. This method retains all event information but lacks structure, making it difficult to discern the underlying pattern of the digit “9.” The frame-based representation method accumulates events over a fixed time interval to create a frame, which could blur temporal information and introduce motion artifacts. This VoxelGrid approach balances both spatial and temporal information but may suffer from quantization effects, which may lead to a loss of fine details. Our method extracts both spatial and temporal information through non-linear behavior of memristor device, maintaining clarity in the contours. The event representation methods aim to capture and retain essential features through sparse event behaviors, enabling efficient processing and analysis of dynamic visual scenes. Figure 4B shows the comparison result for raw event data and the data after memritsor-based reservoir encoder. The visualization result utilizes t-distributed stochastic neighbor embedding (t-SNE) method for dimensionality reduction. It contains 2,000 samples randomly chosen from test set of NMNIST dataset with the same experimental setup. Samples from the same category are more distinctly clustered after the proposed encoder, which indicates the information extraction ability of proposed memristor reservoir event representation method.

Figure 4. (A) Different event representation result visualization: Raw event behavior, Integrate-to-frame, VoxelGrid, and our method. (B) T-SNE visualization of NMNIST dataset before and after the proposed encoder.

3 Numericial experiment 3.1 Experiments on DVS dataset

We evaluate the memristor encoding method in DNN/SNN bottleneck structure through event-based datasets, including DVS Gesture (Amir et al., 2017), DVS CIFAR10 (Li et al., 2017), NMNIST (Orchard et al., 2015), and DVSLip (Tan et al., 2022). To integrate the information effectively, the encoder captures data at a frequency of 30 Hz and sent to the subsequent layers. In the simulation, the initial states of neurons in the reservoir are drawn from a normal distribution with a mean of (Rmin+Rmax)/2. The NEIR method allows efficient feature description of the temporal information in all prior spike inputs with simple DNN backpropagation rules across time steps. We compare the NEIR with existing encoding approaches including frame, time surface, and voxel grid. The comparison results are shown in Table 2. With the same experiment setting, we evaluate the event representation methods via spike and non-spike bottleneck network. For image recognition on DVS Gesture dataset, our NEIR method reaches the accuracy of 88.42% in VGG 11 model, which is much higher than the other encoding method (87.11% for time surface).

Table 2. Performance comparison of different Encoding methods.

We also apply spike-based bottleneck network to evaluate the performance of encoding methods. To solve the non-differentiability of spike behavior, we implement the spiking activation function with an approximation of gradient (Wu et al., 2019). We set the default time steps as 5 for each event sample and conduct experiment with different encoding methods. We observed that our method achieved an accuracy of 66.45% on the DVS CIFAR10 dataset using a spike-based VGG 16 model, compared with 64.95% when employing time-surface techniques. The NEIR method exhibits superior performance over other encoding method in both spike and non-spike architecture. The ablation study demonstrates that the NEIR method achieves high accuracy on the test dataset, indicating its effectiveness in capturing essential features from event data. The Pytorch platform and Tonic package were used for all the experiments with methods described above.

3.2 Effects of device variation and ADC resolutions

Figures 5A, B provide a summary of the impact of non-idealities on our system for image recognition on DVS Gesture dataset. Each data point in the figure shows the mean and standard deviation across five arbitrary seed values. Due to the stochastic ion behavior and vacancy forming process, the conductance of memristors displays variations and fluctuations from expected conductance value (Ielmini and Wong, 2018). Figure 5A illustrates the impact of device conductance variation. As device conductance variation increases, network performance gradually declines, dropping from 88.45% at a 5% conductance range to 80.47% at a 25% conductance range, with an increasing of accuracy variation. Figure 5B shows the effect of ADC resolution. With the higher resolution ADC, the precision of information improves and the network achieves better performance.

Figure 5. (A) The effect of device conductance variation. (B) The effect of ADC resolutions.

3.3 Performance evaluation

Table 3 shows the hardware evaluation and comparison for the proposed reservoir circuit node. We conduct the circuit simulation via PSPICE in Cadence and explore critical performance metrics including power consumption, energy usage, and latency. The result shows an average power consumption of 137 μW and normalized energy of 0.69 μJ per event with supply voltage of 3.3V, which highlights efficiency in information encoding at low cost. The expected propagation delay of 280.19 ns represents the total time delay across the critical path, where the memristor crossbar analysis is evaluated in Lu et al. (2021). Compared to other studies, our design demonstrates competitive performance. Zhong et al. (2021) reports a lower power consumption of 50 μW and a normalized energy of 0.006 μJ but emphasized on device measurement. Our evaluation is based on the individual memristor reservoir circuit and achieve competitive accuracies on multiple event-based datasets, which is more complex task across others. This circuit performance evaluation shows the requirements for the memristor-based reservoir node, which reveals the potential application for energy-efficient event sensing system.

Table 3. Performance comparison of proposed reservoir node.

4 Conclusion

In this study, we proposed NEIR, a neuromorphic approach using memristors as reservoir encoders. We utilize the stochastic and non-linear properties of memristors as reservoir node for effective, low-cost feature extraction of event data. Our design does not require a DAC component and directly encode ON and OFF events triggered by digital clock. We also present a simplified memristor model based on the VTEAM model to accurately capture pulse behaviors with fewer parameters. Comparative analyses across four datasets demonstrate that our approach achieves superior accuracy over other methods, illustrating the potential of memristor-based systems in real-time data processing and neuromorphic computing.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Author contributions

HL: Conceptualization, Formal analysis, Investigation, Methodology, Software, Validation, Writing – original draft. DK: Methodology, Supervision, Writing – review & editing. NE-A: Methodology, Funding acquisition, Supervision, Writing – review & editing.

Funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This work was supported by the King Abdullah University of Science and Technology.

Acknowledgments

Authors acknowledge financial support from King Abdullah University of Science and Technology, Saudi Arabia.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Amir, A., Taba, B., Berg, D., Melano, T., McKinstry, J., Di Nolfo, C., et al. (2017). “A low power, fully event-based gesture recognition system,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7243–7252. doi: 10.1109/CVPR.2017.781

PubMed Abstract | Crossref Full Text | Google Scholar

Biolek, Z., Biolek, D., and Biolkova, V. (2009). Spice model of memristor with nonlinear dopant drift. Radio Eng. 18, 210–214.

PubMed Abstract | Google Scholar

Cao, J., Zhang, X., Cheng, H., Qiu, J., Liu, X., Wang, M., et al. (2022). Emerging dynamic memristors for neuromorphic reservoir computing. Nanoscale 14, 289–298. doi: 10.1039/D1NR06680C

PubMed Abstract | Crossref Full Text | Google Scholar

Chen, L., Wang, X., Yang, C., Chen, Z., Zhang, J., and Zeng, Z. (2024). Full-analog reservoir computing circuit based on memristor with a hybrid wide-deep architecture. IEEE Trans. Circuits Syst. I: Regul. Pap. 71, 501–514. doi: 10.1109/TCSI.2023.3334267

More from this channel

A neuromorphic event data interpretation approach with hardware reservoir

留言 (0)