Neuromorphic systems & AI
Memristive devices： The high performance of the brain is due to its special architecture of operation dominated by synaptic activities, through which information processing and storage occur simultaneously. Ions (e.g. Ca2+, Na+, K+) are key to many neurological functions in the brain, while the kinetics of ionic defects in oxides is comparable to that of the ionic motion in the brain; memristors or memristive devices can be used to emulate the information processing in the brain. We develop memristors with controllable volatility based on the migration of ionic defects (e.g. oxygen vacancies, protons, Li-ions) in oxides; oxides are compatible to the state-of-art CMOS technology. We fabricate devices with 2, 3 and 4 terminals. Two-terminal devices take the normal crossbar structure; by proper combination of electrodes and oxides, and by modulating the fabrication process, we can tune the device properties to a quite large extent. In three-terminal devices, there is an extra Gate terminal; by applying electrical voltages to the Gate, both the concentration and distribution of ionic defects in the device can be readily tuned, which offers an extra freedom in tuning device properties. In some special cases, four-terminal devices are necessary, in which there are two Gates, an electrical one and an optical one, therefore, the device properties can be tuned by applying electrical and optical signals. Using the four-terminal device, we successfully emulated the function of astrocyte in a biological synapse.
Artificial synapses： Neuromorphic computing is an innovative technology by introducing systems capable of recognizing patterns and interacting with the external world in a cognitive way. To achieve this goal requires developing bio-realistic synaptic elements and neuronal units. In particular, synaptic plasticity is the foundation for biological intelligence and dominates the architecture of the brain. Given that a memristive device shares a similar structure and working principle to a biological synapse, it is relatively straightforward to emulate the synaptic plasticity with a memristive device. When emulating the synaptic plasticity, the conductance or current of the memristive device is regarded as the synaptic weight, and the external stimuli, like DC voltage, electric pulses, are treated as the nerve spikes. We have successfully realized numerous synaptic functions, like spike-timing-dependent plasticity (STDP), spike-rate-dependent plasticity (SRDP), short-term plasticity (STP), long-term plasticity (LTP), paired-pulse facilitation (PPF), BCM learning rules, metaplasticity, etc.
Artificial neurons： The implementation of artificial neurons, or neuronal units, depends mostly on CMOS circuits via integrating many transistors so far, which suffers from a low level of integration as well as high leakage power dissipation. However, the introduction of memristive devices significantly simplifies the neuronal circuits, as compared with those based on CMOS transistors only. Since the complex dynamics of Na+ and K+ ion channels is involved in the Hodgkin-Huxley (HH) model of neuron, the HH model describe some salient features of a biological neuron. We designed a hybrid circuit (Figure 1) based on the HH model, in the circuit memristive devices are the key elements. With the cicuit, neuronal functions, including the temporal and spatial integration of input signals, the local graded potential (LGP) with leaky feature, and the HH neuron-like spike firing, were successfully realized.
Artificial synapses： Neuromorphic computing is an innovative technology by introducing systems capable of recognizing patterns and interacting with the external world in a cognitive way. To achieve this goal requires developing bio-realistic synaptic elements and neuronal units. In particular, synaptic plasticity is the foundation for biological intelligence and dominates the architecture of the brain. Given that a memristive device shares a similar structure and working principle to a biological synapse, it is relatively straightforward to emulate the synaptic plasticity with a memristive device. When emulating the synaptic plasticity, the conductance or current of the memristive device is regarded as the synaptic weight, and the external stimuli, like DC voltage, electric pulses, are treated as the nerve spikes. We have successfully realized numerous synaptic functions, like spike-timing-dependent plasticity (STDP), spike-rate-dependent plasticity (SRDP), short-term plasticity (STP), long-term plasticity (LTP), paired-pulse facilitation (PPF), BCM learning rules, metaplasticity, etc.
Artificial neurons： The implementation of artificial neurons, or neuronal units, depends mostly on CMOS circuits via integrating many transistors so far, which suffers from a low level of integration as well as high leakage power dissipation. However, the introduction of memristive devices significantly simplifies the neuronal circuits, as compared with those based on CMOS transistors only. Since the complex dynamics of Na+ and K+ ion channels is involved in the Hodgkin-Huxley (HH) model of neuron, the HH model describe some salient features of a biological neuron. We designed a hybrid circuit (Figure 1) based on the HH model, in the circuit memristive devices are the key elements. With the cicuit, neuronal functions, including the temporal and spatial integration of input signals, the local graded potential (LGP) with leaky feature, and the HH neuron-like spike firing, were successfully realized.
Figure 1. HH neuron with leaky integrate-and-fire functions (Advanced Materials, 2019, 31, 1803849). a) Hybrid circuit with two memristive devices, b) spatial integration and bio-inspired fire realized with the circuit, c) temporal integration and bio-inspired fire realized with the circuit.
The footprint of the circuit in Figure 1(a) is relatively large, instead a simple alternative circuit can be used, in which a memristive device is connected with a capacitor in parallel to implement the integration effect via the charging process. However, the alternative circuit is not in line with observed neuronal behaviors, it only gives a bio-plausible description of neural behaviors.
The footprint of the circuit in Figure 1(a) is relatively large, instead a simple alternative circuit can be used, in which a memristive device is connected with a capacitor in parallel to implement the integration effect via the charging process. However, the alternative circuit is not in line with observed neuronal behaviors, it only gives a bio-plausible description of neural behaviors.
Artificial neural networks (ANNs)： By connecting artificial synapses and neurons according to the topological structure of biological neural networks, we obtain ANNs. The sneak path current in an ANN can be effectively suppressed by the one-transistor-one-memristor (1T1R) or one-selector-one-memristor (1S1R) architecture. However, the classification accuracy heavily depends on the performance of memristive synapses, in other words, ideal devices with linear and symmetric weight updating are preferred. However, most memristive synapses exhibit nonlinear dynamics in the switching behavior with asymmetric potentiation/depression. Overfitting is another problem, which usually happens in the case of a small number of training samples. Overfitting means that an ANN shows poor generalization, i.e., high accuracy for training data but very limited ability to make prediction out of the training set, because the network treats noises in training samples as feature.
An effective way to solve the problems of non-ideal device performance and overfitting is to add the dropout function in neuronal units. In this approach, a neuronal unit might drop out randomly at a probability in the training process. Thus, different combinations of neuronal units are activated in each training process, which results in variable networks in training, then the final network is the average of various trained networks. This average process dilutes the noises from training data, enhancing the generalization of the network in the case of small number of samples. Considering that the synaptic weight updating and the dropout in training are all nonlinear, dropout neural networks are more tolerant to the non-ideality of synaptic modulation. Therefore, the dropout function can not only prevent overfitting, but also mitigate the requirement for ideal memristive synapses.
Figure 2. Neuromorphic chip and peripheral circuits for input and output.
By means of device engineering, we fabricated a neuromorphic chip (Figure 2). With the proper design of algorithms and peripheral circuits, the chip successfully achieved the classification of letters with noises.
Until now experimentally demonstrated ANNs are relatively small, can only perform simple functions; to develop large-scale ANNs, one has to seek support from CMOS platforms. Once we could develop ANNs comparable to the brain, we would be able to fabricate artificial brains; to realize such a dream, at least a lifetime hardworking is needed.
The development of the neuromorphic computing is a long-term endeavor; we should try to incorporate existing knowledge along the way — rather than wait until we have understood everything about the brain. Current findings from neuroscience would greatly impact the development of bio-inspired devices and systems.
