The clinical benefits of decoding speech and movement intention

The terms neuromotor prosthetic or more recently motor neuroprosthetic is often used to describe an ICN which records a signal from motor cortex and translates that into movement (frequently virtual movement of a cursor on a screen) [14,15,16]. In the context of the motor signal being used to control a digital device, this output has been described as a digital motor output [16].

For someone with paralysis, a motor neuroprosthetic can be used either to restore their own anatomical movement, or to provide control of an external effector (Fig. 1).

Fig. 1

Categories of movement restoration and effector control that can be driven by a motor neuroprosthesis

Restoration of anatomical movement has been achieved through activating the user’s muscles by stimulating the spinal epidural space or muscles directly [10, 17]. An alternative approach is to power the user’s movement using an external orthosis. In this application, the neuroprosthetic controls movement of the orthosis and the powered orthosis moves the body of the user. This orthosis could be a rigid exoskeleton or based on soft robotic principles [18, 19].

An alternative approach is to provide direct control over a separate external end effector. Effector control can be virtual, for example, allowing the user to control a mouse to operate a graphical user interface on a tablet computer [12]. Control can also be over a physical effector, such as a table-mounted robotic arm, which could support the user with activities of daily living, or an electric wheelchair [20].

An ICN can also contribute to rehabilitation. This is particularly relevant for motor neuroprostheses which are targeted towards restoration of limb movement. There are two mechanisms by which this could occur. First, by reconnecting intention and action, it is hypothesised that a neuroprosthetic can induce neuroplastic changes in native circuits. Second, by facilitating movement in disused muscle groups thus preventing them from becoming deconditioned. This can reduce the likelihood of contractures and permanent deformity, as well as bone resorption and osteopenia from reduced weight bearing stress on long bones [21, 22]. The use of ICNs to facilitate recovery through a combination of immediate movement restoration and rehabilitation has already been demonstrated, and may become the predominant paradigm through which these devices enter clinical use [21].

Restoration of communication is frequently the aim of an ICN. Providing control over an electronic device with virtual movement has been used to facilitate virtual typing with the user able to move a computer cursor across a virtual keyboard [23]. An alternative approach that has been used has been to use the motor signal from the neuroprosthetic to provide a mouse ‘click’ which can be combined with an eye-tracking gaze control interface. In this way, the user is able to move across options with their eye movement and select an option from a graphical user interface using the click provided by the ICN [15].

Whilst control over an external device can facilitate communication, there have also been ICNs designed to directly decode neural signals into speech by recording from the face and laryngeal region of the motor cortex [7]. Directly decoding these signals into speech allows for a more seamless and intuitive communication experience for the participant. Incorporation of recordings of a participant's voice from prior to their injury as well as the use of a virtual avatar modelled on their facial expressions has allowed for the speech to be virtually embodied with some of the sound and appearance of the participant themself [9].

Approaches to recording neural activity

Implanted cortical neuroprosthetics which have reached clinical trials have placed recording electrodes in one of four anatomical compartments: intracortical, subdural, extradural and intravascular (Fig. 2).

Fig. 2

Illustration of devices in the four different anatomical compartments in which motor neuroprostheses have been implanted in human trials

The greatest experience has been with intracortical microelectrode arrays (MEA). The most commonly used is the “Utah array” (Blackrock), a 4 mm square MEA with typically 100 1.5 mm silicon electrode shanks. These intracortical arrays are inserted into the brain surface to allow for dense local electrode coverage and multiple MEAs may be placed to increase spatial coverage of the brain. MEAs require transgression of the cortex itself although the electrodes themselves are relatively small—the shanks of a Utah array are 80 µm in diameter (Fig. 3a) [24]. More recently a system of flexible electrode “threads” has been developed (Neuralink [25]), compared to a Utah array, these threads have a much higher density of recording sites and the advantage of being able to move with the brain parenchyma. The 64 flexible threads on their first implant are 10–12 microns in diameter and each carry 16 electrodes. The first human implantation of a flexible electrode thread system was notable for retraction of some of these recording threads from the brain, which reduced the number of useful recording sites for the INC [26].

Fig. 3

Figure adapted from sources referenced and images reproduced with permission

(a) A Utah Microelectrode Array (panel figure adapted from [14] used with permission) (b) WIMAGINE extradural electrode array (panel figure adapted from [27] used with permission); (c) Stentrode endovascular neuroprosthesis (panel figure from Synchron Corporation [28] used with permission); (d) Subdural electrode arrays (arrays in the panel figure are adapted from Ad-Tech Medical and used for seizure monitoring [29] used with permission)

Subdural electrodes, arranged on an electrocorticography (ECoG) or micro-ECoG array, can be placed without transgressing the pia once the subdural space is accessed by craniotomy. They arguably offer the greatest potential for spatial coverage of all the implanted approaches [30] (Fig. 3d). A relative disadvantage of this approach, compared to an MEA for example, is that it typically requires access to a relatively large surface of neocortex to record signal from. Presently, this necessitates a large craniotomy although some device manufacturers are developing minimally invasive techniques to facilitate ECoG-based approaches [31].

Extradural electrodes do not require opening the dura, potentially reducing the risks of brain injury and cerebral infection, as the dura forms an anatomical barrier between the device and cortical tissue. By removing a disc of calvarial bone and using the ICN device itself as the cranioplasty, it is possible to place a relatively large implant with minimal soft tissue distortion. In this design, the electrodes can be integrated directly into this device so there is no relative movement between the electrodes and the rest of the implant (Fig. 3b) [32]. A disadvantage of this approach is that a layer of dura intervening between the recording electrode and the brain is likely to reduce the resolution compared with a comparable electrode in the subdural compartment.

The first intravascular neuroprosthetics have recorded from electrodes mounted on a stent in the sagittal sinus [15]. By deploying the stent adjacent to the precentral gyrus, it is possible to record correlates of movement intention principally from the motor leg area. An advantage of this approach is that electrodes can be deployed without the need for craniotomy, and the neurointerventional procedure of stent deployment into venous sinuses is well established [33]. As with extradural systems, a limitation of deployment within the sagittal sinus is that the neural signal recorded is attenuated by an intervening layer to the cortical surface—the dura of the sinus itself. A specific risk of intravascular stents is inducing venous sinus thrombosis, although in trials this has been preempted by antiplatelet prophylaxis with no thrombosis reported to date (Fig. 3c) [15].

Signal decoding, calibration and training

Once a signal is recorded from the brain, a mapping function is needed to relate features of the neural signal to a movement or speech intention. This algorithm is called a decoder. The decoder is calibrated to the specific neural activity of an individual user by a training period, during which the user attempts (or imagines attempting) relevant tasks. The spatiotemporal relationship between the neural activity recorded and the intended action is the basis for calibration of the decoder.

Recent progress in ICNs has been made possible by advanced decoding strategies combining, for example, Bayesian classifiers, recurrent neural networks and language models for word prediction [7, 9]. The decoding strategy is also dependent upon whether the source signal comprises unit activity, local field potential or ECoG signal. To illustrate the principles involved in decoding, we present a relatively simple example of a decoder from one of the earliest clinical studies in intracranial neuroprosthetics—this is not intended to be representative of the mathematical complexity behind how the most advanced current decoders work (see, for example, Fig. 6), but this early example is an excellent illustration of the principles and highlights some of the challenges to these techniques transferring directly into a clinically useful device.

The first human implantation of a Utah microelectrode array was into the hand area of the primary motor cortex (BrainGate trial) [14]. Electrical activity caused by the discharge of individual or small groups of neurons were recorded from electrodes in the array—these electrical discharges are termed units (Fig. 4a).

Fig. 4

Figure adapted from source referenced and images reproduced with permission

Neural recordings from the first human user of a microelectrode array based neuroprosthetic (a) A well-isolated single unit recording from a single electrode (trace shows the superposition of 80 waveforms); (b) Over 80 seconds the participant was asked to imagine performing a series of movements in the arm contralateral to the array. Spiking activity of a recorded unit is shown along the top of the panel with the normalised integrated firing rate immediately below that. This unit demonstrates an increased firing frequency with the instruction to move hands apart/together; (c) Spike rates for two units recorded simultaneously during the performance of movement of the on screen neural cursor. The unit recorded in channel 1 demonstrates increased firing with the cue to move the cursor upwards but not downwards. Conversely, the unit recorded in channel 2 demonstrates increased firing following the cue to move the cursor downwards but not upwards; (d) Research technician and participant neural cursor traces during a 5 second period of the participant tracking the cursor; Panel figures adapted from [14] and used with permission. Channel numbers altered for simplicity of presentation

Changes in the unit activity recorded by each of these electrodes form the basis of the control signal. By asking the participant to imagine performing different movements, they are able to modulate this signal, in particular to increase the firing frequency of particular units (Fig. 4b).

For the participant to be able to control the movement of a cursor on a screen, a decoder was trained. By asking the participant to imagine physically moving a cursor on a screen in particular directions, units which have an increased firing frequency associated with this imagined movement can be identified. If a direction of movement is identified which is associated with an increased firing frequency for a particular neuron, then this neuron is said to have directional tuning.

To control the movement of a cursor on a screen in a two-dimensional space, a decoder was built by creating a mapping (often referred to in the literature as a “filter”) between the set of firing frequencies of the sampled units to a two-dimensional output signal. This mapping was initially constructed by asking the participant to imagine tracking a cursor moved by a member of the study team. Once an initial mapping between neural activity and intended cursor direction was established, it could then be used to place a neurally controlled cursor on the screen. Further calibration with the participant moving the neural cursor on screen enabled refinement of the mapping (closed-loop calibration).

In this first demonstration of a motor neuroprosthetic in the BrainGate trial, the user’s control of the neural cursor was limited to the context of research sessions. Each of these sessions would begin with the training of the decoder, i.e. even if the user wanted to perform the exact same task they had undertaken the day before they would still need to go through the calibration task to calibrate the decoder [34].

Retraining of the decoder was needed because of nonstationarities in the neural signal, i.e. the relationship between neural signal and movement intention is not stable over time. This is likely to be due in part to small movements of the array, changes in the local cellular environment, but also because of the changes in the tuning of the individual neurons themselves. This phenomenon, which is described as representational drift, [35] means that a highly directionally tuned neuron which has an increased firing rate associated with imagined leftwards movement one week may have no such tuning the following week. Consequently, without recalibration, the accuracy of a decoder declines over time.

To reduce the need for retraining, a strategy which has been subsequently employed is to continuously recalibrate the decoder based upon the data implicitly provided by the user as they use the interface [23]. Rather than relying upon a training epoch in which the user is asked to move a neural cursor towards a target specified by the research team, the intended target can be retrospectively inferred during active use and these data are used to calibrate the decoder. The advantage of this technique (retrospective target inference) is that it reduces the need for the user to disrupt their device use to undertake recalibration tasks.

Whilst the earliest implementations of microelectrode-based BCI required daily retraining sessions immediately prior to every use, in the most recent applications of continuous online recalibration, it has been possible to demonstrate stable decoding without retraining for more than 1 year of device use with over 90% accuracy in an online handwriting task [14, 36].

ECoG signals rather than unit activity can also be used as the control signal for a neuroprosthetic. Functional cortical activity is typically associated with an increase in high gamma power and a corresponding reduction in low-frequency power [37]. In a typical application, neural signals recorded by ECoG electrodes can be processed to extract both bands of high-frequency ‘high gamma’ activity (e.g. 70–150 Hz) as well as low-frequency signals (e.g. 0.3–17 Hz) [9, 38]. The power in these bands can then be used as the control signal for a neuroprosthetic. ECoG signals, representing the average activity of large numbers of neurons, might contain less information but have the advantage of increased stability, and consequently, do not require the same retraining as an MEA-based neuroprosthetic [39].

Whichever source signal is used, an important consideration for clinical translation is how much input is needed from engineers or neuroscientists when the system is in use. In applications based in university labs, participants are often engaged in training sessions and activities with an engineer or neuroscientist working alongside them on optimising device function. As more commercially ready devices are being developed for home use, there is a move towards engineers developing software that can run on a smartphone or tablet computer and enable the user to undertake device training themselves without the need for direct supervision from an engineer or researcher. In this paradigm, industry will manufacture devices with paired software that can be prescribed by a treating clinician and trained by the user in their own home.

Challenges and considerations for long-term implantation

High-performance intracranial neuroprosthetics for both speech synthesis and device control have been demonstrated in research settings [7, 12]. This performance has yet to transfer into a reliable device that is suitable for the participant to use any time at home as their primary means of communication. There are two challenges that are specific to these cortical neuroprosthetics when compared with intracranial devices that have moved into routine practice such as cochlear implants or DBS.

As outlined above, one key challenge is signal stability. Most of the experience using implanted cortical neuroprosthetics has been with MEA where there is considerable change in the neural control signal over time. Some of these changes—hypothesised to be due both to small movements of the probe within the brain as well as due to representational drift—can be overcome with recalibration of the decoder. However, recalibration of the decoder has typically relied upon regular training sessions with highly motivated cognitively preserved participants. The intensive input needed from both the research team and the participant has meant that recruiting sites are typically limited to very small numbers of participants, often just one. The need for retraining has been a significant barrier to the development of a device which the participant can use at home without the supervision of the research team.

Another changes seen with MEA is a decay of the quality of the neural signals which starts within months after implantation [40, 41]. This is thought to be a consequence of both degradation of the arrays and a biological changes in the implanted tissue, including glial encapsulation and neuronal loss surrounding the arrays [42, 43]. To overcome this, many innovative electrode designs are being produced using new materials, for which we lack long-term stability data. Multi-layered structures can delaminate, metals can corrode and even silicon passivation layers are known to hydrate after long immersion periods. More studies are needed to understand their degradation mechanisms in the body environment and whether by-products of this degradation can alter their biocompatibility [44, 45].

The second challenge is data transfer. Neuromodulatory systems, like DBS, are typically programmed wirelessly. Once stimulation settings are determined, the device runs without requirement for data transfer between the implanted device and external system. Cochlear implants do require data transfer but the amount of data transferred is much lower than can be generated using, for example, a microelectrode array.

A typical goal of neuroprosthetics for speech or motor restoration is transmission of data (from which to infer intention or control commands) to an external effector. Intracortical neuroprosthetics typically require high sampling rates (often several kHz for unit recordings) and consequently generate significant volumes of data. Electromagnetic transfer of data using fully implanted systems is limited due to factors such as signal attenuation from passing through the tissues, heating of the device, and challenges in optimising transmission with sufficiently small implanted devices [46, 47]. Performing data preprocessing on the internalised part of the system can reduce the amount of data that needs to be transmitted, but this introduces power consumption and heat dissipation challenges. Consequently, fully implantable neuromodulatory systems have typically relied on ECoG signal with a lower demands for data transmission [15, 38].