In the news: clever coding gets the most out of retinal prosthetics

This is something of an update to a previous post, but I thought interesting enough for its own blog entry. Just out in PNAS, Nirenberg and Pandarinath describe how they mimic the retina’s neural code to improve the effective resolution of an optogenetic prosthetic device (for a good review, see Nature News).

As we have described previously, retinal degeneration affects the photoreceptors (i.e., rod and cone cells), but often spares the ganglion cells that would otherwise carry the visual information to the optic nerve (see retina diagram below). By stimulating these intact output cells, visual information can bypass the damaged retinal circuitry to reach the brain. Although the results from recent clinical trials are promising, restored vision is still fairly modest at best. To put it in perspective, Nirenberg and Pandarinath write:

[current devices enable] "discrimination of objects or letters if they span ∼7 ° of visual angle; this corresponds to about 20/1,400 vision; for comparison, 20/200 is the acuity-based legal definition of blindness in the United States"

Obviously, this poor resolution must be improved upon. Typically, the problem is framed as a limit in the resolution of the stimulating hardware, but Nirenberg and Pandarinath show that software matters too. In fact, they demonstrate that software matters a great deal.

This research focuses on a specific implementation of retinal prosthesis based on optogenetics (for more on approach check out this Guardian article, and for an early empirical demonstration). Basically, intact retinal ganglion cells are injected with a genetically engineered virus that produces a light sensitive protein. These modified cells will now respond to light coming into the eye, just as the rods and cones do in the healthy retina. This approach, although still being developed in mouse models, promises a more powerful and less invasive alternative to electrode arrays previously trialled in humans. But it is not the hardware that is the focus of this research. Rather, Nirenberg and Pandarinath show how the efficacy of the these prosthetic devices critically depends on the type of signal used to activate the ganglion cells. As schematised below, they developed a special type of encoder to convert natural images into a format that more closely matches the neural code expected by the brain. 

The steps from visual input to retinal output proceed as follows: Images enter a device that contains the encoder and a stimulator [a modified minidigital light projector (mini-DLP)]. The encoder converts the images into streams of electrical pulses, analogous to the streams of action potentials that would be produced by the normal retina in response to the same images. The electrical pulses are then converted into light pulses (via the mini-DLP) to drive the ChR2, which is expressed in the ganglion cells.

This neural code is illustrated in the image below: 

The key result of this research paper is a dramatic increase in the amount of information that is transduced to the retinal output cells. They used a neural decoding procedure to quantify the information content in the activity patterns elicited during visual stimulation of a healthy retina, compared to optogenetic activation of ganglion cells in the degenerated retina via encoded or unencoded stimulation. Sure enough, the encoded signals were able to reinstate activity patterns that contained much more information than the raw signals. In a more dramatic, and illustrative, demonstration of this improvement, they used an image reconstruction method to show how the original image (baby's face in panel A) is first encoded by the device (reconstructed in panel B) to activate a pattern of ganglion cells (image-reconstructed in panel C). Clearly, the details are well-preserved, especially in comparison to the image-reconstruction of a non-encoded transduction (in panel D). In a final demonstration, they also found that the experimental mice could track a moving stimulus using the coded signal, but not the raw unprocessed input.

According to James Weiland, ophthalmologist at University of Southern California (quoted in by Geoff Brumfiel Nature News), there has been considerable debate whether it is more important to try to mimic the neural code, or just allow the system to adapt to an unprocessed signal. Nirenberg and Pandarinath argue that clever pre-processing will be particularly important for retinal prosthetics, as there appears to be less plasticity in the visual system than say the auditory system. Therefore, it is essential that researchers crack the neural code of the retina rather than hope the visual system will learn to adapt to an artificial input. The team are optimistic:

"the combined effect of using the code and high-resolution stimulation is able to bring prosthetic capabilities into the realm of normal image representation"

But only time, and clinical trials, will tell.


References:

Bi A, et al. (2006) Ectopic expression of a microbial-type rhodopsin restores visual responses in mice with photoreceptor degeneration. Neuron 50(1):23–33.

Nirenberg and Pandarinath (2012). Retinal prosthetic strategy with the capacity to restore normal vision. PNAS

In the news: Mind Reading

Mind reading tends to capture the headlines. And these days we don't need charlatan mentalists to perform parlour tricks before a faithful audience - we now have true scientific mind reading. Modern brain imaging tools allow us to read the patterns of brain activity that constitute mind... well, sort of. I thought to write this post in response to a recent Nature News Feature on research into methods for reading the minds of patients without any other means of communication. In this post, I consider what modern brain imaging brings to the art of mind reading.

Mind reading as a tool for neuroscience research

First, it should be noted that almost any application of brain imaging in cognitive neuroscience can be thought of as a form of mind reading. Standard analytic approaches test whether we can predict brain activity from the changes in cognitive state (e.g., in statistical parametric mapping). It is straightforward to turn this equation round to predict mental state from brain activity. With this simple transformation, the huge majority of brain imaging studies are doing mind reading. Moreover, a class of analytic methods known as multivariate (or multivoxel) pattern analysis (or classification) have come even closer to mind reading for research purposes. Essentially, these methods rely on a two-stage procedure. The first step is to learn which patterns of brain activity correspond to which cognitive states. Next, these learned relationships are used to predict the cognitive state associated with brain activity. This train/test procedure is strictly "mind reading", but essentially as a by-product.

In fact, the main advantage of this form of mind reading in research neuroscience is that it provides a powerful method for exploring how complex patterns in brain data vary with the experimental condition. Multivariate analysis can also be performed the other way around (by predicting brain activity from behaviour, see here), and similarly, there is no reason why train-test procedures can't be used for univariate analyses. In this type of research, the purpose is not actually to read the mind of cash-poor undergraduates who tend to volunteer for these experiments, but rather to understand the relationship between mind and brain.

Statistical methods for prediction provide a formal framework for this endeavour, and although they are a form of mind reading, it is unlikely to capture the popular imagination once the finer details are explained. Experiments may sometimes get dressed up like a mentalist's parlour trick (e.g., "using fMRI, scientists could read the contents of consciousness"), but such hype invariably leaves those who actually read the scientific paper a bit disappointed by the more banal reality (e.g., "statistical analysis could predict significantly above chance whether participants were seeing a left or right tilted grating"... hardly the Jedi mind trick, but very cool from a neuroscientific perspective), or contribute to paranoid conspiracy theories in those who didn't read the paper, but have an active imagination.

Mind reading as a tool for clinical neuroscience

So, in neuroscientific research, mind reading is most typically used as a convenient tool for studying mind-brain relationships. However, the ability to infer mental states from brain activity has some very important practical applications. For example, in neural prosthesis, internal thoughts are decoded by "mind reading" algorithms to control external devices (see previous post here). Mind reading may also provide a vital line of communication to patients who are otherwise completely unable to control any voluntary movement.

Imagine you are in an accident. You suffer serious brain damage that leaves you with eye blinking as your only voluntary movement for communicating with the outside world. That's bad, very bad in fact - but in time you might perfect this new form of communication, and eventually you might even write a good novel, with sufficient blinking and heroic patience. But now imagine that your brain damage is just a little bit worse, and now you can't even blink your eyes. You are completely locked in, unable to show the world any sign of your conscious existence. To anyone outside, you appear completely without a mind. But inside, your mind is active. Maybe not as sharp and clear as it used to be, but still alive with thoughts, feelings, emotions, hopes and fears. Now mind reading, at any level, becomes more than just a parlour trick.

"It is difficult to imagine a worse experience than to be a functioning mind trapped in a body over which you have absolutely no control" Prof Chris Frith, UCL [source here]

As a graduate student in Cambridge, I volunteered as a control participant in a study conducted by Adrian Owen to read mental states with fMRI for just this kind of clinical application (since published in Science). While I lay in the scanner, I was instructed to either imagine playing tennis or to spatially navigate around a familiar environment. The order was up to me, but it was up to Adrian and his group to use my brain response to predict which of these two tasks I was doing at any given time. I think I was quite bad at spatially navigating, but whatever I did inside my brain was good enough for the team to decode my mental state with remarkable accuracy.

Once validated in healthy volunteers (who, conveniently enough, can reveal which task they were doing inside their head, thus the accuracy of the predictions can be confirmed), Adrian and his team then applied this neuroscientific knowledge to track the mental state of a patient who appeared to be in a persistent vegetative state. When they asked her to imagine playing tennis, her brain response looked just like mine (and other control participants), and when asked to spatially navigate, her brain looked just like other brains (if not mine) engaged in spatial navigation.

In this kind of study, nothing very exciting is learned about the brain, but something else extremely important has happened: someone has been able to communicate for the first time since being diagnosed as completely non-conscious. Adrian and his team have further provided proof-of-principle that this form of mind reading can be applied in other patients to test their level conscious awareness (see here). By following the instructions, some patients were able to demonstrate for the first time a level of awareness that was previously completely undetected. In one further example, they even show that this brain signal can be used to answer some basic yes/no questions.

This research has generated an enormous amount of scientific, clinical and public interest [see his website for examples]. As quoted in a recent Nature New Feature, Adrian has since been "awarded a 7-year Can$10-million Canada Excellence Research Chair and another $10 million from the University of Western Ontario" and "is pressing forward with the help of three new faculty members and a troop of postdocs and graduate students". Their first goal is to develop cheaper and more effective means of using non-invasive methods like fMRI and EEG to restore communication. However, one could also imagine a future for invasive recording methods. Bob Knight's team in Berkeley have been using electrical recording made directly from the brain surface to decode speech signals (see here for a great summary in the Guardian by Ian Sample). Presumably, this kind of method could be considered for patients identified as partially conscious.

See also an interesting interview with Adrian by Mo Constandi in the Guardian

References:
Monti, al. (2010). Willful modulation of brain activity in disorders of consciousness. New England Journal of Medicine
Owen, et al (2006). Detecting awareness in the vegetative state. Science

Pasley,  et al (2012). Reconstructing Speech from Human Auditory Cortex. PLoS Biology

In the news: More neural prosthetics

Last week we heard about the retinal implant, this week is all about the neural prosthetic arm (video). As part of a clinical trial conducted by the BrainGate team, patients suffering long-term tretraplegia (paralysis including all limbs and torso) were implanted with tiny 4x4mm 96-channel microelectrode arrays. Signals from the primary motor cortex were then recorded, and analysed, to decode action commands that could then be used to drive a robotic arm. According to one of the patients:

"At the very beginning I had to concentrate and focus on the muscles I would use to perform certain functions. BrainGate felt natural and comfortable, so I quickly got accustomed to the trial."

Plugging directly into the motor cortex to control a robotic arm could open a whole host of possibilities, if the even larger host of methodological obstacles can be over come. Neuroscientists have become increasingly good at decoding brain signals, especially those controlling action, and are continually fine tuning these skills (see here in the same issue of Nature for another great example of the basic science that ultimately underpins these kinds of clinical applications). The biggest problem, however, is likely to be the bioengineering challenge of developing implants that can read brain activity without damaging neurons over time. The build up of scar tissue around the electrodes will inevitably reduce the quality of the signal. As noted by the authors:

"The use of neural interface systems to restore functional movement will become practical only if chronically implanted sensors function for many years" 

They go on to say that one of their experimental participants had been implanted with their electrode array some 5 years earlier. Although they concede that the quality of the signal had degraded over that time, it was still sufficiently rich to decode purposeful action. They suggest that:

"the goal of creating long-term intracortical interfaces is feasible"

These results are certainly encouraging, however such high-profile trials should not overshadow other excellent research into non-invasive methods for brain computer interface. To avoid neurosurgical procedures has obvious appeal, and would also allow for more flexibility in updating hardware as new developments arise.



References:

Hochberg, Bacher, Jarosiewicz, Masse, Simeral, Vogel, Haddadin, Liu, Cash, van der Smagt & Donoghue. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature, 485(7398):372-5

Ethier, Oby, Bauman & Miller (2012) Restoration of grasp following paralysis through brain-controlled stimulation of muscles. Nature, 485(7398):368-71.

In The News: The "Bionic Eye"

It is a big news story in the UK at the moment. Surgeons at the John Radcliffe Hospital in Oxford have implanted UK's first subretinal prosthetic device (see link). This is an exciting development toward restoring useful vision to people suffering retinal degeneration. According to the NHS website, the clinical trial involves a number of patients with retinitis pigmentosa, which is a progressive eye disease affecting photoreceptors (rods and cones). So far, the results are promising. Quoting from the NHS press release:

"When his electronic retina was switched on for the first time, three weeks after the operation, James was able to distinguish light against a black background in both eyes. He is now reported to be able to recognise a plate on a table and other basic shapes, and his vision continues to improve"

Although these effects might seem modest to the sighted, they could provide major improvement in the quality of life for those involved in the trial. To a fully blind patient, even partial vision could dramatically increase their independence. According to the manufacturer, after implantation of the chip the patient’s visual ability should meet the following criteria:

• Orientation in space 
• visual field: 8° - 12° 
• Capacity to see without visual aids (except glasses): at least ability to count fingers, at best ability to recognize faces. 
• Ability to recognize the letters of the alphabet with additional visual aids. 
• Ability to see in surround brightness from 10 Lux to 100.000 Lux.

To achieve all these would certainly make a real difference to a fully blind patient. So, how does the device work? In general, there are two type of retinal implant being developed: epiretinal and subretinal. Both essentially work by converting light energy to electrical energy to stimulate intact retinal cells, which is normally done by damaged photoreceptors (rods and cones). The epiretinal variety consists of an external video camera that transmits a processed signal to the implant, which in turn activates the reintal cells corresponding to the pixelated representation of the image. The subretinal implant, used in this trial, is fitted behind the retina and microphotodiodes directly convert light into electrical impulses to stimulate retinal cells. The principal advantage of the subretinal device, everything is internal to the implant (except for a small power source fitted under the skin). To quote Professor MacLaren:

 

"What makes this unique is that all functions of the retina are integrated into the chip. It has 1,500 light sensing diodes and small electrodes that stimulate the overlying nerves to create a pixellated image. Apart from a hearing aid-like device behind the ear, you would not know a patient had one implanted."



Moreover, by directly stimulating retinal cells rather than ganglion cells results in a more direct and natural correspondence between the implant and the underlying biology. Essentially, this reflects a general trade-off principle in neuroprosthetics. The simplest and most effective devices (e.g., cochlear implants) utilize the existing organisation of primary receptor surfaces, and/or their close neighbours, thereby minimizing the engineering challenge of interfacing with the more complex neural coding schemes. But this only works if those structures remain intact. To by-pass the entire sensory organ and project directly to the cortex is an entirely different game.