Journal Club: Decoding spatial activity patterns with high temporal resolution

by Michael Wolff

on: Cichy, Ramirez and Pantazis (2015) Can visual information encoded in cortical columns be decoded from magnetoencephalography data in humans? NeuroImage

Knowing what information the brain is holding at any given time is an intriguing prospect. It would enable researchers to explore how and where information are processed and formed in the brain, as well as how they guide behaviour.

A big step towards this possibility was made in 2005 when Kamitani and Tong decoded simple visual grating stimuli in the human brain using functional magnetic resonance imaging (fMRI). The defining new feature of this study was that instead of looking for differences in overall activity levels between conditions (or in this case visual stimuli), they tested the differences in activity patterns across voxels between stimuli. This method is now more generally known as multivariate pattern analysis (MVPA). A classifier (usually linear) is trained on a subset of data to discriminate between conditions/stimuli, and then tested on the left-out data. This is repeated many times, and the percentages of correctly labelled test data are reported. Crucially, this process is carried out separately for each participant, as subtle individual differences in activity patterns and cortical folding would be lost when averaged, defeating the purpose of the analysis. MVPA has since revolutionised fMRI research and, in combination with the increased power of computers, has become a widely used technique.

The differential brain patterns observed by Kamitani and Tong are thought to arise from the orientation columns in the primary visual cortex (V1), discovered by Hubel and Wiesel more than 50 years ago. They showed that columns contain neurons that are excited differentially by visual stimuli of varying orientations. Since these columns are very small (<1 mm) it is surprising that their activity patterns can apparently be picked up by conventional fMRI with about 2-3mm spatial resolution. More surprising still is that even magnetoencephalography (MEG) and electroencephalography (EEG) seem to be able to decode visual information, which are generally considered to have a spatial resolution of several centimetres! How is this possible?

Critics have raised alternative possible origins of the decodable patterns, which could result in more coarse-level activity patterns (e.g. by global form properties or overrepresentation of specific stimuli), and thus confound the interpretation of decodable patterns in the brain.

In response to these criticisms, a recent study by Cichy, Ramirez, and Pantazis (2015) investigated to what extent specific confounds could affect decodable patters by systematically changing the properties of presented stimuli. They used MEG as the physiological measure instead of fMRI. This enabled them to explore the time-course of decoding, which can be used to infer at which visual processing stage decodable patterns arise.

In the first experiment they showed that neither the cardinal bias (over representation of horizontal or vertical gratings) nor the phase of gratings (and thus local luminance) is necessary to reliably decode the stimuli.

Figure 1. From Cichy et al., in press

As can be seen from the decoding time-course the decodability is significant approximately 50 ms after stimulus presentation and ramps up extremely quickly, peaking at about 100 ms. This time-course alone, which was very similar in the other experiments testing for different possible confounds, suggests that the decodable patterns arise early in the visual processing pathway, probably in V1.

The other confounds that were tested involved the radial-bias (neural overrepresentation of lines parallel to fixation), the edge effect (gratings could be represented as ellipses elongated in the orientation of the gratings), and global form (where gratings are perceived as coherent tilted objects). None of these biases could fully explain the decodable patterns, casting doubt on the notion of coarse-level driven decoding. Again, how is this possible, when the spatial resolution of MEG should be far too coarse to pick up such small neural differences?

The authors tested the possibility of decoding neural activity from the orientation columns with MEG more directly. They projected neurophysiologically realistic activity patterns on to the modelled surface of V1 of one subject (A). The distance between each activity node was comparable to the actual size of the orientation columns. The corresponding MEG scalp recordings were obtained by forward modelling (B) and their differences decoded (C and D). The activity patterns could be reliably discriminated across a wide range of signal to noise ratios (SNR) and, most crucially, at the same SNR as in the first experiment.

Figure 2. From Cichy et al., in press

This procedure nicely demonstrates the theoretical feasibility of discriminating neural activity at V1 with MEG, and suggests that the well-known “inverse-problem” inherent to MEG and EEG source localisation does not necessarily mean that small activation differences on the sub-millimetre scale are not present in the activation topographies. While it remains impossible to say where the origin of a neural activation pattern lies, the activation pattern of MEG is still spatially rich.

Even with EEG it is possible to decode the orientations of gratings (Wolff, Ding, Myers, & Stokes, in press); and this can be observed more than 1.5 seconds after stimulus presentation. We believe that there is a bright future ahead for EEG and MEG decoding research: not only is EEG considerably cheaper than fMRI, but the time-resolved decoding offered by both methods could nicely complement the more spatially resolved decoding of fMRI.



References

Cichy, R. M., & Pantazis, D. (in press). Can visual information encoded in cortical columns be decoded from magnetoencephalography data in humans? NeuroImage.

Hubel, D. H., & Wiesel, T. N. (1959). Receptive fields of single neurones in the cat's striate cortex. The Journal of physiology, 148(3), 574-591.

Kamitani, Y., & Tong, F. (2005). Decoding the visual and subjective contents of the human brain. Nature Neuroscience, 8(5), 679-685.

Wolff, M. J., Ding, J., Myers, N. E., & Stokes, M. G. (in press). Revealing hidden states in visual working memory using EEG. Frontiers in Systems Neuroscience.

Journal Club: Brains Resonating to the Dream Machine

By George Wallis

On: VanRullen and Macdonald (2012). PerceptualEchoes at 10Hz in the Human Brain

One day in 1958 the artist Brion Gysin was sleeping on a bus in the south of France. The bus passed a row of trees, through which the sun was shining. As the flickering light illuminated Gysin, he awoke and with his eyes closed, began to hallucinate, seeing:

“an overwhelming flood of intensely bright patterns in supernatural colours… Was that a vision?”.  

By the turn of the decade Gysin was living with William S Burroughs in the flophouse in Paris that became known as the Beat Hotel. Gysin told Burroughs of his experience, and they decided to build a device to recreate the flickering stimulation. The ‘Dream Machine’ is a cylinder of cardboard, cut at regular intervals with windows, which can be spun on a 78rpm record player, a light bulb inside to throw off a flickering light.   The light flickers around ten times per second (10Hz). Some, like the poet Ginsberg (“it sets up optical fields as religious and mandalic as the hallucinogenic drugs”), claim to have experienced vivid hallucinations when seated eyes closed before a spinning Dream Machine (although, most devotees admitted that the effect was much stronger in combination with psychedelic drugs).

Gysin and Burroughs had rediscovered a phenomenon that had been known to scientists for some time. The great neurophysiologist Purkinje documented the hallucinatory effect of flickering light by waving an open-fingered hand in front of a gaslight. Another neuro-luminary, Hermann von Helmholtz, investigated the same phenomenon in Physiological Optics, calling the resulting hallucinations ‘shadow patterns’. In the 1930s Adrian and Matthews, investigating the rhythmic EEG signal recently discovered by Hans Berger, shone a car headlamp through a bicycle wheel and found that they could ‘entrain’ the EEG recording of their subject to the stimulation, in ‘a coordinated beat’. And from there investigation of the magical 10Hz flicker continued, on and off, until the present day (for a very readable review, see the paper by ter Meulen, Tavy and Jacobs referenced at the bottom of this post – from which the above quotations from Gysin and Ginsberg are taken; see also a relate post by Mind Hacks).

This week’s journal club paper is not about flicker-induced hallucinations. However, it does use EEG to address the related idea that there is something rather special to the visual system about the 10Hz rhythm. The paper, by Rufin VanRullen and James Macdonald, and published this month in Current Biology, used a very particular type of flickering stimulation to probe the ‘impulse response’ of the brain. They found – perhaps to their surprise – that the brain seems to ‘echo back’ their stimulation at about 10 echoes per second.

Macdonald and VanRullen’s participants were ‘plugged in’ during the experiment – electroencephalography (EEG) was used to measure the tiny, constantly changing voltages on their scalps that reflect the workings of the millions of neurons in the brain beneath. The stimulus sequence presented (with appropriate controls to ensure the participants paid attention) was a flickering patch on a screen. The flicker was of a very particular kind. It was a flat spectrum sequence, a type of signal used by engineers to probe the ‘impulse response’ of a system. The impulse response is the response of a system to a very short, sharp stimulation. Imagine clicking your fingers in an empty cathedral – that short, sharp click is transformed into a long, echoing sound that slowly dies away. This is the impulse response of the cathedral: VanRullen and MacDonald were trying to measure the impulse response of the brain’s visual system. Because of its property of very low autocorrelation (the value of the signal at one point in time says nothing about what the value of the signal will be at any other time), the kind of signal the authors flashed at their participants can be used to mathematically extract the impulse response of a system (for more details, see the paper by Lalor at al., referenced at the bottom of this post).

To extract the impulse response, you do a ‘cross-correlation’ of the input signal (the flickering patch on the screen) with the output of the system – which, in this case, was the EEG signal from over the visual cortex of the participants (the occipital lobe). Cross-correlation involves lining up the input signal with the output at many different points in time and seeing how similar the signals are. So, you start with the input lined up exactly with the output, and ask how similar the input and output signals look. Then you move the input signal so it’s lined up with the output signal 1ms later – how similar now? And so on… all the way up to around 1s ‘misalignment’, in this paper.  Here, for two example subjects (S1 and S2), is the result:

The grey curves are the cross-correlation functions, stretched out over time. Up until about 0.2 seconds you see the classic ‘visual evoked potential’ response, but after that time a striking 10Hz ‘echo’ emerges. The authors perform various controls, to show, for example, that these ‘echoes’ are not induced only by the brightest or darkest values in their stimulus sequence. They argue that because of the special nature of the stimuli they used, this effect must represent the brain actually ‘echoing back’ the input signal at a later time. In their discussion, they propose that this could be a mechanism for remembering stimuli over short periods of time: replaying them 10 times per second.

This is a bold hypothesis. Are these 10Hz reverberations really ‘echoes’ of the visual input, used for visual short term memory? We weren’t sure. We already know that the EEG resonates by far the most easily to flickering stimuli at 10Hz (see the paper by Hermann, referenced below), so despite the sophisticated stimulus used here, it is easy to suspect that the result of this experiment depends more on this ‘ringing’ quality of the EEG than on mnemonic echoes of stimuli themselves. We felt that in order to really nail this question you would need to show, for example, that our sensitivity to specific stimuli we have just been shown changes with a 10Hz rhythm in the seconds after we encounter it. However, this is the sort of thing that could be achieved with behavioural experiments.

Perhaps a new theory of short term memories will emerge.  

In the meantime, why not build yourself a dream machine and see if you can have your own visionary insights with the help of some 10Hz flickering light?  You’ll need the diagram below (blow up; cut out; fold into a cylinder), an old 78rpm record player, and a light-bulb.

References:

Current Biology, 2012: Perceptual Echoes at 10Hz in the Human Brain, Rufin VanRullen and James S.P. Macdonald

European Neurology, 2009: FromStroboscope to Dream Machine: A History of Flicker-Induced Hallucination,  B.C. ter Meulen, D. Tavy and B.C. Jacobs

NeuroImage, 2006: The VESPA: a method for the rapid estimation of a visual evoked potential.  Edmund C. Lalor, Barak A. Pearlmutter, Richard B. Reilly, Gary McDarby and John J. Foxe

Experimental Brain Research, 2001: Human EEG responses to 1-100 Hzflicker: resonance phenomena in visual cortex and their potential correlationto cognitive phenomena.

Journal Club:Twists and turns through memory space

By Nick Myers

You enter an unfamiliar building for a job interview. The receptionist tells you to make a left turn at the end of the corridor to get to your interviewer’s office. Easy instructions, but your brain has to remember them nonetheless. For the past decade, theoretical neuroscientists have proposed that, to do this job, neurons in the parietal cortex act as a kind of memory container: once you have learned that you need to make a left, dedicated ‘left-turn’ neurons are persistently active until you have reached the end of the hall, have turned, and can forget about it again. In addition to having lots of supporting evidence and enjoying intuitive appeal, the memory-container model has the advantage that, once the appropriate neurons are activated, they can potentially hold on to the ‘left-turn’ memory indefinitely (for instance, allowing you to get a drink of water before heading to the office).

However, a recent paper in the journal Nature has added to a growing list of evidence contradicting this model. In the paper, Princeton researchers Christopher Harvey, Philip Coen, and David Tank describe how ‘left-turn’ neurons in the parietal cortex of mice fire in a stereotypical cascade as the animals navigate along a virtual-reality corridor. The sequence begins with a small number of ‘left-turn’ neurons activating the next group and then falling silent again (see image), while the new group in turn activates yet another subset, and so forth until the end of the cascade is reached at the end of the corridor. In contrast to the memory-container model, this kind of dynamic activation sequence could be more similar to your car’s sat nav, constantly keeping you up to date on when you will have to turn left. Like a sat nav, dynamic memories could become more prominent when you are navigating through a complicated environment and have to make a left turn at the right time or in the right place (say, for instance, that there are lots of possible left turns, and you must remember to turn behind the drink fountain). Perhaps previous researchers may have failed to pick up on such dynamics because their memory tasks did not involve this aspect (in a typical experiment, a participant will receive instructions to make an eye movement to a certain location, remember the location for a few seconds, and then execute the movement).

After instructions to make a left- or right-hand turn at the end of a virtual reality corridor, left- or right-turn neurons activate in a specific sequence. Single neurons fall completely silent following a brief activation burst, so that the average activity during the memory delay is low. Nevertheless, the sparse but specific activation sequence is sufficient to predict whether the animal will make a left or right turn at the end of the corridor.

The notion of dynamical memories is particularly interesting to our research because it relates to the idea that memories are an anticipation to act in a certain way (turn left) at a specified place (the end of the corridor) and a specified time (in about 10 seconds) – something we have been exploring in recent papers as well (i.e., research briefing from May 7^th).

The new empirical evidence for dynamic memories now raises the theoretical challenge of showing how the brain is capable of quickly creating new sequences. After all, we are able to remember which way to go within seconds of entering a completely new environment. Another open question, which was not addressed in the article, is whether or not we can use dynamic memories to remember continuous quantities: the receptionist may tell you that the office is 40 feet away. Do you now have an activation sequence remembering ’40 feet’ in the parietal cortex? Is this sequence more similar to the ’30 feet’ sequence than to the ’20 feet’ sequence? Further, when we see a sign in the corridor indicating that the location of the interview has been moved, can we integrate this new information into the ongoing memory sequence? Does it then branch off into a new sequence? The many open questions will direct memory research toward exciting new directions.



Reference:
Harvey CD, Coen P and Tank DW (2012) Choice-specific sequences in parietal cortex during a virtual-navigation decision task. Nature; 484(7392):62-8