Journal Club: Decoding spatial activity patterns with high temporal resolution

by Michael Wolff

on: Cichy, Ramirez and Pantazis (2015) Can visual information encoded in cortical columns be decoded from magnetoencephalography data in humans? NeuroImage

Knowing what information the brain is holding at any given time is an intriguing prospect. It would enable researchers to explore how and where information are processed and formed in the brain, as well as how they guide behaviour.

A big step towards this possibility was made in 2005 when Kamitani and Tong decoded simple visual grating stimuli in the human brain using functional magnetic resonance imaging (fMRI). The defining new feature of this study was that instead of looking for differences in overall activity levels between conditions (or in this case visual stimuli), they tested the differences in activity patterns across voxels between stimuli. This method is now more generally known as multivariate pattern analysis (MVPA). A classifier (usually linear) is trained on a subset of data to discriminate between conditions/stimuli, and then tested on the left-out data. This is repeated many times, and the percentages of correctly labelled test data are reported. Crucially, this process is carried out separately for each participant, as subtle individual differences in activity patterns and cortical folding would be lost when averaged, defeating the purpose of the analysis. MVPA has since revolutionised fMRI research and, in combination with the increased power of computers, has become a widely used technique.

The differential brain patterns observed by Kamitani and Tong are thought to arise from the orientation columns in the primary visual cortex (V1), discovered by Hubel and Wiesel more than 50 years ago. They showed that columns contain neurons that are excited differentially by visual stimuli of varying orientations. Since these columns are very small (<1 mm) it is surprising that their activity patterns can apparently be picked up by conventional fMRI with about 2-3mm spatial resolution. More surprising still is that even magnetoencephalography (MEG) and electroencephalography (EEG) seem to be able to decode visual information, which are generally considered to have a spatial resolution of several centimetres! How is this possible?

Critics have raised alternative possible origins of the decodable patterns, which could result in more coarse-level activity patterns (e.g. by global form properties or overrepresentation of specific stimuli), and thus confound the interpretation of decodable patterns in the brain.

In response to these criticisms, a recent study by Cichy, Ramirez, and Pantazis (2015) investigated to what extent specific confounds could affect decodable patters by systematically changing the properties of presented stimuli. They used MEG as the physiological measure instead of fMRI. This enabled them to explore the time-course of decoding, which can be used to infer at which visual processing stage decodable patterns arise.

In the first experiment they showed that neither the cardinal bias (over representation of horizontal or vertical gratings) nor the phase of gratings (and thus local luminance) is necessary to reliably decode the stimuli.

Figure 1. From Cichy et al., in press

As can be seen from the decoding time-course the decodability is significant approximately 50 ms after stimulus presentation and ramps up extremely quickly, peaking at about 100 ms. This time-course alone, which was very similar in the other experiments testing for different possible confounds, suggests that the decodable patterns arise early in the visual processing pathway, probably in V1.

The other confounds that were tested involved the radial-bias (neural overrepresentation of lines parallel to fixation), the edge effect (gratings could be represented as ellipses elongated in the orientation of the gratings), and global form (where gratings are perceived as coherent tilted objects). None of these biases could fully explain the decodable patterns, casting doubt on the notion of coarse-level driven decoding. Again, how is this possible, when the spatial resolution of MEG should be far too coarse to pick up such small neural differences?

The authors tested the possibility of decoding neural activity from the orientation columns with MEG more directly. They projected neurophysiologically realistic activity patterns on to the modelled surface of V1 of one subject (A). The distance between each activity node was comparable to the actual size of the orientation columns. The corresponding MEG scalp recordings were obtained by forward modelling (B) and their differences decoded (C and D). The activity patterns could be reliably discriminated across a wide range of signal to noise ratios (SNR) and, most crucially, at the same SNR as in the first experiment.

Figure 2. From Cichy et al., in press

This procedure nicely demonstrates the theoretical feasibility of discriminating neural activity at V1 with MEG, and suggests that the well-known “inverse-problem” inherent to MEG and EEG source localisation does not necessarily mean that small activation differences on the sub-millimetre scale are not present in the activation topographies. While it remains impossible to say where the origin of a neural activation pattern lies, the activation pattern of MEG is still spatially rich.

Even with EEG it is possible to decode the orientations of gratings (Wolff, Ding, Myers, & Stokes, in press); and this can be observed more than 1.5 seconds after stimulus presentation. We believe that there is a bright future ahead for EEG and MEG decoding research: not only is EEG considerably cheaper than fMRI, but the time-resolved decoding offered by both methods could nicely complement the more spatially resolved decoding of fMRI.



References

Cichy, R. M., & Pantazis, D. (in press). Can visual information encoded in cortical columns be decoded from magnetoencephalography data in humans? NeuroImage.

Hubel, D. H., & Wiesel, T. N. (1959). Receptive fields of single neurones in the cat's striate cortex. The Journal of physiology, 148(3), 574-591.

Kamitani, Y., & Tong, F. (2005). Decoding the visual and subjective contents of the human brain. Nature Neuroscience, 8(5), 679-685.

Wolff, M. J., Ding, J., Myers, N. E., & Stokes, M. G. (in press). Revealing hidden states in visual working memory using EEG. Frontiers in Systems Neuroscience.

Research Briefing: Dynamic population coding for flexible cognition

Dynamic population coding in prefrontal cortex

Our environment is in constant flux. At any given moment there could be a shift in scenario that demands an equally rapid shift in how we interpret the world around us. For example, the meaning of a simple traffic light critically depends on whether you are driving to work or travelling on foot. Our brains must constantly adapt to accommodate an enormous range of such possible scenarios - in this study, we applied new analysis tools to explore how patterns of brain activity change for different task contexts, allowing for flexible cognitive processing (in Stokes et al., 2013, Neuron; see also Comment by Miller and Fusi in the same issue).

Prefontal Cortex

Adapted from Fig 1

We focused our investigation on an area in the frontal lobe known as lateral prefrontal cortex. This brain area has long been implicated in flexible cognitive processing. Damage to prefrontal cortex is classically associated with reduced cognitive flexibility (Luria, 1966) as part of a more general dysexecutive syndrome. In studies using functional magnetic resonance imaging (fMRI), lateral frontal cortex is also usually more active when participants perform tasks that demand cognitive flexibility (Wager et al., 2004). It it widely believed that prefrontal cortex is especially important for representing information about our environment and task goals in mind for guiding flexible behaviour (Baddeley, 2003; Miller, 2000).

Dynamic coding population coding

Dynamic trajectory through state-space

In this study, we observe a highly dynamic process underlying flexible cognitive processing using a statistical approach that allows us to decode the patterns of population-level activity in prefrontal cortex at high temporal resolution. During a task that requires a different stimulus-response mapping according to trial-by-trial instruction cues (see Fig 1), we found that the pattern of activity rapidly changes during processing of the instructive cue stimulus. After this complex cascade through activity state-space (for more info, see Stokes, 2011), overall activity levels return to baseline for the remainder of a delay period spanning the instruction cue and a possible target stimulus.

Adapted from Fig 5

However, the effect of the cue response lingers on. Subsequent stimuli elicit a population response that critically depends on the previous cue identity. In other words, the dynamic population response triggered by the cue stimulus shifts the response profile of the network of prefrontal cells. This shift in tuning profile allows us to decode the current task-rule (i.e., cue indentify) based on a simple driving stimulus (i.e., neutral stimuli, see Fig 5).

Adapted from Fig 6

More importantly, the shift in the network response profile could also underlie task-dependent target processing (i.e., choice stimuli, see Fig 6). The population response to potential target stimuli rapidly evolved from a stimulus-specific coding scheme, to a more abstract code that distinguishes only between different target and non-target items. This dynamic tuning property is ideal for flexible cognition (Duncan, 2001).

Putative mechanism: flexible connectivity

The flow of brain activity critically depends on the pattern of connections between neurons. Contrary to intuition, these connections are always changing. The pattern of connections that make up the very essence of personal experience is constantly adjusting and adapting to the myriad changes experienced throughout life.

Synaptic Plasticity [wiki commons]

Extensive research focuses on long-term structural changes in connectivity through synaptic plasticity, however the rapid changes we experience from moment-to-moment requires a more flexible kind of memory that can represent the transient features of a given scenario. This kind of flexible "online" memory is typically referred to as ‘working memory’.

It has long been assumed that working memory is maintained by keeping a specific thought in mind, like a static snapshot of a visual image or an abstract goal such as ‘turn left at the next set of lights’. However, more recent evidence suggests that working memory can also be stored by laying down specific, but temporary neural pathways (e.g., Mongillo, Barak & Tsodyks, 2008). Neural pathways are formed by synaptic connections. In a comprehensive review of the literature on short-term synaptic plasticity, Zucker (1989) writes: “Chemical synapses are not static. Postsynaptic potentials wax and wane, depending on the recent history of presynaptic activity”. Short-term plasticity could provide a key mechanisms for flexible connectivity that is necessary for rapid, but temporary changes in network behaviour.

This new idea allows for a more dynamic theory of brain function, which is more consistent with the everyday experience of continuous thought processes that seem to evolve through time, rather than persist as a static representation. We suggest that short-term plasticity could help explain our data:

Adapted from Fig 7

The initial instruction cue stimulus establishes a specific (but temporary) connectivity state during the most active phase of the response. This would explain why the pattern constantly changes - if the synapse are constantly changing, then even identical input to the system will result in constantly shifting output patterns (Buonomano and Maass, 2009). This temporary shift in the response sensitivity of the prefrontal network allows the identity of previous input to be decoded by the patterned response to subsequent input, consistent with the silent memory hypothesis. Finally, dynamic changes in connectivity could also be used to rapidly shift the tuning profile of the prefrontal network to accommodate changes in what specific stimuli mean for behaviour (see Fig. 7).

Broader implications

Brain activity is inherently non-stationary - the continuity/stability of cognitive states are unlikely to depend on static activity states, but rather rapid changes in temporary connectivity patterns. This research also raises the intriguing possibility that cognitive capacity limits are not so much constrained by the sheer amount of information that we can keep in mind, but rather how we can put that information to use. Further research in our lab will explore these exciting possibilities.


Reference:

Stokes, Kusunoki, Sigala, Nili, Gaffan and Duncan (2013). Dynamic Coding for Cognitive Control in Prefrontal Cortex. Neuron, 78, 364-375 [here]

Also see coverage: Miller Lab (MIT), Neuron Preview


Other literature cited:

Baddeley, A. (2003). Working memory: looking back and looking forward. Nat. Rev. Neurosci. 4, 829–839. [here]

Buonomano, D.V., and Maass, W. (2009). State-dependent computations: spatiotemporal processing in cortical networks. Nat. Rev. Neurosci. 10, 113–125. [here]

Luria, A.R. (1966). Higher Cortical Functions in Man (New York: Basic Books).

Miller, E.K. (2000). The prefrontal cortex and cognitive control. Nat. Rev. Neurosci. 1, 59–65. [here]

Mongillo, G., Barak, O., and Tsodyks, M. (2008). Synaptic theory of working memory. Science 319, 1543–1546. [here]

Wager, T.D., Jonides, J., and Reading, S. (2004). Neuroimaging studies of shifting attention: a meta-analysis. Neuroimage 22, 1679–1693. [here]

Zucker (1989) Short-term synaptic plasticity. Ann. Rev. Neurosci, 12: 13-31 [here]

Journal Club:Twists and turns through memory space

By Nick Myers

You enter an unfamiliar building for a job interview. The receptionist tells you to make a left turn at the end of the corridor to get to your interviewer’s office. Easy instructions, but your brain has to remember them nonetheless. For the past decade, theoretical neuroscientists have proposed that, to do this job, neurons in the parietal cortex act as a kind of memory container: once you have learned that you need to make a left, dedicated ‘left-turn’ neurons are persistently active until you have reached the end of the hall, have turned, and can forget about it again. In addition to having lots of supporting evidence and enjoying intuitive appeal, the memory-container model has the advantage that, once the appropriate neurons are activated, they can potentially hold on to the ‘left-turn’ memory indefinitely (for instance, allowing you to get a drink of water before heading to the office).

However, a recent paper in the journal Nature has added to a growing list of evidence contradicting this model. In the paper, Princeton researchers Christopher Harvey, Philip Coen, and David Tank describe how ‘left-turn’ neurons in the parietal cortex of mice fire in a stereotypical cascade as the animals navigate along a virtual-reality corridor. The sequence begins with a small number of ‘left-turn’ neurons activating the next group and then falling silent again (see image), while the new group in turn activates yet another subset, and so forth until the end of the cascade is reached at the end of the corridor. In contrast to the memory-container model, this kind of dynamic activation sequence could be more similar to your car’s sat nav, constantly keeping you up to date on when you will have to turn left. Like a sat nav, dynamic memories could become more prominent when you are navigating through a complicated environment and have to make a left turn at the right time or in the right place (say, for instance, that there are lots of possible left turns, and you must remember to turn behind the drink fountain). Perhaps previous researchers may have failed to pick up on such dynamics because their memory tasks did not involve this aspect (in a typical experiment, a participant will receive instructions to make an eye movement to a certain location, remember the location for a few seconds, and then execute the movement).

After instructions to make a left- or right-hand turn at the end of a virtual reality corridor, left- or right-turn neurons activate in a specific sequence. Single neurons fall completely silent following a brief activation burst, so that the average activity during the memory delay is low. Nevertheless, the sparse but specific activation sequence is sufficient to predict whether the animal will make a left or right turn at the end of the corridor.

The notion of dynamical memories is particularly interesting to our research because it relates to the idea that memories are an anticipation to act in a certain way (turn left) at a specified place (the end of the corridor) and a specified time (in about 10 seconds) – something we have been exploring in recent papers as well (i.e., research briefing from May 7^th).

The new empirical evidence for dynamic memories now raises the theoretical challenge of showing how the brain is capable of quickly creating new sequences. After all, we are able to remember which way to go within seconds of entering a completely new environment. Another open question, which was not addressed in the article, is whether or not we can use dynamic memories to remember continuous quantities: the receptionist may tell you that the office is 40 feet away. Do you now have an activation sequence remembering ’40 feet’ in the parietal cortex? Is this sequence more similar to the ’30 feet’ sequence than to the ’20 feet’ sequence? Further, when we see a sign in the corridor indicating that the location of the interview has been moved, can we integrate this new information into the ongoing memory sequence? Does it then branch off into a new sequence? The many open questions will direct memory research toward exciting new directions.



Reference:
Harvey CD, Coen P and Tank DW (2012) Choice-specific sequences in parietal cortex during a virtual-navigation decision task. Nature; 484(7392):62-8