Research Briefing: reward-guided working memory

George Wallis

Research Briefing, by George Wallis

In almost any situation, there are hundreds of things (or ‘stimuli’) that could attract our attention - just count the number of objects you can see from where you are now.  In order to get on with life and avoid total mental chaos, we have to be extremely selective about what we process – most stimuli are essentially ignored.  It is a long-established finding that the number of things we can hold in mind (or hold in ‘working memory’) is really very small – about 2-8 depending on the experiment we run.  Clearly we possess powerful mechanisms that let us filter in only certain stimuli.  The ways in which we can select what gets into working memory is a much-studied topic in psychology, and often psychologists run experiments in which they present ‘cues’ (e.g. arrows) that tell people which items in the experiment to ‘select’.  This is the experimental equivalent of pointing out something with your finger.

Harry Styles of One Direction
giving an attentional cue to the crowd

However, most of the time, this isn’t how we select what gets into working memory: in the real world people aren’t on hand to continuously tell us what to pay attention to.  One real-world factor psychologists think may be important in determining whether an item gets into memory is its ‘reward value’.  For example, a twenty-pound note is more likely to grab our attention than a piece of scrap paper, even if they are about the same size and appearance.  Our paper recently published in Visual Cognition (Wallis, Stokes, Arnold, & Nobre, 2015) describe the results of two experiments we ran in which we looked at how reward value affects the likelihood that a stimulus will get into working memory.

Anderson and colleagues performed experiments showing that items displayed in colours that the experimenters had previously associated with a high monetary reward ‘grab’ attention as people look around a visual scene for a particular target, slowing them down slightly (e.g. Anderson, Laurent, & Yantis, 2011).  We adapted their experiment to look specifically at memory, presenting four nonsense-shapes briefly, and then testing how well people remembered which shapes had been presented a few seconds later.  Before running the memory task we associated some of the shapes with high reward and some with low reward. 

An experimental trial from our shapes experiment

When we asked people to remember four shapes out of which some items were worth more than others, people didn’t remember the high value items any better than the low value items.  This was a surprise!  On the basis of the paper by Anderson, we expected the high value items would be better remembered – after all, they ought to grab attention.  However, we did find a curious effect.  If all of the shapes in a display were high-value (a ‘high value trial’), then any one of them was remembered better than if all the shapes were low value (a ‘low value trial’).  More curious yet, if half the shapes were high value, and half were low value, any shape was remembered about equally well – but they were remembered a little less well than when all the items were high value, and a little better than when all the items were low value.

We reasoned that this could have been because people simply made more effort when the shapes in the memory array were higher in value, on average, and so they did a bit better.  However, a more specific (and interesting) explanation was also possible. We know that a chemical in the brain, dopamine, is involved in processing reward – in studies on monkeys, where the dopamine neurons are measured directly, experimenters see ‘pulses’ of dopamine release when rewarded items are presented to monkeys.  We also know that the prefrontal cortex (PFC) – the part of the brain thought to be most important in controlling working memory, is ‘soaked’ in dopamine: dopamine is released throughout the PFC.  Some have suggested that the dopamine pulses ‘open the gate’ to working memory, and the more dopamine released at a given time, the wider the gate is opened (Braver & Cohen, 2000).

We couldn’t record dopamine firing in our volunteers, so we couldn’t test this possibility directly.  However, it made us wonder – what would happen if instead of showing our memory items all at the same time, we presented them very quickly one after the other, in a row?  If subjects simply made more effort on those trials where they had encountered a higher value item, then we would still expect all of the items we showed to benefit from this.  However, we know that dopamine pulses are only a fraction of a second long (about a third of a second).  If we presented each item for about this length of time, one after the other, then a ‘reward pulse’ might be able to ‘pick out’ the high reward item, and not the other less valuable items.   So, we ran the experiment, with a few adaptations: rather than shapes, we used coloured lines, presented one after the other, and asked people to remember the orientation of the lines.  Certain colours were given high or low reward values.

An experimental trial from our 
second, sequential experiment

We found that indeed, only the high-value item in this experiment was more likely to be encoded, and not its near-neighbours.  This doesn’t prove that dopamine pulses are responsible – we didn’t measure our volunteers’ dopamine neurons – but it does suggest that the reward effect is quite tightly localized in time: a ‘pulse’ tied to an item, not a more general ‘making an effort’ effect.  This was an intriguing finding and it opens up several questions.  Firstly, is this effect really down to dopamine, like we speculate?  To find out, we’d need to see what dopamine neurons are doing at the same time as running the task.  Interestingly, there is some evidence that the diameter of people’s pupils responds rapidly to dopamine release, so maybe measuring pupil diameter would be a way of getting some more evidence without having to get inside the brain. 

Secondly – what’s the point of this rather weird-seeming ‘pulse’ mechanism?  And why would it be useful – like in our first experiment – for unrewarded items to get ‘caught by the pulse’?  Our speculative answer to this question is that our experiment was unnatural – we asked our volunteers to keep staring at the centre of the screen and flashed up the shapes all together, just for a moment.  They had no chance to move their eyes (indeed we deliberately tried to prevent that!).  However, in more natural settings, our eyes constantly flit around the scene.  This is pretty hard to notice in yourself but watch a friend’s eyes for a while (without freaking them out too much) – they jump from place to place continually, ‘fixating’ first this object, then that.  In fact they move about 3 or 4 times per second, jumping from looking at one item to another.  If our putative working-memory-updating pulses were ‘tied’ to these fixations this might provide a mechanism by which the more rewarded items in the scene were more likely to enter memory.

                                                         

References:

Anderson, B. A., Laurent, P. A., & Yantis, S. (2011). Value-driven attentional capture. Proceedings of the National Academy of Sciences, 108(25), 10367–10371. doi:10.1073/pnas.1104047108

Braver, T. S., & Cohen, J. D. (2000). On the control of control: The role of dopamine in regulating prefrontal function and working memory. Control of Cognitive Processes: Attention and Performance XVIII, 713–737.

Wallis, G., Stokes, M. G., Arnold, C., & Nobre, A. C. (2015). Reward boosts working memory encoding over a brief temporal window. Visual Cognition, 23(1-2), 291–312. doi:10.1080/13506285.2015.1013168

Research Briefing: organising the contents of working memory

Figure 1. Nicholas Myers

Research Briefing, by Nicholas Myers

Everyone has been in this situation: you are stuck in an endless meeting, and a colleague drones on about a topic of marginal relevance. You begin to zone out and focus on the art hanging in your boss’s office, when suddenly you hear your name mentioned. On high alert, you suddenly shift back to the meeting and scramble to retrieve your colleague’s last sentences. Miraculously, you are able to retrieve a few key words – they must have entered your memory a moment ago, but would have been quickly forgotten if hearing your name had not cued them as potentially vital bits of information.

This phenomenon, while elusive in everyday situations, has been studied experimentally for a number of years now: cues indicating the relevance of a particular item in working memory have a striking benefit to our ability to recall it, even if the cue is presented after the item has already entered memory. See our previous Research Briefing on how retrospective cueing can restore information to the focus of attention in working memory.

In a new article, published in the Journal of Cognitive Neuroscience, we describe a recent experiment that set out to add to our expanding knowledge of how the brain orchestrates these retrospective shifts of attention. We were particularly interested in the potential role of neural synchronization of 10 Hz (or alpha-band) oscillations, because they are important in similar prospective shifts of attention.

Figure 2. Experimental Task Design. [from Myers et al, 2014]

We wanted to examine the similarity of alpha-band responses (and other neural signatures of the engagement of attention) both to retrospective and prospective attention shifts. We needed to come up with a new task that allowed for this comparison. On each trial in our task, experiment volunteers first memorized two visual stimuli. Two seconds later, a second set of two stimuli appeared, so that a total of four stimuli was kept in mind. After a further delay, participants recalled one of the four items.  

In between the presentation of the first and the second set of two items, we sometimes presented a cue: this cue indicated which of the four items would likely be tested at the end of the trial. Crucially, this cue could have either a prospective or a retrospective function, depending on whether it pointed to location where an item had already been presented (a retrospective cue, or retrocue), or to a location where a stimulus was yet to appear (a prospective cue, or precue). This allowed us to examine neural responses to attention-guiding cues that were identical with respect to everything but their forwards- or backwards-looking nature. See Figure 2 for a task schematic.

Figure 3. Results: retro-cueing and pre-cueing
trigger different attention-related ERPs.
[from Myers et al, 2014]

We found marked differences in event-related potential (ERP) profiles between the precue and retrocue conditions. We found evidence that precues primarily generate an anticipatory shift of attention toward the location of an upcoming item: potentials just before the expected appearance of the second set of stimuli reflected the location where volunteers were attending. These included the so-called early directing attention negativity (or 'EDAN') and the late directing attention-related positivity (or 'LDAP'; see Figure 3, middle panel; and see here for a review of attention-related ERPs). Retrocues elicited a different pattern of ERPs that was compatible with an early selection mechanism, but not with stimulus anticipation (i.e., no LDAP, see Figure 3, upper panel). The latter seems plausible, since the cued information was already in memory, and upcoming stimuli were therefore not deserving of attention. In contrast to the distinct ERP patterns, alpha band (8-14 Hz) lateralization was indistinguishable between cue types (reflecting, in both conditions, the location of the cued item; see Figure 4).

Figure 4. Results: retro-cueing and pre-cueing trigger similar patters
of de-synchronisation in low frequency activity (alpha band at ~10Hz).
[from Myers et al, 2014]

What did we learn from this study? Taken together with the ERP results, it seems that alpha-band lateralization can have two distinct roles: after a precue it likely enables anticipatory attention. After a retrocue, however, the alpha-band response may reflect the controlled retrieval of a recently memorized piece of information that has turned out to be more useful than expected, without influencing the brain’s response to upcoming stimuli.

It seems that our senses are capable of storing a limited amount of information on the off chance that it may suddenly become relevant. When this turns out to be the case, top-down control allows us to pick out the relevant information from among all the items quietly rumbling around in sensory brain regions.

Many interesting questions remain that we were not able to address in this study. For example, how do cortical areas responsible for top-down control activate in response to a retrocue, and how do they shuttle cued information into a state that can guide behaviour? 



Key Reference: 

Myers, Walther, Wallis, Stokes & Nobre (2014) Temporal Dynamics of Attention during Encoding versus Maintenance of Working Memory: Complementary Views from Event-related Potentials and Alpha-band Oscillations. Journal of Cognitive Neuroscience (Open Access)

Research Briefing: Oscillatory Brain State and Variability in Working Memory

Hot off the press: Oscillatory Brain State and Variability in Working Memory

In a new paper, Nick Myers and colleagues show how spontaneous fluctuations in

alpha-band synchronization over visual cortex predict the trial-by-trial accuracy of items stored in visual working memory. The pre-stimulus desynchronization of alpha oscillations correlated with the accuracy of memory recall. A model-based analysis indicated that this effect arises from a modulation in the precision of memorized items, but not the likelihood of remembering them (the recall rate). The phase of posterior alpha oscillations preceding the memorized item also predicted memory accuracy. The study highlights the influence of spontaneous changes in cortical excitability on higher visual cognition, and how these state changes contribute to large amounts of variability in what is normally thought of as a stable aspect of behavior.

 

From Figure 2 in Myers et al. (2014)

Reference: 

Myers, N. E., M. G. Stokes, et al. (2014). "Oscillatory brain state predicts variability in working memory." J Neurosci 34(23): 7735-7743 http://www.jneurosci.org/content/34/23/7735.short

Research Briefing: Attention restores forgotten items to visual short-term memory

Our paper, just out in Psychological Science, describes the final series of experiments conducted by Alexandra Murray during her PhD with Kia Nobre and myself at the Department of Experimental Psychology, Oxford University. Building on previous research by Kia and others in the Brain and Cognition Lab, these studies were designed to test how selective attention modulates information being held in mind, in a format known as visual short-term memory (VSTM).

Typically, VSTM is thought of as a temporary buffer for storing a select subset of information extracted during perceptual processing. This buffer is typically assumed to be insulated from the constant flux of sensory input streaming continuously into the brain, allowing the most important information to be held in mind beyond the duration of sensory stimulation. This way, VSTM enables us to use visual information to achieve longer-term goals, helping to free us from direct stimulus-response contingencies (right).

Previous studies have shown that attention is important for keeping visual information in mind. For example, Ed Awh and colleagues have suggested that selective attention is crucial for rehearsing spatial information in VSTM, just like inner speech helps us keep a telephone number in mind. Our results described in this paper further suggest that attention is not simply a mechanisms for maintenance, but is also important for converting information into a retrievable format.

In long term-memory research, retrieval mechanisms are often considered as important to memory performance as the storage format. It is all well and good if the information is stored, but to what end if it cannot be retrieved? We think that retrieval is also important in VSTM - valuable information could be stored in short-term traces that are not directly available for memory retrieval. In this study, we show that attention can be directed to such memory traces to convert them into a format that is easier to use (i.e., retrieve). In this respect, attention can be used to restore information to VSTM for accurate recall.

We combined behavioural and psychophysical approaches to show that attention, directed to memory items about one second after they had been presented, increases the discrete probability of recall, rather than a more perceptual improvement in the precision of recall judgements (for relevant methods, see also here). This combination of approaches was necessary to infer a discrete state transition between retrievable and non-retrievable formats.

Next step? Tom Hartley asked on twitter: what happened to the unattended items in memory? We did not address this question in this study, and the current literature presents a mixed picture, some suggesting the attention during maintenance impairs memory for unattended items (see), whereas others find no such suppression effect (see). It is possible that differences in strategy could account for some of the confusion.

To test the effect on unattended items in behavioural studies, researchers typically probe memory for unattended items every so often. This presents a contradiction to the participant - sometimes uncued items will be relevant for task performance, therefore individuals need to decide on an optimal strategy (i.e., how much attention to allocate to uncued items, just in case...). A cleaner approach is to use brain imaging to measure the neural consequence for unattended items. The principal advantage is that you don't need to confuse your participants with a mixed message: attend to the cued item, even though we might ask you about one of the other ones!!

References:

Awh & Jonides (2001) Overlapping mechanisms of attention and spatial working memory. TICS (pdf)

Bays & Husain (2008) Dynamic shifts of limited working memory resources in human vision. Science (pdf)

Landman, Spekreijse, & Lamme (2003). Large capacity storage of integrated objects before change blindness. Vision Research (link).

Matsukura, Luck, & Vecera (2007). Attention effects during visual short-term memory maintenance: Protection or prioritization? Perception & Psychophysics (link).

Murray, Nobre, Clark, Cravo & Stokes (2013) Attention Restores Discrete Items to Visual Short-Term Memory. Psychological Science (pdf)

Research Briefing: Lacking Control over the Trade-off between Quality and Quantity in Visual Short-Term Memory

This paper, just out in PLoS One, describes research led by Alexandra Murray during her doctoral studies with Kia Nobre and myself. The series of behavioural experiments began with a relatively simple question: how do people prepare for encoding into visual short-term memory (VSTM)?

VSTM is capacity limited. To some extent, increasing the number of items in memory reduces the quality of each representation. However, this trade-off does not seem to continue ad infinitum. If there are too many items to encode, people tend to remember only a subset of possible items, but with reasonable precision, rather than a more vague recollection of all the items. 

Previously, we and others had shown that directing participants to encode only a subset of items from a larger set of possible memory items increases the likelihood that the cued items would be recalled after a memory delay. Using electroencephalogram (EEG), we further showed that the brain mechanisms associated with preparation for selective VSTM encoding were similar to those previously associated with selective attention. 

To follow up on this previous research, Murray further asked whether can people strategically fine tune the trade-off between the number and quality of items in VSTM? Given foreknowledge of the likely demands (i.e., many or few memory items, difficult or easy memory test), can people engage an encoding strategy that favours quality over quality, or vice versa?  

From the outset, we were pretty confident that people would be able to fine-tune their encoding strategy according to such foreknowledge. Extensive previous evidence, including our own mentioned above, had revealed a variety of control mechanisms that optimise VSTM encoding according to expected task demands. Our first goal was simply to develop a nice behavioural task that would allow us to explore in future brain imaging experiments the neural principles underlying preparation for encoding strategy, relative to other forms of preparatory control. But this particular line of enquiry never got that far! Instead, we encountered a stubborn failure of our manipulations to influence encoding strategy. We started with quite an optimistic design in the first experiment, but progressively increased the power of our experiments to detect any influence of foreknowledge over expected memory demands - and still nothing at all! The figure on the right summarises the final experiment in the series. The red squares in the data plot (i.e., panel b) highlight the two conditions that should differ if our hypothesis was correct.  

By this stage it was clear that we would have to rethink our plans for subsequent brain imaging experiments. But in the interim, we had also potentially uncovered an important limit to VSTM encoding flexibility that we had not expected. The data just kept on telling us: people seem to encode as many task-relevant items as possible, irrespective of how many items they expect, or how difficult the expected memory test at the end of the trial. In other words, this null effect had revealed an important boundary condition for encoding flexibility in VSTM. Rather than condemn these data to the file draw, shelved as a dead-end line of enquiry, we decided that we should definitely try to publish this important, and somewhat surprising null effect. We decided PLoS One would be the perfect home for this kind of robust null effect. The experimental designs were sensible, with a logical progression of manipulations, the experiments were well-conducted and the data were otherwise clean. There was just no evidence that our key manipulations influenced short-term memory performance. 

As we were preparing our manuscript for submission, a highly relevant paper by Zhang and Luck came out in Psychological Sciences (see here). Like us, they found no evidence that people can strategically alter the trade-off between remembering many items poorly and/or few items well. If it is possible to be scooped on a null effect, then I guess we were scooped! But in a way, the precedent only increased our confidence that our null effect was real and interesting, and definitely worth publishing. Further, PLoS One is also a great place for replication studies, and so surely a replication of a null effect makes it a doubly ideal! 

For further details, see:

Murray, Nobre & Stokes (2012) Lacking control over the trade-off between quality and quantity in VSTM. PLoS One

Murray, Nobre & Stokes (2011). Markers of preparatory attention predict visual short-term memory. Neuropsychologia, 49:1458-1465.

Zhang W, Luck SJ (2011) The number and quality of representations in working memory. Psychol Sci. 22: 1434–1441