A Tale of Two Evils: Bad statistical inference and just bad inference

Evil 1:  Flawed statistical inference

There has been a recent lively debate on the hazards of functional magnetic resonance imaging (fMRI), and what claims to believe or not in the scientific and/or popular literature [here, and here]. The focus has been on flawed statistical methods for assessing fMRI data, and in particular failure to correct for multiple comparisons [see also here at the Brain Box]. There was quite good consensus within this debate that the field is pretty well attuned to the problem, and has taken sound and serious steps to preserve the validity of statistical inferences in the face of mass data collection. Agreed, there are certainly papers out there that have failed to use appropriate corrections, and therefore the resulting statistical inferences are certainly flawed. But hopefully these can be identified, and reconsidered by the field. A freer and more dynamic system of publication could really help in this kind of situation [e.g., see here]. The same problems, and solutions apply to non-brain imaging field [e.g., see here].

But I feel that it may be worth pointing out that the consequence of such failures is a matter of degree, not kind. Although statistical significance is often presented as a category value (sig vs ns), the threshold is of course arbitrary, as undergraduates are often horrified to learn (why P<.05? yes, why indeed??). When we fail to correct for multiple comparisons, the expected probabilities change, therefore the reported statistical significance is incorrectly represented. Yes, this is bad, this is Evil 1. But perhaps there is a greater, more insidious evil to beware.

Evil 2: Flawed inference, period.

Whatever our statistical test say, or do not say, ultimately it is the scientist, journalist, politician, skeptic, whoever, who interprets the result. One of the most serious and common problems is flawed causal inference: "because brain area X lights up when I think about/do/say/hear/dream/hallucinate Y, area X must cause Y". Again, this is a very well known error, undergraduates typically have it drilled into them, and most should be able to recite like mantra: "fMRI is correlational, not causal". Yet time and again we see this flawed logic hanging around, causing trouble.

There are of course other conceptual errors at play in the literature (e.g., there must be a direct mapping between function and structure; each cognitive concept that we can imagine must have its own dedicated bit of brain, etc), but I would argue perhaps that fMRI is actually doing more to banish than reinforce ideas that we largely inherited from the 19th Century. The mass of brain imaging data, corrected or otherwise, will only further challenge these old ideas, as it becomes increasingly obvious that function is mediated via a distributed network of interrelated brain areas (ironically, ultra-conservative statistical approaches may actually obscure the network approach to brain function). However, brain imaging, even in principle, cannot disentangle correlation from causality. Other methods can, but as Vaughan Bell poetically notes:

Perhaps the most important problem is not that brain scans can be misleading, but that they are beautiful. Like all other neuroscientists, I find them beguiling. They have us enchanted and we are far from breaking their spell. [from here]

In contrast, the handful of methods (natural lesions, TMS, tDCS, animal ablation studies) that allow us to test the causal role of brain function do not readily generate beautiful pictures, and perhaps, therefore suffer a prejudice that keeps them under-represented in peer-review journals, and/or popular press. It would be interesting to assess the role of beauty in publication bias...

Update - For even more related discussion, see:
http://thermaltoy.wordpress.com/2012/05/28/devils-advocate-uncorrected-stats-and-the-trouble-with-fmri/
http://www.danielbor.com/dilemma-weak-neuroimaging/
http://neuroskeptic.blogspot.co.uk/2012/04/fixing-science-systems-and-politics.html

The Science in the Middle

When it comes to controversies, science can find itself stuck between GMO doomsayers on the left and climate change deniers on the right. To the former, science may be the evil arm of big business interests, but to the latter, a bunch of left wing saboteurs intent on halting progress and civilisations. Both sides of the argument can be high-jacked for political point scoring.

Today, a united front of self-proclaimed "Geeks in the Park" staged a protest against anti-GMO group "Take the Flour Back" hoping to halt an experimental trial in Harpenden to test a genetically modified wheat crop. It is a pretty fiery debate. Following it live on Twitter today, quite a few inflammatory things were said on both sides.

From a Tweeting Greens Party politician on the anti-GMO side:

The mouth frother is still here. Being debated with. I must say, very brave of him to mix with us. Credit for that.

And from a Tweeting Labour Party politician, on the anti-anti-GMO side:

Have lots of anti-histamines & am tempted to offer them to any sneezing anti-GM protestors. But animal tested

But jokes aside, this is obviously an important issue. Research into how we (i.e., a very large number of people, who show every sign of becoming an ever-larger number!) are going to feed ourselves into the 21st century is probably one of the most pressing issues facing science today. There will be many routes that need to be explored, including genetic modification (following in the tradition of the great 19th century Augustinian friar Gregor Mendel) as well as other agricultural developments (which will also have potential risks and unforeseen side effects - everything does!). Moreover, pending climate change makes this research even more urgent. But here we find science attacked from the other side of the political spectrum. The list of strongly worded claims and counter claims is pretty long, but for a taste of the controversy see here for one scientist's perspective (and ensuing comments) and here for the latest views from climate-change skeptics.