Bold predictions for good science

Undergraduates are taught proper scientific method. First, the experimenter makes a prediction, then s/he collects data to test that prediction. Standard statistical methods assume this hypothesis driven approach, most statistical inferences are invalid unless this rigid model is followed.

But very often it is not. Very often experimenters change their hypotheses (and/or analyses methods) after data collection. Indeed, students conducting their first proper research project are often surprised by this 'real-world' truth: "oh, that is how we really do it!". They learn to treat research malpractice like a cheeky misdemeanour. 

After recent interest in science malpractice, fuelled by revelations of outright fraud, commentators are starting to treat the problem more seriously, especially in psychology and neuroscience. This month, Perspectives on Psychological Science devoted an entire issue to the problem of peer-reviewed results that fail to replicate, because they were born of bad scientific practice. 

Arguably, scientific journals share much of the responsibility for allowing bad research practices to flourish. Although there maybe little journals can do to stop outright fraud, they can certainly do a lot to improve research culture more generally. Recently, the journal Cortex has announced that it will try to do just that. Chris Chambers, associate editor, has outlined a new submission format that will strictly demand that researchers conform to the classic experimental model: predictions before data. With the proposed Registration Report, authors will be required to set out their predictions (and design/analysis details) before they collect the data, thus cutting off the myriad opportunities to capitalise on random vagaries in observed data. And although researchers could still lie and make up data, cleaning up the grey area of more routine bad behaviour could have important knock on effects. As I have argued elsewhere, bad scientific practice is presumably a fertile breeding ground for more serious acts of fraud.

This is a bold new initiative, and if successful, could precipitate a major change in the way science is done. For further details, and some interesting discussion, check out this panel discussion on Fixing the Fraud at SpotOn London 2012 and this article in the Guardian.