'It's not a crisis, it's a reformation'

One of the founding fathers of the replication crisis-turned-reformation, Professor Brian Nosek, opened the 2018 British Psychological Society Annual Conference with a startling picture of how our science has changed in a few short years. Co-founder and Director of the Centre for Open Science, Nosek was named in Nature’s 2015 list of the ten most influential scientists.

Nosek permeated the broader consciousness of the scientific community after his 2015 paper in Science which outlined attempts to replicate 100 psychology studies – many of which failed. But, he said, psychology isn’t in crisis: in fact it is leading the way for better and more robust research throughout science as a whole.  

He pointed to a survey in Nature which asked individuals from chemistry, biology, physics and many other sciences whether there was a problem with reproducing experiments in their field. More than half said there was, and that it was a significant crisis. Yet these are social and cultural issues, Nosek said, and psychology is in a particularly strong position to inform the ways science and scientists can change. We have turned our own knowledge to face the ways we do science and the biases inherent in the system. Psychology can tell us how our practices can be shaped and nudged, so the values that should inform research are embedded in our everyday practice.  

Nosek pointed to the norms of science, or the values scientists should have in the motivations behind their research. These are communality, or a willingness for open sharing of work and data; universalism, in terms of evaluating research on its own merit; disinterestedness, in terms of a motivation to learn and discover; organised scepticism, in terms of considering all new evidence, even that which is against one’s own findings; and quality. While surveys show researchers endorse all of these norms, they admit their own practice doesn’t necessarily reflect them. When asked about the culture around them, however, most say they see the complete opposite in many of their colleagues’ motivation. ‘Counternorms’ are at play, in terms of secrecy, particularlism, self-interestedness, dogmatism, and quantity over quality. This, Nosek said, is an important gap.

Why is there a gap between our perception of scientific norms and the practices we see? And how can that culture, and incentive structure behind it, change to make scientific practices more in-line with the values behind being a scientist?

Nosek said the incentive structure in science has made publishing the main reward – we focus on getting it published over getting it right. That leads to all kinds of questionable behaviours, such as ‘p-hacking’, and telling a ‘tidy, beautiful narrative’ over a robust one.

But this can change, he added, with two simple practices, both of which we learn in our earliest scholastic experiences – show your work, and share. With this approach science will be self-correcting. If your claims and data are openly out in the literature, and are truly credible, they can be properly validated and critiqued.  

This does present the need for change across all kinds of domains – within publishing, learned societies such as the BPS, universities and funders. We need technology and infrastructure to make open science practices possible and easy to do, so that they become the norm, and are incentivised. We could even add policy where necessary to make these practices mandatory (a particularly useful strategy for the ‘laggards’ in any attempted change of a research culture). 

Indeed, much of this large-scale change has been happening for several years. Nosek’s Centre for Open Science also started the OSF – a framework for scientists to pre-register their study protocols and share data. Around 8,000 journals have also signed up to the guidelines to better promote transparency.  

Some journals have also added incentives for open science, including the use of badges on published articles which have shared data, materials and/or pre-registered their study. While Nosek said the badges are ‘silly’ in and of themselves, the practices they represent are far from it. And they work. Before introducing them around 3 per cent of studies in Psychological Science shared their data; by early 2015 this number had soared to 39 per cent.

Pre-registration, whereby researchers publish their study design, methods and intended statistical analyses online, has also become popular. Thanks to incentives such as the pre-registration challenge, which offered cash rewards to researchers who published results after pre-registering, standard practice may be shifting.   

There is evidence more and more researchers are adopting these practices. The OSF has more than 90,000 users, and now stores almost 11,000 pre-prints of journal articles. In 2012 the OSF recorded just 38 pre-registered studies… by 2017 this had reached 12,090. More than 100 journals now accept registered reports.  

Nosek said most of his presentations these days are to non-psychology audiences, and the change in culture within our field is beginning to have influence in other disciplines. ‘This isn’t a crisis – there’s been recognition of these behaviours since the 1960s and 70s… but now we’re stepping up as a community to try to shift our norms and practices. It’s not lost on the scientific community that we’ve identified ways to be self-critical in productive ways… and some of these methods can be exported elsewhere.’ 

- View Nosek’s slides. 

http://improvingpsych.org

Find much more on replication, including Brian Nosek's work, in our archive.