Psychologist logo

‘Every discipline does something well, and there’s a lot we can learn’

The UK Reproducibility Network is a peer-led consortium to investigate factors which contribute to robust research, provide training, and disseminate best practice…

31 January 2019

We spoke to one of its founders, psychologist Professor Marcus Munafo (University of Bristol), along with other stakeholders. 

Why has psychology led the reproducibility charge?  

Issues around reproducibility are related to current incentives in academic research being misaligned – what is good for scientists is not necessarily good for science. To improve quality we need to understand these influences on the behaviour of scientists, and work out how to change them (and therefore those behaviours). Psychologists are well placed to lead on these issues, since our business is understanding behaviour.

Is psychology still ‘in crisis’, or are we now over the hump?

I’ve never really liked the ‘crisis’ narrative. It perhaps served a purpose early on in raising awareness of the issues, but now I think we’re in a much more exciting phase of exploring how we can improve our ways of working. But we’re still at an early stage when it comes to changing the incentive structures that we work within, for example.

What’s the main practical role you see for the UK Reproducibility Network?

Hopefully it will serve to support individual researchers and groups of researchers, as well as relevant professional services staff (e.g. library services) at individual institutions, for example by helping them to set up local activities and working groups, and providing training and resources.
As well as this, we want to coordinate activity between these individuals and groups, the institutions themselves, and key stakeholders (e.g. funders and publishers). There’s a lot of activity intended to improve reproducibility, but this will be more effective if its coordinated.

Why is a multi-disciplinary approach to reproducibility so important?  

Some of the issues are relatively universal and apply across disciplines; others are more specific to individual disciplines. One thing I’ve learned from working across different areas is that every discipline does something well, and there’s a lot we can learn simply by sharing best practice and solutions that have already been identified and implemented somewhere.

Science reform can be seen as a behaviour change problem. Of all the ‘stakeholders’, who might be the hardest to nudge? 

I suspect more established researchers might, as a group, be slowest to adopt new ways of working, simply because they’ve become used to the current system and (by definition) have done reasonably well within it. Perhaps understandably, changing how they work might be perceived as onerous and potentially a threat to their continued success. Institutional change can also be quite slow, so this may take longer than, for example, encouraging change among individual researchers.

Are the people who really need to join the UKRN precisely those who won’t? 

I think this would be a danger if it was simply a peer-led initiative working at the grassroots. However, by making sure we’re working with institutions and stakeholders the hope is that we will be able to work at different levels (bottom up and top down, if you like), and engage the wider community that way. For example, if funders and publishers change the incentives (what you need to do to get a grant or to publish, for example), that will encourage a change of behaviour at multiple levels (individuals and institutions), and UKRN will be there to provide support and training, as well as to evaluate whether those incentives are having the intended consequences (and no unintended ones!).

Is the reproducibility movement really penetrating those grassroots? 

In my experience the most engaged group are the early career researchers. In part I think that’s because they are acutely aware of the pressure to, for example, publish and get grants in order to succeed. But it’s also because they see science through fresh eyes and can see how curious and in many ways antiquated our ways of working are.

How are we looking to the outside?  

I think there’s a very positive framing we can put on this. Science continues to make advances, but we’re now engaged in an ongoing process of improvement to ensure that it continues to do so, that investment in science continues to represent good value for money, and so on. This isn’t a one-off event; we should always be looking at how we work and whether we could be doing better.

Do you have any messages for our practitioner readers, who might feel that this whole debate is of less relevance to them?

Ultimately, we want to make sure that science produces robust evidence. We know that the evidence for the efficacy of drug treatments, for example, is influenced by publication bias and other biases. It seems that the same is also true for psychological treatments. If the evidence for a particular treatment is not as robust as the published literature might have us believe, then practitioners need to know this.

If the current appetite for reform at a national level is sustained, where do you see us in 20 years’ time?  

I would like to see meta-research – research into factors that influence the quality of research outputs – mature into a distinct discipline that cuts across traditional disciplinary boundaries, and provides evidence to inform how science (and scientists) should work. I’d also like to see UKRN as an independent body that provides advice and training to UK institutions, informed by this evidence.

Connect with the UKRN via email [email protected] and @ukrepro on Twitter

 

‘The issue for the library community that has been raised by the reproducibility debate is whether what we are spending all this money on has value? Is the scholarly record as valuable as we think it is? Is peer review sufficient to ensure that what we purchase and make available is actually good research? We want to make sure that what we are paying for is of quality and helps to advance scholarship.’

David C Prosser, Executive Director, Research Libraries UK

‘We’ve made huge strides over the last couple of years and I truly believe that psychology is undergoing a renaissance. Science is a collection of behaviours, and I argued in my Royal Society talk last January that we should use well established behaviour change approaches (such as the COM-B) to help move psychological science forward. We should also recognise that change needs to happen at all levels: addressing questionable research practices (p-hacking, HARKing etc.) as well as the research culture and incentive structure (i.e., publish or perish).

The UKRN is an excellent initiative which has the potential to change grassroots approaches to promoting open science, as well as to help change the scientific incentive structure by influencing Universities, funders and other relevant agencies.’

Professor Daryl O’Connor, Chair of the British Psychological Society’s Research Board

‘Reproducibility in research impacts across all disciplines. Having a network which brings together people from a wide range of institutions and variety of subjects to consider how reproducibility in research can be addressed, and to develop good procedures, practices and processes that improve the reliability of research, is a very positive step forward.  

This is not a new situation, but the research landscape is becoming ever-more complex. New areas of research are emerging, and we are seeing more multi-discipline and multi-method research, with large amounts and multiple types of experimental data being collected and analysed in many studies. It is essential to also have a multi-disciplinary approach to reproducibility.’

JT Janssen, Director of Research at the National Physical Laboratory

‘Many of the issues around reproducibility are common across a range of research disciplines. At the heart are questions of good experimental design and appropriate and robust statistical analysis.

Meaningful change will require all stakeholders to be engaged. But more importantly, the different groups are influenced by one another. For example, the questions have been raised by researchers themselves, which has resulted in funders and publishers becoming more aware of the issue. Changing funder and publisher policies will then raise awareness of more researchers. So, there’s the potential for a virtuous cycle… a researcher-led initiative like the UKRN has an important role to play.’

Steven Hill, Director of Research, Research England

‘Impact is a massive issue for publicly funded charities and we cannot possibly hope to have impact if the work we fund is not reproducible. Currently it takes an average of 17-years for new basic science discoveries to reach the clinic. That will be much faster if more work is reproducible. 

Junior researchers are the ones most motivated by the reproducibility movement. People go into science wanting to discover new things, and often to also help particular groups. They want their work to be reproducible and are highly motivated to ensure it is. Many have also been caught out by working to replicate something from a senior researcher that turns out not to be reproducible (as I was – it’s one of the reasons I left frontline science). That actually gives me hope about the longevity of the reproducibility movement, as one day these idealistic young researchers will become senior leaders – as long as they can produce reproducible work within the current incentive structures.

It’s a shared problem – no one discipline is worse off. Institutions – especially big names - are the ones who most need to be at UKRN’s centre. They need to incentivise senior researchers to ensure that their own work is reproducible, and that they are developing a cadre of junior scientists who have fully adopted reproducible practices. Incentives is the biggest issue that will lead to behaviour change becoming mainstream, so any body with capacity to give incentives needs to be at the table.

In my field, a lot of service-delivery based charities do research in house, but many of those researchers do not have the benefit of a climate where methods are discussed and developed. I’d love to see researchers outside mainstream academia – in charities, research institutes, or think tanks, whether service implementation or theoretical in focus – understand the importance of the issues, and importantly be equipped to conduct their work in a reproducible way.’

Sarah Shenow – MQ: Transforming Mental Health