My lost finding

Academic research can be a tough, messy business. Add in the wonderful complexity of the people that are usually the subject matter of psychology and it's little wonder that a career can be a path of wrong turnings, forking paths and dead ends. The 'replication crisis' of recent years has renewed attention on this unseen or unloved data: what's in the file drawer, down the back of the sofa, or out there but underappreciated? For want of a better word, what's the significance of it all? 

We want to hear from you about your 'lost finding'. Perhaps you've never shared it with anyone. Perhaps it was an 'ugly' initial finding which you're prepared to admit you transformed into something beautiful before unleashing it on the world (the 'chrysalis effect'). Perhaps you published it only to see it lie dormant before something woke it (a 'sleeping beauty' paper). Perhaps it's still dormant, unloved, and you can't work out why.

Speaking personally, there's a whole load of data I collected while at Glasgow Caledonian University which never got written up because I jumped ship to this job. And there's a published finding on how children, boys in particular, underplay or hide their academic effort, which I've always felt deserved as much attention as my bullying and social cognition stuff.

We'll be calling for contributions via @psychmag on Twitter using #MyLostFinding - but you can also email me with your thoughts on [email protected]. No more than 400 words, written in a style that's going to engage and inform our large and diverse audience of professional psychologists. We're hoping to collect enough of these to publish a selection in The Psychologist, which goes to 50,000+ members of the British Psychological Society each month. Can an exploration of what has been lost reveal something about the best way forward?

Dr Jon Sutton
Managing Editor

Thanks to Dr Tom Stafford from the University of Sheffield for getting the ball rolling!

'Our paper fell through a disciplinary crack'

Hidden between pages 267 to 272 of Behavioural Brain Research (volume 243) is the paper I’m proudest of. While I love it to an unreasonable level, it is a result which was difficult to publish and hasn’t had much attention since publication either.

The reason I’m so proud of paper is it represents the kind of science I aspire to – a carefully crafted behavioural experiment which directly verifies a prediction, a prediction based in computational analysis of neural circuits.

The neural circuits are those underlying reward and habit learning, circuits which rely on that famous neurotransmitter dopamine. Pete Redgrave, head of our neuroscience group, and Kevin Gurney, professor of computational neuroscience, had a theory that dopamine signals could support learning new actions (contrary to the dominant theory, which is that dopamine signals the value of events).

The key to our experiment was the realisation that signals that first trigger dopamine release come from a subcortical area called the colliculus, and the colliculus only gets some visual information from the eye. In particular, it is insensitive to colour changes which affect the short-wave photoreceptors. What this means you can design a stimulus that is invisible to the colliculus (or at least invisible until the information arrives via visual cortex, which can detect such changes).

In practice, this stimulus looks like a kind of mauve, but it has to be tailored for each individual, since everyone has a different distribution of colour photoreceptors. This takes about an hour of delicate psychophysical measurement, before you even start the main experiment. Fortunately we worked with Martin Thirkettle who had the expertise – and patience – to implement all this.

After months and months of arduous testing (for Martin and the participants) we had a result, and it turned out just like the theory predicted – action acquisition is slowed if the information that signals a correct actions is delivered via one of these stimuli specifically crafted to match the insensitivity of the colliculus.

It’s a dramatic confirmation of a prediction which requires both a computational analysis and a detailed knowledge of the specific neural circuits involved. The understanding of colour perception and psychophysics builds on over one hundred years of experimental psychology. It was a true multidisciplinary team effort, and funded by the EU incidentally.

We were excited to try and publish, but that excitement turned sour as we clocked up rejection after rejection. I learnt that the work you put in doesn’t automatically translate into kudos from other people. I could see how difficult it had been to run the experiments, and how neat the result was, but to many editors it obviously seemed a bit…meh. 'The actual results are rather meagre', said one reviewer. Meagre?! Years of planning and experiment design, months and months of testing! And meagre!? One journal found us two reviewers, one of whom said they could review the psychophysics, but didn’t know about the neuroscience; the other said they could review the neuroscience but not the psychophysics. They couldn’t put the pieces together, and neither could the editor, so our paper fell through a disciplinary crack. With hindsight, I only blame ourselves for this – the rejections made us hone and hone the presentation of the result, so that not only was what we did clear, but what we thought it meant.

Dr Tom Stafford
University of Sheffield

Thirkettle, M., Walton, T., Shah, A., Gurney, K., Redgrave, P., & Stafford, T. (2013).The path to learning: Action acquisition is impaired when visual reinforcement signals must first access cortex. Behavioural Brain Research, 243, 267–272. doi:10.1016/j.bbr.2013.01.023

(no paywall link http://eprints.whiterose.ac.uk/83640/1/Thirkettle2013_postprint.pdf)

 

Lost, but in a good cause

There is something relentless about the grant-driven research business these days. The very idea that some findings will not be published does not fit its business model. What if we had a research culture that borrows from creative play? Here ideas flow constantly, and even if they seem brilliant when shiny and new, only few are good enough to keep. It’s unlikely that all research findings are worth talking about. If in doubt, the filing drawer might be a good place for them. 

Do you, like me, hardly manage to read the abstracts let alone the published papers in your field? Do you find that it is rare to read something that makes you sit up and listen? Do you feel like having to constantly sift through tedious dross when you read the literature? If so, I am sure it will have occurred to you that some of your own findings might be part of the dross. 

Here I would like to make a radical suggestion. When there is overproduction of goods, there often is a reward for ‘set-aside’. It sounds drastic, and does not fit a frugal model of the market. In our current research culture, I doubt if we have a frugal model, but we pretend we do. In grant applications, we justify our research as not only worthwhile, but also of crucial importance, and in pressing need to be done. 

You know where this is leading: I suggest applications should be allowed to be less pretentious and more tentative instead. If so, it would be worth rewarding researchers to keep a proportion of their results in the file drawer. This would have the pleasing effect of a less cluttered literature. There could be a prize, not for the most productive researcher, but for the most discerning. 

In research, you need hindsight to know what the best ideas and the most robust findings are. Keep a record of your unpublished results by all means. Mull over them, unearth them when they suddenly make sense, and share them when asked. My only concern is this: is your file drawer big enough?

- Uta Frith DBE FRS FBA FmedSci is Emeritus Professor of Cognitive Development at UCL Institute of Cognitive Neuroscience and President British Science Association. Find much more in our archive and at
https://sites.google.com/site/utafrith/
http://frithmind.org/socialminds/

 

Ahead of its time? My favourite paper that I ever wrote is my 37th most cited paper. Nosek and Hansen (2008), 'The associations in our heads belong to us: Searching for attitudes and knowledge in implicit evaluation' hasn't been ignored, but its impact in the discipline is incongruous with its impact on me and my understanding of implicit social cognition and implicit measures. It also captured what I believed to be some best practices in research, but was not appreciated as such by reviewers at the time. Why is it my favourite? Here's the abstract: "Explicitly, humans can distinguish their own attitudes from evaluations possessed by others. Implicitly, the viability of a distinction between attitudes and evaluative knowledge is less clear. We investigated relations between explicit attitudes, cultural knowledge and the Implicit Association Test (IAT). In seven studies (158 samples, N=107,709), the IAT was reliably and variably related to explicit attitudes, and explicit attitudes accounted for the relationship between the IAT and cultural knowledge. We suggest that people do not have introspective access to the associations formed via experience in a culture. Ownership of mental associations is established by presence in mind and influence on thinking, feeling and doing. Regardless of origin, associations are influential depending on their availability, accessibility, salience, and applicability. Distinguishing associations as ‘‘not mine’’ is a self-regulatory act and contributes to the distinction between explicit evaluation, where such acts are routine, and implicit evaluation, where they are not." This paper is an exemplar of my theoretical, methodological, and meta-scientific priorities all rolled up into one: * A question of substantial theoretical importance – are the outcomes of the Implicit Association Test better interpreted as attitudes belonging to the person or something cultural?* A methodological strategy that examined the question relatively comprehensively across >100 topical domains with extremely high-powered designs and highly diverse samples allowing for relatively strong and unambiguous conclusions.* A paper in which the introduction focuses on the question and problem, and most of the theoretical work is in the discussion. Lots and lots of replications providing robustness of the conclusions and generalizability across topical domains. The paper has more evidence than most meta-analyses and the benefit of no publication bias because all the topics examined in the mega-study were included in the paper. Moreover, meta-analyses are biased by what topics were most popular among researchers.  For example, meta-analyses of implicit-explicit relations are dominated by studies for which little implicit-explicit correspondence is expected because they are presumed to be the most productive areas for applying implicit measures. These topics covered in Nosek and Hansen (2008) were much more heterogeneous and better represented the attitude landscape, greatly expanding the construct validity of implicit attitudes as measured by the IAT – providing evidence about both what it does measure and what it doesn't. My enthusiasm for this paper was not shared by all reviewers and editors. From January 2004 until it ultimate publication in 2008, the paper was first rejected by JPSP, then rejected again by JPSP in a second full review process, then was rejected at JEP:General, and finally was accepted at Cognition and Emotion with one empathetic reviewer saying 'I have read a previous version of this Ms already a while ago, and I found it very strong. The fact that it is now submitted to Cognition and Emotion probably indicates a very unfortunate reviewing history." With the paper's emphasis on power, replication, and the still timely evidence for debates about the meaning and validity of the Implicit Association Test, I am curious what reception the paper would receive in today's peer review environment.  Thanks to the merciful Editor and reviewers at Cognition and Emotion, I do not need to find out. - Brian Nosek is Executive Director at the Center for Open Science, Professor at the University of Virginia, and co-Founder of Project Implicit. He will be speaking at the British Psychological Society's 2018 Annual Conference in Nottingham.  'This was my firstborn, and its loss still stings' 

It's April 1996 and I'm nearly four years into what's shaping up to be a five-year PhD. My nerve-racking qualifying exam behind me, I've advanced to doctoral candidacy and have about two-thirds of my data in the bag. Down the street from my lab, cafés and surf shops spill into wide, white sands and then the Pacific Ocean. It's sunny in California but the uncertainties and self-doubts of graduate school cast a grey pall.

My elder brother is autistic, unable to speak, assaulted by his unmodulated senses, and often unable to get control of his thinking or behaviour. My niece has been born a few weeks ago; I'm anxious about her but haven't mentioned to my sister. (The diagnosis will come a couple of years hence.) All this is why I'm here: one and the same ponderous attention to detail and regularity has made me a scientist and my brother autistic, and I want to map the developmental watershed between us so that we can steer people across it.

I'm developing a new way to measure the speed with which people shift attention across the visual field. My data show that attention enhances a flash-evoked brain electrical rhythm, like an amplifier: the signal grows as attention enters the contralateral hemifield, then shrinks when attention shifts away. During a late night I have stared at the statistics and experienced that moment of transcendent discovery that all fortunate young scientists have: at that instant I knew something nobody else in the world did. I have mentioned this result to the head of an EEG lab up the hill, and found out that his postdoc has been pursuing it too. The race is on. 

Being a computer scientist originally, since our programmer left I've been taking up the slack. It seems everyone has some kind of small computational task I can help with, and everybody's feverishly preparing posters or slides for the Cognitive Neuroscience Society. My own poster is the last to be finished; I work all night, leave for the airport directly from the lab, give a successful presentation, then arrive back at the lab directly from the airport without sleep. Anxious and exhausted, I'm irritable and my social perspective-taking, which never has been great in the first place, goes out the window: I get into a stupid argument with another student and get myself expelled from the lab. All my data are on tapes in the lab and I have no access to them.

After I'm gone, the lab hires a programmer. After many months I get my data back but the result already has been published, by the other group, through the 'Communicated by' route in PNAS (Morgan et al., 1996). For a year I don't touch science. In the end I send my confirmatory paper to Cognitive Brain Research (Belmonte, 1998).

I will eventually finish a PhD elsewhere, with a different project, and go on to publish more findings that will influence my field. But this was my first big result, and its loss still stings. I now try to remember that students need time to be students, that people under pressure often are not themselves, and that everybody can benefit when high-stress academic workplaces support rather than exclude those with mental health needs. 

- Matthew K. Belmonte is at The Com DEALL Trust, Bangalore; Centre for Autism at the University of Reading, and Reader in Psychology at Nottingham Trent University.

 

References

Morgan, S.T., Hansen, J.C. & Hillyard, S.A. (1996). Selective attention to stimulus location modulates the steady-state visual evoked potential. Proceedings of the National Academy of Sciences of the United States of America, 93(10), 4770-4774. http://www.pnas.org/content/93/10/4770

Belmonte, M.K. (1998). Shifts of visual spatial attention modulate a steady-state visual evoked potential. Cognitive Brain Research, 6(4), 295-307. https://doi.org/10.1016/S0926-6410(98)00007-X