Serious power failure
In the May issue of Nature Reviews Neuroscience
Psychology has had a torrid time of late, with fraud scandals and question marks about the replicability of many of the discipline’s key findings. Now it is joined in the dock by its more biologically oriented sibling: neuroscience. A team led by Katherine Button at the School of Experimental Psychology in Bristol, and including psychologist Brian Nosek, founder of the new Center for Open Science, make the case in a new paper that the majority of neuroscience studies involve woefully small sample sizes, rendering their results highly unreliable. ‘Low statistical power is an endemic problem in neuroscience,’ they write.
At the heart of their case is a comprehensive analysis of 49 neuroscience meta-analyses published in 2011 (that’s all the meta-analyses published that year that contained the information required for their purposes). This took in 730 individual papers, including genetic studies, drug research and papers on brain abnormalities.
Meta-analyses collate all the findings in a given field as a way to provide the most accurate estimate possible about the size of any relevant effects. Button’s team compared these effect size estimates for neuroscience’s subfields against the average sample sizes used in those same areas of research. If the meta-analyses for a particular subfield suggested an effect – such as a brain abnormality associated with a mental illness – is real, but subtle, then this would indicate that suitable investigations in that field ought to involve large samples in order to be adequately powered. A larger effect size would require more modest samples.
Based on this, the researchers’ estimate is that the median statistical power of a neuroscience study is 21 per cent. This means that the vast majority (around 79 per cent) of real effects in brain science are likely being missed. More worrying still, when underpowered studies do uncover a significant result, the lack of power means the chances are increased that the finding is spurious. Thirdly, significant effect sizes uncovered by underpowered studies tend to be overestimates of the true effect size, even when the reported effect is in fact real. This is because, by their very nature, underpowered studies are only likely to turn up significant results in data where the effect size happens to be large.
It gets more worrying. The aforementioned issues are what you get when all else in the methodology is sound, bar the inadequate sample size. Trouble is, Button and her colleagues say underpowered studies often have other problems too. For instance, small studies are more vulnerable to the ‘file-drawer effect’, in which negative results tend to get swept under the carpet (simply because it’s easier to ignore a quick and easy study than a massive, expensive one). Underpowered studies are also more vulnerable to an issue known as ‘vibration of effects’ whereby the results vary considerably with the particular choice of analysis. And yes, there is often a huge choice of analysis methods in neuroscience. A recent paper documented how 241 fMRI studies involved 223 unique analysis strategies.
Because of the relative paucity of brain-imaging papers in their main analysis, Button’s team also turned their attention specifically to the brain-imaging field. Based on findings from 461 studies published between 2006 and 2009, they estimate that the median statistical power in the subdiscipline of brain volume abnormality research is just 8 per cent.
Switching targets to the field of animal research (focusing on studies involving rats and mazes), they estimate most studies had a ‘severely’ inadequate statistical power in the range of 18 to 31 per cent. This raises important ethical issues, Button’s team said, because it makes it highly likely that animals are being sacrificed with minimal chance of discovering true effects. It’s clearly a sensitive area, but one logical implication is that it would be more justifiable to conduct studies with larger samples of animals, because at least then there would be a more realistic chance of discovering the effects under investigation (a similar logic can also be applied to human studies).
The prevalence of inadequately powered studies in neuroscience is all the more disconcerting, Button and her colleagues conclude, because most of the low-lying fruit in brain science has already been picked. Today, the discipline is largely on the search for more subtle effects, and for this mission, suitable studies need to be as highly powered as possible. Yet sample sizes have stood still, while at the same time it has become easier than ever to run repeated, varied analyses on the same data, until a seemingly positive result crops up. This leads to a ‘disquieting conclusion’, the researchers said – ‘a dramatic increase in the likelihood that statistically significant findings are spurious.’ They end their paper with a number of suggestions for how to rehabilitate the field, including performing routine power calculations prior to conducting studies (to ensure they are suitably powered), disclosing methods and findings transparently, and working collaboratively to increase study power.
The love lives of the men and women who have no sense of smell
In the February issue of Biological Psychology
Around one in 7500 otherwise healthy people are born with no sense of smell, a condition known as isolated congenital anosmia (ICA). So dominant are sight and hearing to our lives, you might think this lack of smell would be fairly inconsequential. In fact, a study of individuals with ICA published last year showed just how important smell is to humans. Compared with controls, the people with ICA were more insecure in their relationships, more prone to depression and to household accidents.
Now, in a follow-up paper involving the same 32 patients with ICA, Ilona Croy and her colleagues have looked at how this lack of a sense of smell affects their sexual relationships. The researchers’ analysis uncovered an intriguing sex difference. Compared with 15 age-matched controls, the 10 men with no sense of smell reported having substantially fewer sexual partners in their lifetime (male controls averaged five times the number of partners). In contrast, women with no sense of smell averaged just as many sexual partners as women with smell.
On the other hand, the 22 women (but not the men) without a sense of smell tended to report feeling more insecure in their relationship with their current partner, than did the healthy controls. This insecurity was specific to their sexual partner and wasn’t found in relation to friendships or maternal attachment.
Across both sexes, the impact of a loss of smell makes sense given the mounting evidence for the social importance of smell, for example we can use smell to detect other people’s anxiety; people with more empathy are more likely to remember your smell; and smells convey at least some personality traits. Also, common sense suggests people without a sense of smell might worry about any odours they could be exuding without their knowledge. But the question still remains – why should not having a sense of smell affect men and women differently?
The researchers surmised that not having smell reduces men’s ‘exploratory sexual behaviour’, perhaps due to their lack of social confidence. Consistent with this interpretation, there was a negative correlation between the male (but not female) patients’ levels of social insecurity and their number of sexual partners.
On the other hand, the researchers think the effect of a lack of smell on women makes sense in light of past research suggesting that smell is more important for their relationship security, than it is for men. For instance, a study published in 2008 found that a half of the women surveyed had worn someone else’s clothes (usually a partner’s) because of its smell, compared with just 13 per cent of men. Also relevant – the female patients’ had lower social confidence than the female controls, and this correlated with their lack of relationship security.
Other research has shown that odour is more important to women than it is to men in choosing a partner: women supposedly prioritise good odour over good looks, men the opposite, although it’s not clear how this fits with the current findings. Women also seem to have a superior sense of smell, on average, compared with men, and value the sense more highly.
Croy and her colleagues acknowledged the need for caution given their small sample size, but they said their results emphasise ‘the importance of the sense of smell for intimate relationships’.
Female political role models have an empowering effect on women
In the May issue of the Journal of Experimental Social Psychology
The late Margaret Thatcher – Britain’s first and, so far, only female Prime Minister – is criticised for failing to do more to help other women get ahead in politics. Supporters argue, however, that the example she set will, on its own, have been of profound benefit to women with leadership ambitions. A new study puts this principle to the test, examining the effect on women of reminders about the contemporary female political high-flyers Angela Merkel and Hillary Clinton.
Ioana Latu and her colleagues recruited 149 Swiss student participants (81 women) to make a persuasive public speech against the rise in student fees. The speeches were made in a virtual reality room in front of a virtual audience of 12 men and women. Crucially, some of the participants performed their speech in a room with a poster of Hillary Clinton on the back wall; others with Merkel on the wall; a third group with Bill Clinton’s poster on the back wall; and for a final group,
there was no poster.
The key result is that the female students spoke for significantly longer (a sign of dominance) when Merkel or Hillary Clinton was on the back wall (as opposed to Bill or no poster) – an increase of 49 per cent and 24 per cent, respectively, making their speeches just as long as the men’s. These female students’ speeches were also rated as better quality by two coders blind to the experimental condition, and they also evaluated their own performance more positively. The presence of the different posters made no difference to the performance of the male students.
‘We believe these findings are important because although a wealth of research has studied the effects of role models on academic and math performance, there is no research that investigates the effect of female political role models on successful leadership behaviour,’ the researchers concluded. ‘Yet, exactly such behaviour is crucial because not only is an increase in female politicians the goal of equality, it can also be (as our results show) the engine that drives it.’
The material in this section is taken from the Society’s Research Digest blog at www.researchdigest.org.uk/blog, and is written by its editor Dr Christian Jarrett. Visit the blog for full coverage including references and links, additional current reports, an archive, comment and more.
Subscribe by RSS or e-mail at
Become a fan at
Follow the Digest editor at
BPS Members can discuss this article
Already a member? Or Create an account
Not a member? Find out about becoming a member or subscriber