Hugely important, or lacking interpretation?
I write with reference to the recent BPS report The impact of Covid-19 on students, staff, and Departments of Psychology in UK Universities.
Unfortunately, the reporting and interpretation of the statistical results of this research have so many of the shortcomings that have bedevilled our discipline for decades. For example:
- An almost exclusive reliance on the outcome of statistical significance tests for interpreting the results of the survey.
- A large sample size, meaning that trivial differences between groups potentially lead to statistically significant test results.
- The assumption that a statistically significant result means that differences between groups are important from a practical perspective. For example, the article contends that “Staff with caring responsibilities indicated that time to engage in all research related activities (except presenting research) declined significantly more than for colleagues without caring responsibilities……This is a hugely important finding for the submissions to the REF 2021…………”
Whether this is indeed “a hugely important finding” depends entirely on the magnitude of the effects. However, the magnitudes of effects (either standardized or raw) are never discussed – a grave and important omission in a report which is clearly intended to have an impact on practice.
This report is being widely circulated in University psychology departments. The poor example it sets in terms of the reporting and interpreting of statistics is of concern. Furthermore, there is every possibility that it mightlead to the expenditure of time, money and effort to solve a problem that could be inconsequential. If, on the other hand, the results are indeed “hugely important”, the authors must make a more informed case for their interpretation.
Dr Paul Morris
Department of Psychology
University of Portsmouth
BPS response from Dr Debra Malpass, Director of Knowledge and Insight
The statistical analyses are one element within a mixed methods study. The report also includes large amounts of detailed descriptive data and qualitative analysis, including a focus group with Heads of Psychology Departments in the UK. The statistical analyses were intended primarily to highlight apparent differences between groups.
In order to avoid any trivial differences between groups leading to statistically significant test results, the authors applied the Bonferroni correction to control for type I errors (see page 122 of the report). The analysis used non-parametric tests; while we recognise that there are effect size statistics associated with such tests, they are recognised as having various technical problems, and are not widely understood. Therefore they were not included in the report.
We stand by the results of the survey and have received feedback from universities that the findings reflect the experiences of their staff and students. The study also reaffirms the results from other research on the impact of COVID-19 on higher education.
For example, our research found that staff with caring responsibilities were more likely to experience a significant decline in time to engage in research related activities, compared to staff without caring responsibilities. The UKRI report on The Impact of Covid on Researchers found that between February and March 2021 88% of respondents with child caring responsibilities reported that their caring responsibilities had a negative impact on time for research.
We found that 70% of students reported that since Covid-19, they had experienced a decline in their wellbeing. The ONS report on Covid and The Impact on Students in HE in December 2020 found that undergraduate students reported lower levels of well-being compared with the general population aged 16 to 24. Our research also found that students with disabilities reported the most significant decline in wellbeing and lower levels of engagement in attending lectures, compared to non-disabled students. A report by the Disabled Students’ Commission (2021) on disabled students in UK universities found that 80% of respondents reported that Covid-19 had a negative impact on their mental health and wellbeing, with 81.6% reporting a lack of motivation.
This research is an important evidence base for how the BPS and course providers can provide general and targeted support to groups who may be further disadvantaged in their studies and careers by the pandemic.
BPS Members can discuss this article
Already a member? Or Create an account
Not a member? Find out about becoming a member or subscriber