Advancing Science Through the Use of “New Statistics”

There are several steps that researchers can take to bolster the integrity of their work, but embracing the use of the “new statistics” of effect sizes, estimation, and meta-analysis is a particularly important one, argues psychological scientist Geoff Cumming of La Trobe University in Australia.

As Cumming notes in a new tutorial published online in Psychological Science, the critical flaw of the traditional statistical approach – null-hypothesis significance testing (NHST) – is that it disposes scientists to think of their research aims and results in black and white. A statistically “significant” finding derived from NHST reveals little about the true significance of the result in the broader research context or in the real world.

New-statistics techniques, on the other hand, provide more useful information, allowing researchers to gauge the relative size of an effect or the extent of uncertainty surrounding a particular result. And evidence suggests that people are more likely to interpret research results correctly when they’re presented using a new-statistics technique like a confidence interval than with NHST.

While the strategies and techniques that comprise the new statistics are not themselves new, they are likely unfamiliar to many researchers given that NHST remains deeply entwined with scientific thinking.

Cumming acknowledges that moving away from NHST is no easy feat: “These are not mere tweaks to business as usual, but substantial changes that will require effort, as well as changes in attitudes and established practices.”

But these obstacles are worth tackling in light of the potential payoffs. Using new-statistics techniques obviates the need to categorize replication attempts as “successes” or “failures,” alleviates pressure to achieve results that reach statistical significance, and weakens the temptation to consider a single result as definitive.

Such changes help to strengthen the overall integrity of scientific research, conveying benefits not only for researchers within the scientific community but also for the general public.

“The changes will prove highly worthwhile: Our publicly available literature will become more trustworthy, our discipline more quantitative, and our research progress more rapid,” Cumming concludes.

Part of a larger suite of initiatives by the journal Psychological Science to promote robust research practices, the tutorial outlines 25 guidelines to help scientists incorporate the new statistics into their own research. Cumming also highlights a wide range of new-statistics resources – including textbooks and freely available software – that provide guidance for deriving and presenting effect sizes and confidence intervals in a wide range of situations, including meta-analysis. And the Association for Psychological Science plans to work with Cumming to develop and disseminate workshops and presentations extending the tutorial’s impact.

Read the tutorial, which is freely available to the public, to learn more about the new statistics and how to use them.

Cumming, G. (2013). The new statistics: Why and gow. Psychological Science. DOI: 10.1177/0956797613504966