Strengthening Psychological Science with Specialized Statistical Review

Checks and balances are an important part of any system — in science, peer review provides one such check, helping to ensure the rigor and accuracy of research findings that may be published. Although subject matter experts routinely review journal submissions for theoretical accuracy, a key component may still be missing from many psychology journals’ publication processes: specialized statistical review.

“Serious concerns about the credibility of psychological research have been raised, and the misunderstanding and misuse of statistical methods have been implicated as an important cause,” writes Tom E. Hardwicke (Stanford University) and colleagues in Advances in Methods and Practices in Psychological Science.

Focused technical assessments by statistical experts that occurs separately from the standard peer-review process could help identify routine errors, as well as address recurring issues in the psychological science literature such as inadequate statistical power due to small sample sizes, Hardwicke and colleagues continued. Although statistical review has been a standard preventative measure against the improper use of statistics in biomedical journals since the 1970s — with 89% of 107 editors surveyed reporting that at least some, if not all, articles published in their journals had undergone additional technical review — psychology journals have been slower to adopt this practice.

The editorial team for Psychological Science recruited a pool of 6 statistical advisors back in 2016. But in a survey of 39 psychology journal editors, Hardwicke and colleagues found that 71% did not differentiate between peer review and statistical review, with 44% indicating that they perceived a separate technical review process to be unnecessary.

But data on specialized statistical review in medical journals suggest a different story. In an Annals of Internal Medicine survey of 337 corresponding authors who published in the journal between 2012 and 2016, 57% reported that specialized statistical review resulted in a moderate to large increase in their articles’ overall quality. A randomized control trial of 115 biomedical articles in Medicina Clinica, on the other hand, found evidence of a small but consistent bump in technical quality after articles underwent an additional statistical review verses the peer review process alone.

Ideally, all articles likely to be published in a psychology journal would have the opportunity to undergo specialized statistical review, Hardwicke and colleagues wrote, but editorial teams looking to jumpstart this process may reap the greatest benefit from targeting common statistical errors that occur in standard analyses.

“Many of the statistical ailments in the psychology literature relate to foundational issues, not advanced techniques,” Hardwicke and colleagues write. “Consequently, the most impactful contribution of statistical review might come from evaluating what appear to be routine analyses.”

The adoption of specialized statistical review goes hand in hand with open science, the researchers note, as it relies on psychological scientists making their data publicly available in order to test the accuracy of the statistics supporting their theoretical claims.

“Psychological science is in the midst of a credibility revolution, and this is an opportune time for journal editors to consider adoption of statistical review,” the authors write.

Statistical review is not a “cure all” for psychology’s statistical ailments — but pairing technical review with improvements in the statistical training process of early-career researchers could help break the cycle of statistical error in psychological science.

Reference

Hardwicke, T. E., Frank, M. C., Vazire, S., & Goodman, S. N. (2019). Should psychology journals adopt specialized statistical review? Advances in methods and practices in psychological science. doi: 10.1177/2515245919858428