Textbook Analysis Uncovers Erroneous Explanations of Statistical Significance

For nearly 90 years, null-hypothesis significance testing (NHST) has been the most widely used statistical approach in psychological science. Methodologists have long warned that researchers regularly misinterpret and misapply the technique, however, and a group of psychological scientists in Canada believe they may have traced these misunderstandings back to the source: introductory psychology curriculum.

In an analysis of 30 introductory psychology textbooks, including some of the best-selling volumes in North America, scientists based at the University of Guelph found that the vast majority defined or explained statistical significance inaccurately. Many future psychological scientists get their first exposure to statistical methods in those introductory volumes, the researchers emphasize in an article for Advances in Methods and Practices in Psychological Science.

“Psychology researchers may be learning to interpret statistical significance incorrectly quite early in their careers and are likely learning incorrect interpretations in their classes, from their instructors,” write researchers Scott A. Cassidy, Ralitza Dimova, Benjamin Giguère, Jeffrey R. Spence, and David J. Stanley.

In their analysis, the scientists coded the definitions and explanations of statistical significance in each of the 30 textbooks, with a particular focus on common fallacies. They found that 89% of the books that covered statistical significance defined or explained it incorrectly. The most common misconception identified in the books was the odds-against-chance fallacy (defining statistical significance as a less than a 5% likelihood of a result being random.)

The research team observed that in many cases the textbooks could be improved by simply indicating that NHST begins by assuming the null hypothesis is true. Additionally, authors could be encouraged to remove discussions of statistical significance from their books, and simply discuss findings in general terms. An introduction to NHST could be delayed until students’ first statistics course, they suggest.

“Overall, these results suggest that students’ misinterpretations of statistical significance may not be the result of their failing to remember the correct interpretation they were taught,” the scientists write. “Instead, students may be accurately recalling incorrect pedagogy.”

Reference

Cassidy, S.A., Dimova, R., Giguère, B., Spence, J.R., Stanley, D.J. (2019). Failing grade: 89% of introduction-to-psychology textbooks that define or explain statistical significance do so incorrectly. Advances in Methods and Practices in Psychological Science, doi.org/10.1177/2515245919858072