Social Psychology Research Today Has More Participants, Online Studies, Self-Report Measures

Concern about false positives in psychological research — typically, study results that incorrectly support rejecting the null hypothesis — has motivated a push toward more rigorous and statistically robust practices. An effective way to improve the statistical power of research is to collect data from large samples, but this can be challenging for researchers given limits on laboratory space, access to participants, money, and time. Collecting data from online participant pools and using self-report measures are two strategies that allow for increased sample sizes while drawing on relatively fewer resources — but have social psychology researchers adopted these strategies?

Findings published in Advances in Methods and Practices in Psychological Science suggest the answer is “yes.” Scientists Kai Sassenberg and Lara Ditrich found that social psychology research articles published in 2016 and 2018 had larger sample sizes compared with articles published in 2009 and 2011, and this change was associated with increasing use of online data collection and self-report measures.

Sassenberg and Ditrich compared articles published in social psychology journals before and after the journals developed new guidelines for statistical power, a process that took place between 2012 and 2015. They reviewed 1,300 studies from 466 empirical articles published in 2009, 2011, 2016, and 2018 in four social psychology journals that were early to adopt higher statistical power requirements: Journal of Experimental Social Psychology (JESP), Journal of Personality and Social Psychology (JPSP), Personality and Social Psychology Bulletin (PSPB), and Social Psychology and Personality Science (SPPS).

Research assistants who were blind to the study hypotheses determined each article’s sample size and classified the study according to how data were collected. A study was considered an “online study” if all of the study data were collected online; if any data were collected offline, it was considered an offline study. The researchers also identified the types of measures used in a study (e.g., self-report, non-self-report, combination) and the total number of studies in each article.

As predicted, studies published in 2016 and 2018 had larger sample sizes compared with studies published in 2009 and 2011, although this increase was relatively larger for JPSP and SPPS than JESP and PSPB. A higher percentage of studies published in 2016 and 2018 relied on online data collection compared with studies published in 2009 and 2011, and studies published in JPSP showed a steeper increase in this percentage than did studies published in SPPS.

The results also showed that the percentage of studies using exclusively self-report measures was higher in 2016 and 2018 than in 2009 and 2011 for JPSP, PSPB, and SPPS; there was no change for studies published in JESP. In 2016 and 2018, the number of studies per article was larger than in 2009 and 2011.

The results of this investigation suggest that when research policies change, research strategy and methodology change as well. But the differences in trends among the four journals examined indicate that these findings do not generalize to every journal. Future research should examine researchers’ decision making behind these changes to expand on the retrospective findings from this study.

Online data collection and self-report measures are strategic ways that researchers can increase the sample sizes in their studies without incurring high costs. However, these strategies are not suitable for answering every research question. For example, in studies on choice or performance, there are often substantial differences between actual behavior and self-reported behavior. Areas of psychological research that require more labor-intensive research methods may have to adapt to methodological changes in the field through other means.

Reference

Sassenberg, K., & Ditrich, L. (2019). Research in social psychology changed between 2011 and 2016: Larger sample sizes, more self-report measures, and more online studies. Advances in Methods and Practices in Psychological Science. doi:10.1177/2515245919838781