Watch Out for the Experienced Study Participant

When conducting psychology studies online or in the lab, researchers might not think about participants’ past experiences as a research subject. But research published in Psychological Science suggests that these experiences could make a difference in study outcomes.

The research shows that participants who are not naïve to research methods or materials—those who have performed the same task multiple times across different studies, for example—may produce varying results, even when they do not remember having participated in similar experiments before.

Psychology researcher Jesse Chandler of the University of Michigan and Mathematica Policy Research and colleagues followed up with participants who earlier had completed a series of psychology tasks online (see Klein et al., 2014). Participants were contacted a few days, about a week, or about a month after initial participation and were asked to complete the same tasks online a second time. Some participants were assigned to the same condition they were in initially, whereas others were assigned to different conditions.

Chandler and colleagues found that the effects of the experimental manipulations were smaller in the second experiment (weighted Cohen’s d = 0.63, on average) compared to the first (d = 0.82). The decrease in effect size was greatest when participants were assigned to a different condition in the second study than they were assigned to in the first.

The authors were uncertain about the reason for this finding, writing, “There is no direct evidence of a mechanism underlying this decline.” But they speculate that there could be several potential factors at work, including practice effects, cognitive elaboration, and motivation to perform a certain way.

To ensure that study findings are as accurate as they can be, the authors suggest that that participants’ participation history should be tracked when possible and that experimenters should consider using new procedures, new materials, and larger sample sizes when conducting subsequent studies.

All data and materials from the study have been made publicly available via Open Science Framework and the article has received badges for Open Data and Open Materials.

You can learn more about some of Chandler and colleagues’ other work on collecting data online in Paolacci and Chandler (2014) and Shapiro, Chandler, and Mueller (2013).

Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26 (7), 1131-1139. DOI: 10.1177/0956797615585115