Restocking Our Subject Pools

Elizabeth A. Phelps

My guest columnists this month are two early career scientists, Jay J. Van Bavel and David G. Rand, with interesting ideas about how to utilize new technology to advance psychological science. Their “call to action” suggests a way we can all get involved to enhance our research capabilities and the quality of our science.  –Elizabeth A. Phelps

Over the past few years, psychological science has faced considerable scrutiny. As a recent article in Science pointed out, psychology’s “culture that too heavily favors new and counterintuitive ideas over the confirmation of existing results has led to too many findings that are striking for their novelty and published in respected journals — but are nonetheless false.” Many of the top scientists in the field have called for a more transparent science and reaffirmed the value of replication. For instance, one of APS’s leading journals, Perspectives on Psychological Science, has created a section to publish registered replication reports and several psychologists have launched an unprecedented, large-scale effort to replicate many of the most prominent psychology experiments.

Jay J. Van Bavel

Why isn’t replication more common? A major part of the answer is that replications are costly: when it can take months to run a study, replicating a new result comes at the expense of other, more exciting experiments that push scientific understanding further (assuming the original result was real). Furthermore, as the vast majority of psychology experiments have historically been conducted with a limited pool of undergraduate psychology students, replications can strain subject pools that are already stretched to the limit.

A recent technological innovation is offering the opportunity to reduce these barriers to replication. Various online experimental platforms, such as Amazon’s Mechanical Turk (MTurk), crowdcrafting, and Yale University’s eLab, give researchers the ability to cheaply collect hundreds or thousands of subjects within mere hours or days. In addition to providing more demographically, ideologically, and geographically diverse samples than typical undergrad subject pools, the speed at which data can be collected makes replication almost trivial. Despite some skepticism about the quality of these resources, numerous papers have now shown that many classic effects from psychology, economics, and other fields can be successfully replicated online (e.g., Crump et al., 2013; Horton et al., 2011). These platforms also provide direct intellectual and economic benefits to participants, and with less coercion (and less subject time investment) than one might expect in a lab environment.

David G. Rand

But there is one major problem: the median MTurk subject has completed 300 studies overall, and more studies in the last week than the median physical lab subject completes over his or her entire career (Rand et al., 2013). This means subjects are far from naïve, making it difficult to successfully implement standard psychological manipulations that involve deception or examine more intuitive processes — features that are at the heart of many psychology studies. This obviously compromises MTurk’s usefulness for replication studies: a failed replication may result from the subject pool’s substantial level of experience, rather than the original effect in question being spurious. For instance, in their work showing that cooperation is an automatic, intuitive response, Rand and colleagues have found a steady decline over the last two years in the effect of time pressure on cooperation studies run using MTurk: As participants gained experience in certain economic dilemmas, they were more likely to discard their intuitive responses in favor of alternative strategies that maximize their economic benefit (Rand et al., 2013).

In short, psychologists and other social scientists are now facing a tragedy of the commons: We have exploited the online subject pools for our immediate personal benefit and now the pool of naïve subjects is depleted. This compromises our ability to use online platforms like MTurk to study many psychological phenomena, and prevents them from offering a powerful solution to the need for replication.

We are writing this column as a call to action. We can help salvage our new subject pool by doing what we as scientists do best: disseminating an important idea. We need you to help recruit new participants to join MTurk and other online subject pools by spreading the word in your classrooms and communities. In just a few minutes, you can help resolve this tragedy and renew this resource before it is depleted. Post a slide about signing up for MTurk or these other platforms in your class lectures, place a poster on (or off campus), and place a link on your website. Even more importantly, we need to reach out beyond the psychology classroom: Tell your friends and family over Facebook, Twitter, or Google+. Invite them to learn more about science and earn money while they help save psychology. It only takes one minute and seven seconds to set up an account on MTurk (we timed it)! And don’t forget to use all the tools of social influence: Provide personal appeals, tell them how easy it is to get started, and remind them that everyone is doing it (people from more than 190 countries are already on MTurk).

And finally, reach out to your colleagues, friends, and family to help in the effort. If we can increase the pool of active users who are interested in participating in science, everyone benefits. We can help provide an economic benefit to participants, allow people from all walks of life to learn more about psychological science, and bolster our efforts to build a more robust field.

References

Crump, M. J. C., McDonnell, J. V., &  Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLoS ONE, 8, e57410.

Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14, 399–425.

Rand, D. G., Peysakhovich, A., Kraft-Todd, G. T., Newman, G.E., Wurzbacher, O., Nowak, M. A., & Greene, J. D. (2013). Intuitive cooperation and the Social Heuristics Hypothesis: Evidence from 15 time constraint studies. Available at http://ssrn.com/abstract=2222683.