These Aren’t the Bots You’re Looking For

This summer, researchers in psychological science and other fields noticed a sudden increase in low-quality responses to surveys and other experimental measures posted to Amazon Mechanical Turk (MTurk). Many of the responses originated from a small set of geolocations, leading some researchers to suspect bots as the source.

Researchers raised concerns about the integrity of their data,  leading Turk Prime to launch a thorough investigation of the issue. Their results, published online in mid-September, revealed that around 60 of these repeated locations could be traced to server farms. In light of this, Turk Prime launched two tools that allow researchers to block known suspicious locations and duplicate locations.

To learn more, Turk Prime also recruited 140 respondents from known server-farm geolocations and 100 nonfarm respondents to complete a new survey. The results showed that farmers did not provide typical responses to a Big Five personality trait measure, a “trolley problem” moral dilemma, or an anchoring task compared with findings reported in previous research and responses from nonfarmers. In addition, farmers’ answers to open-ended questions were almost all grammatically incorrect and low quality.

Intriguingly, both farmers and nonfarmers successfully completed tasks intended to catch bots, but farmers were less likely to pass an English proficiency screener and specific cultural checks.

Based on this evidence, the Turk Prime researchers concluded that the low-quality responses have been coming from humans with limited knowledge of English.

“Our goal throughout the summer and this investigation has been to identify the source of low quality data on MTurk and then to erect strong and intelligent safeguards for the research community,” they write. “People seeking fraudulent access to restricted portals are a problem as ancient as Troy, and no platform, whether physical or virtual, is impenetrable.”

In addition to the two tools for blocking respondents with suspicious or duplicate locations, TurkPrime has recently developed a “Universal Exclude” feature, which allows researchers to add workers to a persistent exclude list with one click.