AMPPS Review Shows Shortcomings in Policy Statements on the Effects of Media Use

As different forms of media infuse everyday life, several organizations and associations have issued public statements about the effects of media exposure. However, a scholarly review suggests that many of these statements do not accurately reflect the available scientific evidence, offering overly simplified or one-sided accounts of the scientific research. The findings are published in Advances in Methods and Practices in Psychological Science.

“Although there certainly are some pretty good media policy statements out there, many of the policy statements were not very accurate and where there were inaccuracies, these tended to lean in the direction of conclusions that were generally scarier than could be defended by the actual data,” says psychology researcher Christopher J. Ferguson of Stetson University, who coauthored the paper with fellow media researchers. “There’s no assumption of bad faith, of course, but it seems many professional organizations are struggling to develop policy statements that effectively communicate the complicated, messy and nuanced nature of many media effects fields.”

Ferguson and his coauthors are all researchers with expertise in some aspect of media effects, although they don’t always draw the same conclusions about the impact of different forms of media. They consider ongoing discussion and debate to be an important part of the scientific process, but they noticed that many organizations’ policy statements about media effects didn’t acknowledge that any such debate was taking place.

“We were curious to know how often this was happening and, if this was happening a lot, point out directions that could lead to more accurate statements in the future,” says Ferguson.

Policies Date Back to the 90s

Using Google Scholar and targeted web searches, the team of researchers identified media effects policy statements produced by professional advocacy organizations that represent scholars or clinicians in relevant fields (e.g., American Academy of Pediatrics, American Psychological Association). These searches produced 24 public statements, with the earliest issued in the early 1990s. The statements covered impacts resulting from media violence, screen time, sexual content, and more general effects.

The research team broke into subgroups to evaluate each type of statement, using a standardized rubric focused on specific characteristics: citation bias, false consistency, lack of clarity of transparency, overgeneralization, exaggeration, insulation, and noncredible sources.

In general, the researchers found a noticeable increase in the frequency of media effects statements over the last 30 years. Most of the 24 policy statements came from organization-based committees and were produced by scholars who had interest and expertise in the field.

The research team found that the majority of statements, 19 out of 24, showed citation bias, citing evidence that supported a specific conclusion without mentioning existing evidence that did not support the conclusion. Similarly, 22 out of 24 statements were characterized by false consistency, implying that the evidence of media effects was more consistent than it was in actuality. And only one statement made any reference to the existence of diverse viewpoints among scholars in that area.

The team concluded that 15 out of the 24 statements overgeneralized results, applying media effects findings to contexts far beyond the scope of the original research. And 19 out of 24 statements made exaggerated claims about media effects, suggesting public health or other societal impacts without noting the small or trivial size of the effects found in many research studies.

Most of the statements did not provide detailed information or supporting data.

Cautionary Notes

Ferguson and colleagues suggest that these findings have important implications for both policymakers and parents.

“Since these are ‘policy statements,’ presumably they are staking out policy positions the organizations would like to see policymakers move on. But policymakers may need to be cautious not to mistake these policy positions for a fair summary of current research,” Ferguson says. “The other group of concern is parents, since many parents may become needlessly worried about media effects when policy statements proclaim the evidence to be stronger, more consistent, or more applicable to real life behaviors than it actually is.”

Based on their findings, the research team devised a checklist for best practices that, if followed, would substantively improve the accuracy and quality of such policy statements:

• Acknowledge disconfirmatory data
• Focus on the magnitude of effects
• Acknowledge limitations of research methods
• Solicit balanced views
• Avoid secondary sources
• Distinguish scientific statements from advocacy statements
• Release fewer statements
• Be mindful of unintended harms
• Prioritize and encourage open science practices

Reference

Elson, M., Ferguson, C. J., Gregerson, M., Hogg, J. L., Ivory, J., Klisanin, D., … Wilson, J. (2019). Do policy statements on media effects faithfully represent the science? Advances in Methods and Practices in Psychological Science. https://doi.org/10.1177/2515245918811301