National Academies Issues Consensus Report on Reproducibility and Replicability in Science

You can learn more about the Reproducibility and Replicability in Science report at the 31st APS Annual Convention. Click here for more information »

The National Academies of Sciences, Engineering, and Medicine (NASEM) has released a consensus report on reproducibility and replicability in science. The report defines key terms, examines the state of reproducibility and replicability in science, and reviews current activities aimed at strengthening the reliability of the scientific enterprise.

Reproducibility and Replicability in Science, funded by the National Science Foundation and the Alfred P. Sloan Foundation, concludes a thorough process that spanned more than a year. The report was authored by a multidisciplinary committee including APS William James Fellow Timothy Wilson (University of Virginia) and APS Fellow Wendy Wood (University of Southern California).

Recognizing that different fields use the same terms in different ways, the report established clear definitions of reproducibility and replicability. The report defines reproducibility as “achieving consistent results using the same input data, computational steps, methods, code, and conditions of analysis as prior studies—known as computational reproducibility within some fields.” Replicability is defined as “obtaining consistent results across studies that are aimed at answering the same scientific question but have obtained independent data.”

The report also assesses the current state of reproducibility and replicability in science.

“There is no crisis, but also no time for complacency,” said the chair of the committee, physician Harvey Fineberg, in an event marking the public release of the report.

The committee concludes that efforts are needed to strengthen both reproducibility and replicability in science, recognizing that these aspects are important but not always easy to attain. Given that replicability of individual studies can vary, the report notes, integrating multiple channels of evidence from a variety of studies is essential to understanding the reliability of scientific knowledge. The study also provides suggestions for how reproducibility and replication can be improved.

The report makes a variety of recommendations for scientists and researchers in presenting their research findings, suggesting that they:

• Convey clear information about computational methods and data products that support published reports
• Provide accurate and appropriate characterization of relevant uncertainties when they report research findings
• Provide a complete description of how a reported result was reached
• Avoid overstating the implications of research findings and exercise caution in their review of research-related press releases

The report also includes recommendations for universities, science funders, journalists, policymakers, and other stakeholders; it also discusses how concerns about reproducibility and replicability might have the potential to affect how the public views the scientific enterprise.

To read the new National Academies report Reproducibility and Replicability in Science, click here.