Instructions for Reviewers of a Replication Report Proposal
As an expert (typically, an author of the original study) on this topic, the editors seek your help in ensuring that the protocol for replications of the study is as accurate as possible. By providing missing information, specifying when deviations from the original design are important, and noting any manipulation checks necessary in a replication, you can make sure that replication attempts mimic the original design closely, thereby increasing the chances that other labs will reproduce the original result. Your task as a reviewer is to vet the design of the proposed replication to verify that it accurately and completely replicates the original study design. To that end, your review should focus exclusively on the information in the proposal form.
The form completed by the proposer reports the details gleaned from the methods section of the original study and the corresponding details of the proposed replication. This detailed form should help you identify any important discrepancies between the designs. For each item, we ask you to verify whether the description of your study is accurate and complete, to note any discrepancies or manipulation checks that are needed, and to add any other comments or qualifications not covered by the other items. Most important, we ask you to identify the tolerance range for each value. That is, what range of values would be acceptable for this study to still be a direct replication of your study. Please specify the widest acceptable range that you believe will produce the same outcome.
In our testing of this review procedure, we have found it to be less time consuming than a traditional review. For each aspect of the studies, you will be asked to answer the following questions:
- Accuracy of the proposer’s description: Is this aspect of your original study described accurately? Enter a “yes” or “no.” If you answered “no,” please explain how the description was inaccurate. If a detail was reported to be missing from the original article, please provide it here. If you no longer have the relevant information, please estimate it and note that it is an estimate. If you think a method detail is applicable but the submitter marked it as “not applicable,” please provide the method detail here. Please also identify additional discrepancies between the original study and the proposed study for this aspect of the design. For each additional discrepancy, indicate whether it is significant enough that the study should not be considered a direct replication. Keep in mind that any discrepancies listed as significant imply that the generality of the original result is limited. As a rule of thumb, think about how you would interpret a positive result in the proposed replication report. If the replication attempt produced the same effect as the original, would you treat it as a direct replication of the original? If the answer is yes, then you should not flag a discrepancy as significant. If the answer is no, then you should.
- Acceptable Values: Please provide the range of values for this aspect of your study that would be appropriate in a direct replication. For what range of values do you think your study should replicate? Keep in mind that specifying an overly restrictive tolerance range limits the generalizability of your study. If you believe your results apply to all people, then the sample should not be restricted. If your believe your effect applies only to young participants, then specify the acceptable tolerance range (e.g., the participants must be between 18 and 24).
- Manipulation checks: Are there any manipulation checks needed? If yes, please describe them and note the acceptable range of values for each check.
- Comments or qualifications: If you have other comments or qualifications about this aspect of the replication proposal that are not covered by the other questions, please describe them.
We depend on your cooperation to ensure the study protocol is appropriate. Unlike a traditional review process, the final protocol plan will be “signed” and open, giving you an opportunity to voice any concerns and objections as part of the public record, prior to data collection. Only the final version of the submission form resulting from the editorial process and collaboration between the proposer and original authors will be public. Intermediate stages, prior to acceptance of the protocol, will not be posted publicly. As a reviewer, you will have a final opportunity to add/remove comments and qualifications before the protocol becomes public. We hope that this formalised review process will make reviewing the methodology efficient and that it will allow you to voice any concerns in a formal and transparent way.
