NIH Center Strives for Fair Review of Grant Applications

The Center for Scientific Review (CSR) has a singular mission — to see that grant applications to the National Institutes of Health (NIH) receive fair, independent, expert, and timely reviews, free from inappropriate influences, so NIH can fund the most promising research. CSR receives and refers all NIH grant applications and performs the first level of review for approximately 75% of them, amounting to review of more than 62,000 grant applications per year and involving more than 18,000 unique reviewers. CSR does this with less than 0.4% of the $39.3 billion NIH budget. Because identifying the most promising research is crucial to NIH’s mission, CSR regularly evaluates the peer-review process and implements improvements informed by data and input from the scientific community.

CSR is seeking broad engagement with the scientific community to more quickly make improvements and respond to problems. CSR hears from the scientific community through multiple avenues. The members of the CSR Advisory Council provide advice to the director of CSR on all matters related to its mission on the basis of their experience as investigators and reviewers. They act as a conduit for input from the broader scientific community. Traditionally the council has comprised full professors and previous study section chairs but, to broaden stakeholder input, the council was recently expanded to include an assistant professor and more junior ad hoc participants.

Another way in which CSR has been able to broaden perspectives in peer review is to include early career scientists in the review process. The Early Career Reviewer Program began in 2011 to provide early career scientists with experience to improve their ability to write a competitive grant application and to broaden and diversify CSR’s pool of well-trained reviewers. A CSR Advisory Council working group, made up of members of the council as well as members of the broader scientific community, recently reevaluated the Early Career Reviewer Program and recommended changes in criteria and expanded reach and support of early-career scientists.

CSR’s efforts to broaden perspectives in peer review go beyond the Early Career Reviewer Program. Primary efforts involve diversifying review panels in terms of gender, race and ethnicity, professional rank, and institution size. In recruiting members of standing panels, the emphasis is on expertise established through funding and publications instead of academic rank. Accordingly, scientific review officers are recruiting more assistant and associate professors to serve as members of standing panels. CSR will also seek recommendations of reviewers from scientific societies through an online form that is under development.

In addition to advice from CSR Advisory Council, CSR is using social media and a new Review Matters blog (csr.nih.gov/reviewmatters/) to engage with the broader scientific community. Comments from these streams give CSR a direct way to communicate with stakeholders. A recent policy change to allow Scientific Review Officers to release summary statements for not-discussed applications first, instead of following those for all discussed applications, was motivated by comments on Twitter (@CSRpeerreview).

CSR also seeks input from the scientific community in decisions about the scientific scope of study sections. In 2019, CSR implemented the Evaluating Panel Quality in Review process, or ENQUIRE, a data-driven process intended to evaluate both the scope of panels and the quality of the review process in each. Among the study sections of potential interest to psychological scientists evaluated in the Healthcare Delivery and Patient Management ENQUIRE group were Behavioral Medicine: Interventions and Outcomes and Psychosocial Risk and Disease Prevention. The Functional Neuroscience ENQUIRE group that was evaluated in 2019 included the study sections Language and Communication; Neuroendocrinology; Neuroimmunology; Rhythms and Sleep; Neurobiology of Learning and Memory; Cognition and Perception; and Mechanisms of Sensory Perceptual, and Cognitive Processes. All of these review proposals using human subjects.

The ENQUIRE process proceeds with the establishment of a panel of external scientists who have a broad interest in the topic area; those with a vested interest in a particular study section are avoided to ensure a broad view of the field. External panels determine how best to group the various topic areas and to consider emerging areas of science. The process has been most effective when the external panel has been asked to think outside the confines of the current organization. Following input from the external panel, an internal panel composed of NIH staff from CSR and from the funding institutes and centers involved focuses on the review process and examines factors such as whether the panel members as a whole are involved in discussions of applications versus a sole focus on the comments of assigned reviewers.

The most important way that CSR receives input from the scientific community is through scientists’ role as peer reviewers. NIH depends on scientists to give their time to serve on panels. Although scientists should undertake review service as a responsibility, there are many rewards. Review meetings provide opportunities to network with colleagues; meet scientists in related fields, possibly leading to new collaborations; see emerging science; and see first-hand how NIH policies are implemented.

Peer review depends not only on the generosity of scientists who serve as reviewers but also on their integrity. NIH and CSR have recently raised awareness of the importance of integrity and confidentiality in the peer-review process; an accurate assessment of grant proposals relies on the ability of reviewers to speak freely without concern that the confidentiality of the peer-review meeting will be broken. CSR has taken multiple actions to address this ranging from increasing reporting avenues, increasing awareness among investigators and reviewers, developing online integrity-training modules, and working with NIH to investigate allegations and respond with appropriate consequences ranging from deferral of applications to pursuing government-wide suspension and disbarment.

New to NIH?

NIH has two levels of review for grant applications. The first level is conducted by a scientific review group (also called a panel or study section) and assesses the overall scientific impact of the proposal. The second level is conducted by the advisory council of the funding institute to which the proposal is assigned. A two-letter code in the grant number indicates the funding institute to which the proposal is assigned. For example, 1 R01 MH 123456 is assigned to the National Institute of Mental Health.

The funding institute bases a funding decision largely on the first level of review but also takes into account their research priorities. What is CSR’s role?

The Center for Scientific Review (CSR) is one of the 27 institutes and centers of NIH. Unlike most of the institutes and centers, CSR does not fund grants. The focus is solely on the first level of review. All grant applications to NIH are received by CSR. CSR then refers those applications to scientific review groups. Approximately 75% of grant proposals submitted to NIH are reviewed by groups convened by CSR.

What (and who) is a study section?

Grant proposals might be reviewed in a standing study section or in a special emphasis panel. Standing study sections are review groups that recur every review cycle, 3 times per year, and have a slate of members who are extensively vetted and typically serve a 4-year term. Standing study sections depend primarily on the members, but scientific review officers recruit additional reviewers each meeting (called ad hoc or temporary members) as needed to ensure proper expertise to evaluate the proposals. Special emphasis panels are review groups that do not have a set list of members. The list of reviewers for a particular review group (roster) is usually published at least 30 days before the meeting.

The list of members of a standing panel is always available at CSR’s website.

Why is it important to use a rigorous process to determine scientific scope of study sections?

For standing study sections, raw overall impact scores are assigned a percentile on the basis of that study section’s scores over the last three review cycles. Funding institutes and centers perform a second level of review and their advisory councils make funding decisions. Although the research priorities of the funding institute bear on funding decisions, the outcome of the first level of review is a major factor. Because an application’s assigned percentile is based on the scoring behavior of the particular study section, and because only the top 10% to 15% are likely to be funded, grouping of scientific topics is important.

How should you prepare to write a grant application?

Identify a likely funding institute and contact a program officer there before you go beyond developing specific aims. Because the priorities of the funding institute are important in the second level of review, make sure your research goals fit the institute’s priorities. You can find a program officer by using NIH RePORTER and using the Matchmaker function to find funded proposals on similar topics.

After confirming that your goals fit the priorities of a funding institute, construct your proposal with the help of a successful model. Ask your colleagues who have been successful in competing for NIH awards if you may see their funded proposal and use it as a guide. NIAID also has posted copies of funded proposals.

Aim to finish your proposal well in advance of the grant deadline. This will give you time to get feedback from colleagues. Submit your proposal to grants.gov in advance of the deadline so that you can check your submission for errors and omissions. You can correct problems but only if you are able to submit the final version before the deadline. Submitting materials to address omissions or errors are not allowed.

Does CSR consider requests for a particular study section?

Yes, requests can be made using the Assignment Request Form. Requests are optional. CSR will examine the specific aims and the research strategy to find the best fitting scientific review group. Descriptions of study sections are also posted.

How can you figure out what panel to request?

Use CSR’s Assisted Referral Tool. Second, you can log onto NIH RePorter and use the Matchmaker function to see similar funded applications and where they were reviewed. Third, contact a program officer or SRO for advice.

How can you gain review experience?

If you are an early-career scientist who has not yet successfully competed for an R01 or R01-equivalent grant, you might be eligible for the CSR Early Career Reviewer Program.