A Chance to Put More Punch in Peer Review

If they act by October 1, psychologists have a chance to influence a series of changes in peer review procedures that the National Institutes of Health (NIH) plans to try out in the 1997 fiscal year and implement in FY 1998. (Readers can check out the NIH web page (hup://www.nih.gov/grants/dder/rgaupdat.htm) for a summary of comments received to date.) Comments are invited on 10 “principal recommendations” that the NIH Committee on Rating Grant Applications has proposed for re-structuring the scientific review of the more than 40,000 extramural research grant applications the NIH processes each year.

Not yet merged with the NIH system, however, are the grant review study sections of the National Institute of Mental Health (NIMH) and the National Institute on Drug Abuse (NIDA), while the National Institute of Alcohol Abuse and Alcoholism (NIAAA) has almost completely merged its panels with NIH’s. (See accompanying box “Merging Peer Review” on page 15.)

NIH spends about 83 percent of its entire budget on extramural research. In the 1994 fiscal year it devoted about $9 billion of its overall budget of $ 10.9 billion to extramural research. The peer review process, by which 16 or 18 “outside” scientists evaluate each grant, is a key step not only in determining which extramural research applications get funded, peer reviewers also serve as sensitive antennae in NIH’s efforts to identify and support the applications most likely to advance science and benefit society. In addition, the peer review process provides both NIH program staff and the applicants with fairly detailed summaries of the reviewers’ evaluations .

Coherence Across the Institutes

The current NIH drive to improve peer review was launched in late 1994, partly in response to the Clinton Administration’s move to “reinvent government,” but also as a broad effort to make the entire extramural grant review process for NIH’s 24 institutes and offices more coherent. At that time, the NIH appointed an internal Committee on Rating Grant Applications that then began to look at the logistics of scientific review and ask: Do we have a good scoring system? Does it work well? Is it efficient and clear? Does it give the maximum information to the program staff at each Institute who in turn make recommendations to their respective National Advisory Councils for funding? How can it be improved?

In May 1996 NIH issued the committee’s 10 recommendations (see accompanying box) calling for changes in the peer review process. The committee recommended fewer but more precise and explicit measures of the quality of the research ideas proposed in the applications than the criteria used at present.

The recommendations contain three concise criteria for weighing the scientific merit of each grant, along with new rating scales and procedures. They recommend that discussion among the reviewers and their written comments should be focused within the more concise criteria, using numerical scoring scales that would eliminate the 41 gradations used in the current grading scales.

“This is not a take-it-or-leave-it proposition,” said NIH Deputy Director for Extramural Research Wendy Baldwin, as the recommendations were being issued. Baldwin, ‘an APS member, emphasized that “peer review is Dot a system in crisis at NIH. It’s a system that works well, but NIH is “trying to make it work even better,” she said.

By mid-August Baldwin’s office had received about 125 email messages commenting on the proposed changes. Additional comments were formulated by some of the peer review groups working in Washington, DC, this summer. A more voluminous flow of comments is expected as the deadline date approaches. Some of the comments are in effect alternative recommendations that depart dramatically from the present procedures and the recommended changes of the NIH Committee on Rating Grant Applications. But most of the incoming e-mail and other comments suggest moderate changes, large or small, in the committee’s proposals. Baldwin’s office says the early comments focus heavily on the three review criteria recommended by the committee on Rating Grant Applications: significance, approach and feasibility.

Many of the comments argue for sttucturing these categories differently, referring to them by different labels, or adding at least one more category, and the question appeared to be “still up for grabs” at mid-August, NIH staff members said.

Currently, six review criteria are used in the grant review process:

• Originality of the proposed research,

• Experimental approach and methodology,

• Availability of resources to conduct the study,

• Proposed budget and duration of the study, and

• Experience of the principal investigator in the proposed area of research.

The comments received by mid-August also show fairly strong opposition to the NIH recommendation that peer reviewers not assign a global score to each application, Baldwin’s office indicates. The assignment of a global score seems to be extremely important to reviewers, because that is where reviewers think their scientific expertise is most needed, Baldwin’s staff noted.

However, the comments show little sign of opposition to the NIH recommendation that the reviewers should take up one criterion at a time (e.g., significance, approach, or feasibility) and discuss it, comment on it in writing, and score it, before moving on to the next criterion. (At present, reviewers do not score the review categories.)

What remains at issue is whether the reviewers should assign a global score to each application or whether the NIH program staff should arrive at a global score based on the separate criteria scores given by the reviewers.

The recommendation to reverse the current scoring scale so that bigger scores are better, not worse, appears to be “no big deal” in the comments received by Baldwin’s office. Currently the scale ranges from 1.0 for outstanding to 5.0 for acceptable. (These scores are averaged and converted to three-digit priority scores and percentile scores.) But this scale sometimes creates problems when NIH staff try to explain to Congress why the grants with the highest scores don’ t get funded. The standardization issue (see box, recommendation No.8) gets some attention in the comments but not a lot, according to Baldwin’ s office.

Baldwin said she wants everyone to have a chance to comment before a fma1 decision would be made on changes in the review procedures. And she urges that comments be sent directly to her office by e-mail at the following address: [email protected].