More on Peer Review

A Dissident View
‘How Can One Budget for a Discovery?’

Excerpted from: “Peer Review: The Holy Office of Modern Science,” by Maciej Henneberg, Natural Science, February 20, 1997.

“The peer review of grants has its origin in industrial practice. Many people propose to do many things but resources are limited. How can a small number of people who hold the purse strings know everything? The answer is that they cannot, and so they must ask specialists. The best solution is to ask people working on problems similar to the one a grant application proposes to solve. They will be able to highlight merits and point out errors of the proposal. …

“Since people working in similar areas usually know each other and either agree and collaborate or dislike each other and compete among themselves, the result of a review of a particular application will depend on considerations external to the application. … A good grantsman will write an application in such a way as to avoid being criticized. It will either be bland, proposing an innocuous piece of research, or it will propose to continue what the author has been successfully (in terms of grants) doing before. Any testing of somebody else’s theory or encroachment into someone else’s area of research creates a high risk of failure. …

“The need to allocate research monies unavoidably limits the freedom of scientific exploration, with the result that not all scientists get to pursue the research they want. … Giving money for three-year bits of circumscribed research, a practice that seems now to be quite common, is the worst means of fund allocation. It seems to have arisen from the need to give peace of mind to administrators who cannot make too big a mistake by giving away funds in dribs and drabs to projects having tight schedules and detailed budgets. How can one budget for a discovery?”

Paperless Peers

Over the past five years the National Science Foundation has been converting its grants program into a virtually paperless process, from application to review to funding.

Proposals are submitted electronically and posted on a secure web site. Review panel members receive them on compact disks or directly from the Internet and post their comments on the web site. Outside reviewers receive personal identification and proposal numbers to access the proposals on the Internet, then post their comments on the site.

“We’ve stopped sending the dreaded boxes,” says Steven Breckler, NSF program director for social psychology. Exceptions to paperless review are made only by special permission.

The National Institutes of Health is moving in the same direction. Many study sections now ask panel members to post their comments on a secure web site in advance of the meetings. After all reviews are posted, panelists can read each other’s reviews in preparation for discussion. Scoring is also posted on the Internet for automatic calculation of percentile ratings.

“In my view,” says APS Member Marlene Behrmann of Carnegie Mellon University, “although it is burdensome and we have to get the reviews done in advance of the meeting, this has expedited some of the discussion in that we can think about the discrepancies ahead of time and whether or not the criticisms or perceived strengths of proposals are truly justified.”

Although some special emphasis panels (SEPs) operate electronically from start to finish, not all NIH study sections are at that stage yet, but fewer and fewer “dreaded boxes” are being shipped out. The NIH Center for Scientific Review now scans all applications it receives and puts them on CDs. Only the two or three reviewers assigned primary responsibility for a given proposal receive that proposal on paper.

Ultimately, the federal government wants a fully seamless interface with the scientific community. In that utopian future, universities and investigators will need to enter generic information only once, updating personal profile data only as necessary. Even applications to multiple agencies will be accomplished by single-entry data-streaming that tailors the entered information to each agency’s needs.

Peerless?

Those charged with recruiting peers to review research proposals can’t guarantee that each proposal will in fact be reviewed by a true “peer.”

Take clinical scientists, for example. Constance Atwell, Associate Director for Extramural Research at the National Institute of Neurological Disorders and Stroke, says Scientific Review Administrators “sometimes have trouble getting clinical scientists to serve.” This is because clinicians face “special pressure to accumulate billable hours” and it’s difficult to do this if you have to attend review meetings two or three days at a time, three times a year.

Consequently, she says, “You hear that clinical scientists complain that they aren’t being reviewed by their peers, by other clinical scientists, but you can’t have it both ways.”

Peering into the Past

The roots of peer review can be traced to an 1878 yellow fever epidemic in the lower Mississippi valley. Field investigations by the Marine Hospital Service, later to become the Public Health Service, were funded by an eccentric New York philanthropist, Elizabeth Thompson, after Congress refused to do so. A year later, Congress did step in, albeit briefly and timidly.

Some milestones in the early history of NIH peer review (the National Science Foundation’s program grew independently, from an approach used by the Office of Naval Research):

“Federal patronage in biomedicine was firmly established,” wrote historian Richard Mandel in A Half Century of Peer Review: 1946-1996, “and medical research had become a vital prerequisite for developing the national health care system. Moreover, project review by peers had proved effective as a means of bringing together the diverse interests of scientific investigators, academic institutions, Federal policymakers, and user agencies.… [R]eview by experts and the separation of program management and review functions became hallmarks.”

Since then, notes Victoria Harden, NIH historian and director of the DeWitt Stetten, Jr., Museum of Medical Research, “the NIH peer review process has been studied and restudied, tweaked and re-tweaked.” The fine-tuning continues.

1879
Congress creates the National Board of Health to award the first medical research grants, for research on “the origin and causes of epidemic diseases.” A panel of seven non-government experts and four government members decides who receives grants. After a meager $30,000 is awarded to university faculty for yellow fever research, Congress ends funding in 1883.

1887
The Laboratory of Hygiene is established on Staten Island, NY; its advisory board is formed in 1902.

1900-1910
The American Medical Association’s Committee on Research begins a modest program of funding research; grants are decided by volunteering professionals.

1916
Congress charters the National Research Council (NRC) to award medical and military research contracts to academic researchers. Its Medical Division consists of 15 representatives of scientific societies and six to eight at-large members. Special committees of outside experts make awards, adjust budgets, and administer projects.

1930
The Laboratory of Hygiene is renamed the National Institutes of Health and its board is renamed the National Advisory Health Council (NAHC).

1935
Assistant Surgeon General Lewis R. Thompson, Chief of the PHS Division of Scientific Research, begins making extramural grants. He recommends that the NIH Director not be given control of the grant application process but instead that the NAHC establish subcommittees “in more narrow fields of work composed entirely outside the Service. … to recommend approval or disapproval (of contracts) based on scientific worth.”

1941
The wartime Office of Scientific Research and Development (OSRD) is established, replacing efforts by friendly members of Congress to make bloc grants to universities without review. OSRD’s Director, Vannevar Bush, establishes a review process with functions assigned to the NRC, the “first sustained, large-scale exercise of … (peer review) in a biomedical context,” according to a history published by NIH.

1944
Congress begins converting back to peacetime by authorizing the Surgeon General to award grants to public and private institutions and individuals for projects recommended by the NAHC and the National Advisory Cancer Council.

1946
The NIH Division of Research Grants is created to administer extramural grants and fellowships. OSRD projects are transferred to the Public Health Service. NAHC instructs the new NIH division to establish study sections to review grant applications and to explore neglected areas of health research.

SOURCES
“The National Institutes of Health: Chronology of Events – (by Year)” at www.nih.gov/about/almanac/1998/historical-data/chronology.html

A Half Century of Peer Review-1946-1996, Richard Mandel, Division of Research Grants, National Institutes of Health, 1996

“Center for Scientific Review: Important Events in CSR History,” at www.nih.gov/about/almanac/1998/organization/csr/history.html

Reviewer’s Review

Denise C. Park

Denise C. Park, an APS Fellow and Charter Member who specializes in aging and cognition research at the University of Michigan, shared with the Observer her thoughts about her peer review service for the National Institutes of Health.

What is the most important aspect of peer review?

For me it’s that it gives talented people with great ideas a chance to get a hearing and an evaluation of those ideas and to get the funds to do important work. It is always stunning to read a grant that is a tour de force and that has the full package – a strong conceptual framework, great empirical work, and that is interesting and important. It is especially exciting when it is someone who is relatively new to the field. I think the peer review system is generally fair and that if you are good, you can break into the system even if you are not well-known.

Why did you agree to serve as a reviewer?

I receive substantial support for my own work from the NIH, and it seemed like a way to give something back. I felt an obligation, and also I find that I enjoy the focused, predictable nature of the task much more than editing a journal.

Have recent innovations improved the process?

I think that the web-based reviewing system is nice. It essentially requires you to get your reviews done several days before the meeting, and when you meet that deadline, it feels good.

I do think that the old days, when every application was talked about, was also a good service to the investigators. It is often difficult to even get to the point where your grant is discussed on a first submission, and so investigators frequently don’t get a good temperature read from the committee of what issues there was convergence on and how much enthusiasm there was for the project (if it was sent back for revision without discussion). I do appreciate the shorter review meetings, but I always feel badly for the investigators who may get conflicting feedback from different reviewers in the absence of a discussion. There are not many grants that are dreadful, and nearly all are worthy of discussion.

What do you get from it?

I believe that the opportunity to read the best grants and see the organization of projects that do get funded is helpful in developing your own projects and being realistic about what is likely to get funded or not. I also find that I learn a lot, and believe that besides giving something back, I get quite a bit out of the process intellectually. Interacting with other people on the committee and learning how they think and what issues are of concern to them is also a tremendous benefit.

Keith Rayner
“When I was first asked to serve on a review panel back in the early ’80s, I wondered if I really wanted to devote the time and effort that were obviously necessary. I talked with a few close colleagues in my department at the University of Massachusetts and they convinced me that one should look on it as an important service — almost a duty — to the field.

“My first term was completed in 1984, and then I was an ad hoc reviewer for different panels on a number of occasions. A few years ago, after one of my grants received a MERIT award and I was approached again to serve on a review panel, I felt I had an obligation to help.

“The process is easier now. We used to have to write two to three times as many reviews, and we used to meet four times a year instead of three. Obviously, the widespread use of computers and electronic mail has also very much helped the process. We now post our reviews on a secure web-page. No one is allowed to look at the reviews until all reviews are in, but once that takes place, reviewers can look at other reviews. This enables them to come to the meeting with a good sense of what the others think, and communication at the meeting becomes more effective.

“It’s a shame that everyone in the field doesn’t have the opportunity of serving on a review panel, because I think lots of myths about what happens would quickly be dispelled.”

Keith Rayner is an APS Fellow and Charter Member at the University of Massachusetts-Amherst.