New Review? It’s True!

Behavioral science research is experiencing something of a pre-millennium milestone this month: The National Institutes of Health (NIH) is launching a wholly revamped system for reviewing grant proposals from psychologists and other behavioral and social science researchers. Only time will tell what it means for the future of psychological science, but in the near term it is breathing new life into behavioral science at the $15 billion agency.

The restructuring of behavioral science review was prompted largely by the transfer to NIH of the review systems of three of the leading supporters of psychology research grants. Those institutes — the National Institute of Mental Health (NIMH), the National Institute on Drug Abuse (NIDA), and the National Institute on Alcohol Abuse and Alcoholism (NIAAA) – became part of NIH a few years ago, but have maintained separate review systems until now.

The arrival of these three institutes means, among other things, an enormous influx of behavioral science grant applications. But rather than simply shifting their review structures to NIH in toto, the existing NIH review committees, known as study sections, have been reconfigured not only to accommodate the increased number of proposals but also to modernize the structure and reduce barriers between disciplines and institutes.

This was “an enormous opportunity to update the peer review structure,” said APS Member Norman Anderson, Director of the NIH Office of Behavioral and Social Sciences Research, which coordinated much of the planning for the reorganization. “Our field has progressed a great deal” since the former structure was established, he added. “It is critical that our peer review structure reflects the current themes and complexities in behavioral and social sciences research today.”

The study sections now in place are the result of extensive input from the extramural community and a massive internal effort by NIH staff, both from the Center for Scientific Review (CSR, formerly DRG) which houses the NIH review system, and from individual institutes. Altogether, 13 of NIH’s 24 institutes were involved in developing the new review structure. This broad participation is an indication of the widespread importance of behavioral science at NIH.

“The reorganization process was a major commitment of time and energy on the part of the working groups and the NIH committee,” said Anderson. “It again demonstrated to me the amazingly high level of scientific talent that exists in the behavioral and social science community both on the NIH campus and in the extramural community.”

“I am especially appreciative of Virginia Cain’s efforts in coordinating this extraordinarily complex task.,” he added. This acknowledgment of Cain’s enormous role is widely shared. Cain, a 20-year veteran of NIH, is Anderson’s deputy at OBSSR. She oversaw much of the planning activity on this project, and is continuing her central involvement in the NIH staff group that is implementing the new structure.

Most Important
“This might be the single most important issue for psychology at NIH,” said Alan Kraut, APS Executive Director. “If the peer review system isn’t right, it doesn’t matter how much money NIH has,” he said, referring to the 15 percent increase NIH received from Congress in fiscal year 1999. Praising the process, Kraut noted that the early and continual involvement of the extramural community was a critical element in putting together a structure that he hopes will be responsive to the field now as well as in the future.

NIH sought input from the community from the earliest stages of the reorganization. Draft review structures were posted on the Web and individual researchers were strongly encouraged to submit comments. The peer review changes were also a major topic of discussion among the 150 or so participants of the Summit of Psychological Science Societies in May 1998.

The view from the Summit, which was conveyed in a letter to CSR, was that the composition of review panels was the critical element, specifically “whether there will be adequate representation of behavioral scientists on newly-created review panels, and whether the review panels themselves will adequately cover the range of behavioral research that is fundable by NIH.”

“The overriding concern here is the need for applications to be appropriately reviewed by those who are experts both in methodologies and the knowledge base of behavioral science,” wrote Kraut and APS Fellow Richard McCarty, head of the science directorate at the American Psychological Association in a joint letter on behalf of the Summit.

“Summit participants recognized that the process now underway is explicitly designed to be sensitive to behavioral science research issues, and most involved in this process agree that NIH is making a strong effort to systematically consult with the behavioral science research community. Nonetheless, we want to underscore the importance given by Summit participants to ensuring adequate representation of behavioral science researchers and the range of behavioral science research once panels are formed,” they wrote.

The Summit also encouraged CSR to address concerns about the workload involved in serving on NIH study sections, and to conduct a systematic evaluation of the reorganization. NIH has been very responsive and is in the process of addressing these issues.

Some history here: APS has been focusing on the integration of NIMH, NIDA, and NIAAA peer review into NIH ever since Congress first proposed transferring the three institutes there beginning in 1991. APS was among the only scientific organizations supporting the move of these institutes from the former Alcohol, Drug Abuse and Mental Health Administration (ADAMHA), but Kraut repeatedly raised concerns about the impact on peer review in both formal testimony and informal discussions with Congress and in discussions with NIH. As he told the then-new Director of CSR, Ellie Ehrenfeld, in a December 1996 letter, “One key to our support was the provision in the legislation that preserved the ADAMHA institutes’ peer review systems for the first four years after the move” rather than shoehorning their peer review into the NIH system.

Principles and Interest
Scientific focus, flexibility, appropriate expertise — these were the watchwords guiding the reorganization process from the beginning, when the directors of the participating institutes jointly developed the following operating principles:

The array of applications being considered by a study section should be determined by the scientific focus of the research, rather than by the professional affiliation of the principal investigator, the grant mechanism , or the research technique to be used.

The range of science considered by a study section should allow a breadth of perspective, yet this should be balanced by an appropriate depth of scientific expertise.

To allow flexibility in review, the range of scientific expertise of study sections should overlap.

When both clinical and basic research are reviewed by a single study section, representation of expertise in both areas should be adequate.

The structure of the initial review process should be flexible enough to accommodate emerging scientific areas.

Implementing these principles has resulted in a number of important new approaches in behavioral science peer review, according to Ellen Stover, director of the NIMH Division of Mental Disorders, Behavioral Science, and AIDS. One major change is that study sections will not be dedicated to specific institutes or diseases, but rather, will be reviewing across disorders and disease groups, and across the lifespan. Ultimately, she predicts, this should lead to more shared funding arrangements and joint initiatives among institutes as well as increased support for research in areas such as co-morbidity which involves both mental illness and substance abuse, and the behavioral and social risk factors that involved in a range of conditions such as heart disease, diabetes, or cancer.

The new structure could also promote interdisciplinary research, particularly through committees that bridge fields, such as a study section on cognitive neuroscience. In addition, there is a generic methodologies study section designed to advance techniques and technologies, including statistics and modeling, that are relevant to behavioral and social sciences.

Not all of the review activities from NIMH, NIDA, and NIAAA have been transferred to NIH. Research applications in more mission-specific areas, such as treatment and health services, will continue to be reviewed “in-house” by the institutes, as well proposals in response to Requests for Applications (RFA’s) and applications for certain funding mechanisms, such as training and center grants.

One interesting side effect from the reorganization: Stover and several others who were directly involved in the internal reorganization group say that the process yielded not only a good review structure, but also a stronger behavioral and social sciences network within NIH. Anita Miller Sostek, chief of the CSR Behavioral and Social Sciences Initial Review Group (or IRG, which is the umbrella structure under which the new study sections are clustered) credits the process with improving communication among behavioral scientists at NIH and with strengthening the connections between program officers and review staff.

The reorganization of behavioral science peer review was informed by the lessons learned from previous reorganizations in neuroscience and AIDS-related research, areas that also were affected by the arrival of the three new institutes. Stover, Sostek, and APS Charter Fellow Teresa Levitin, director of extramural review at NIDA, all were involved in the earlier reorganizations and relied on those experiences, particularly in dealing with the extramural community.

Ehrenfeld has also asked APS Charter Fellow Leonard Epstein, a professor in the departments of psychology, social and preventive medicine, and nutrition at SUNY-Buffalo, to serve as a liaison to aid in implementing the new structure and to address longer term issues relating to the review of the behavioral and social sciences even after the new system is in place.

Calling All SRAs
With the conceptual elements in place, the logistics of establishing the new system are presenting additional challenges as review staff and functions are relocated to CSR. The system will be used for the review cycle that began with the February 1st receipt date for applications, which means that June meetings of study sections will be the first time the system is operational. Virtually everyone involved agrees that the referral of applications to study sections is critical to the success of the new system. The point person for referral will be Carol Campbell, a long-time NIH review staffer who is well-respected for her knowledge of review and of the field. Getting reviewers, both old and new, acclimated to the system is also an essential first step.

But staffing the system may pose the biggest challenge: There is an urgent need for behavioral and social scientists to serve as Scientific Review Administrators, or SRAs, for the new study section. Established researchers who are interesting in making a transition into health science administration should contact Sostek ([email protected]) as soon as possible.

Hail CSR
Despite the exhaustive effort to develop the best possible structure, there are some unknowns. The big question is, will behavioral research be better off under the new system? That and related issues will be addressed by a formal evaluation effort at CSR. Samuel Rawlings, already on the CSR review staff, has been named Evaluation Officer and has convened an advisory group of nationally-recognized experts in program evaluation to assess the impact of the new system.

Evaluation committee members include: John McArdle (chair), Mark Appelbaum, Gwen Boodoo, Tom Cook, William Meredith, and Barbara Mellers. The group is in the process of organizing, but Rawlings anticipates that they will study the review system using various methods, initially doing descriptive analyses, and later conducting surveys to gauge the perceptions of change as viewed by different groups. He indicated that the group may consider conducting controlled experiments, although this poses enormous difficulty and may not be possible because of the potentially deleterious impact on review.

Evaluation of the peer review changes has been a priority for APS since the transfer of the former ADAMHA institutes was first being discussed. In a 1991 letter urging then-new Director Bernadine Healy to look at the NIMH review system as a model when considering changes to the NIH system, Kraut stressed that psychologists have considerable expertise in areas relating to review, such as multidimensional scaling, statistical analyses, and program evaluation. This was a continuing theme is APS’s discussions with NIH regarding the transfer of the three institutes.

Optimism Abounds
Given the newness of the system, NIH officials are understandably cautious when asked to predict the outcome of the changes. “It is still very early in the implementation,” observes APS Charter Fellow Alan Leshner, Director of NIDA, who championed to other NIH leaders the idea of reorganizing — not just shoehorning — behavioral science peer review. “I am confident we will get broader perspectives on projects and therefore all will benefit.” Closer to home, Leshner also hopes the new system will “bring a broader array of behavioral scientists to NIDA’s grant portfolio.”

But questions aside, there is a new and widespread optimism about the peer review system and about the overall behavioral and social science enterprise at NIH. Judging by the network of enthusiastic behavioral scientists within NIH, the involvement of the extramural community and the obvious commitment of CSR to getting it right, that optimism is well-placed. Watch for updates on the peer review system in future issues of the Observer.

Details on the new behavioral science peer review system, including descriptions of the new study sections, are available on the Web at www.csr.nih.gov/review/bssmain.htm. Applicants are urged to use the Web descriptions to provide a cover letter indicating which study section(s) are most appropriate to review their grant applications.