Nakamura Heads Peer Review at NIH

Richard Nakamura

Nakamura began his career as one of the early researchers in what was to become the field of cognitive neuroscience. Knowing from an early age that he wanted to be a scientist, Nakamura took courses in the New School for Social Research and became fascinated by experimental psychology. He completed his BA in psychology at Earlham College and his graduate work in the primate research program of APS Past President Michael S. Gazzaniga at Albert Einstein College of Medicine.

His evolution from a bench scientist to a science policy leader took place against a backdrop of scientific and social change across several decades. Between undergraduate and graduate school, working on studies of juvenile delinquency, Nakamura first encountered the challenges in translating basic theoretical findings in behavioral science into effective interventions to address social problems such as recidivism among delinquents.

After obtaining his PhD in psychology from SUNY Stony Brook, Nakamura went to the National Institutes of Health (NIH) with a post-doctoral fellowship at NIMH. He then was supported by successive NRSAs from NIMH and the National Eye Institute. Nakamura soon joined the intramural program at NIMH where he studied the brain basis of visual behavior. Electrophysiological studies in the early 1980s formed the basis of theoretical papers on a possible role of cortical synchrony in behavior and decision making. He later switched to an NIMH extramural program position, and in the middle 1980s his program was at the center of congressional attacks on behavioral and animal models research. Nakamura’s success in developing the strategy for convincing legislators of the value of this research led to a full-time job as head of NIMH’s science policy office, and eventually to his appointment as Deputy Director of NIMH.

Now, as director of CSR, he leads CSR’s 450 scientists and administrative staff, overseeing their efforts to manage 80,000 incoming NIH grant applications a year and review the majority of them in CSR peer review groups. CSR holds 1,500 review meetings a year, involving about 16,000 reviewers from the scientific community.

 Alan Kraut: After many years at NIH, you’ve been named Director of the Center for Scientific Review, the place where peer review happens for most of NIH-funded research.

Richard Nakamura: Yes. We evaluate over 70% of applications coming to NIH. In general, CSR reviews the more basic science areas and the vast majority of R01s. The more complex clinically oriented mechanisms and institute-specific RFAs tend to be reviewed within the institutes. Between those poles, there’s a lot of discussion of who should do what. Remember that CSR provides merit scores for the applications we review, and the institutes take the scores and summary statements and make the award decision.

AK: What are some of the issues you’ve found facing NIH peer review? Are there changes you’re hoping to make at CSR?

RN: One issue has been to look at ways to maintain the efficiency of peer review. Taking all of the costs of running CSR and the number of applications we review, it costs less than $2,000 per application.

More importantly, we are paying a lot of attention to the efficacy of review: How good are we at top rating the grant applications that will have maximum payoff? There are those who say the only way to review applications is through review by outstanding scientists and that the main goal for CSR is engaging the best reviewers and focusing their efforts on the best applications. Another perspective is that there are objective measures out there — for instance, citation activity — and that these should be used in evaluating grant applications and the effectiveness of review. How can we effectively resolve such different perspectives? As scientists we should do research on these questions.

Bias in Peer Review

AK: It seems to me that one of the benefits of coming up in this system as a trained psychological scientist is that you have an appreciation for evaluation, statistics, and methodology that you can use in the kinds of organizational self-examinations that you’re talking about.

RN: That’s absolutely correct. In addition to the general problem of review efficacy, one of the biggest issues that I’m facing is whether our peer review is fair to all scientists. CSR’s reputation is based on a sense that we offer a fair playing field. Within that there are several questions: Is it fair across scientific areas? Are we properly evaluating the value of different kinds of science? Is it fair to sub-groupings of scientists? Is it fair to women? Is it fair to minority populations? We have found clear evidence that African Americans are receiving awards at much lower rates compared to scientists of other groups, including Hispanic scientists, White scientists, Asian scientists. This is very disturbing not only to me but to NIH generally, and Francis Collins, in particular. At his and his Advisory Council’s behest, we’ve looked at a number of different possibilities for an artifact like age differences, university differences, and so far we cannot come up with an easy explanation.

CSR has received permission to establish a research unit to really dig into these different issues, and we have set a priority on trying to determine if there is bias in our system, and if that bias could be based on race as opposed to something like quality of a scientific application. There are inherent biases in our system in that we always ask for reviewers with great scientific expertise. But this is a positive bias and we did not think that there could be a negative bias based on something like race. This has really activated NIH, and a key goal of my tenure will be to get the answer about what’s going on here.

AK: In fact, you met with some psychological scientists at the 2012 APS Convention to talk about possibilities, and I know that [APS Past President] John Cacioppo sits on your council.

RN: Yes, we’re expecting that psychological scientists will play a key role, helping us design experiments to determine if and how racial bias might be affecting our scoring system. Francis Collins has already appointed a subcommittee to his Advisory Committee to the Director that includes several psychologists to explore these issues. It reaches to the core of peer review, that we judge all scientists on an equal footing based on science quality alone.

Evaluating Changes in Peer Review

AK: Aside from the specifics of determining what may or may not be behind these apparent biased results, just the whole notion that you’re bringing science to bear on CSR procedures is an entirely new aspect.

RN: I wouldn’t say it’s entirely new. We’ve committed to doing more testing before we make changes and to establish a strong scientific base for peer review.

In addition to the issue of potential bias, we are looking to see if other forms of peer review might either be as good at a cheaper cost or better no matter what the cost in getting strong outcomes. For instance, there’s the possibility that editorial board types of review might do better than our standard face-to-face review. We are also comparing to see if video-based reviews can be as good as face-to-face reviews.

AK: And again, it’s got those same issues — what does “good” mean?

RN: Correct, developing measures of quality is essential. In general, we will do some of the things that we’ve done in the past — use questionnaires, gauge levels of satisfaction. But also we’ll be looking at whether various changes make a difference in which applications are selected? One possibility, for example, is to look at grants that were awarded after their applications were reviewed in one form of review versus another. Maybe 5 years out we could look at the resulting products, including publications, and see whether or not one set is more creative, more interesting, or has produced more compared to the others. Another possibility is to look at a measure like citation activity.

AK: Would any of this mean reviewing the same grants under two different systems? I mean, in some sense, you are the accounting system of a $30-billion NIH. And to compare systems or switch from one accounting system to another, most for-profit firms would run them in parallel for a little bit.

RN: Yes. We would like to take the same set of applications and run them through two different review processes. Because the results of such studies could easily be misinterpreted to the deficit of science as a whole, we want to be careful that there’s a lot of discussion about designs and measures and a lot of discussion on understanding what we expect to find from such kinds of studies.

It only requires a modest improvement over random choice for a peer review system to more than pay for itself, and we think that a modest improvement in that peer review system would more than pay for any increment in cost as well. But it would be really nice to have some data to show what’s improved.

We’ve essentially been doing the same form of peer review for a number of decades. We have tinkered with it a little bit. It’s very evolutionary. But other countries are doing different things, and here in the US the National Science Foundation does it differently. Understanding the pros and cons of each of these systems would be very valuable. Just because NIH has been so successful over the decades doesn’t mean we can’t do better, and it doesn’t mean another country or another agency might come up with a better system. No system of government, of manufacturing, or of human endeavor can afford to sit still. We’ve constantly got to re-think what we’re doing and figure out if we can do it better.

We’ve seen what happens when a country or an industry declares itself number one and thinks that it can rest on its laurels or just do evolutionary changes to stay on top. And we see that as the problem for NIH review as well. We need to keep moving, keep exploring, keep thinking, and be willing to take chances in order to keep NIH processes working well. Yet, we must avoid doing harm.

This is not something that’s going to be dreamed up out of my head or out of any subset of individuals here at NIH. This needs to be done as a scientific community. One of the goals here is to engage the extramural community not only in continuing the research they have done so well all along but to help us think about how to make peer review, how to make science review, a science itself. By all of us pulling together, we can emerge from this stronger than ever.

Replication in All Areas of Science

AK: Let’s talk a little bit about replication. It’s an initiative that we at APS are taking pretty seriously, as you know. But it seems to us that all scientists need to take replication seriously.

RN: NIH is very concerned about the replicability of its science. There are many cases in which attempts to replicate have shown that ability to replicate is pretty low and many interesting findings are not being replicated even though replications are being attempted.

AK: And you’re talking about science beyond psychological science?

RN: Absolutely. In some cases, as at the National Institute of Neurological Disorders and Stroke (NINDS), the director has created a fund to actively support replication studies, and they’re finding that the ability to replicate is low.

There is more interest in creating checklists of dos and don’ts of good science — the feeling is that there are too many shortcuts being taken or that some of the procedures that have long been accepted, like the 0.05% significance line, are inadequate to assure a scientific record which is valid.

A lot of self-examination is going on right now to try and make sure that those studies we are now paying for are in fact replicable and valid. But I think it’s up to all sectors of science to provide more rewards for scientists who do studies that ensure our literature is correct and valid and that all mentors and all students look at data and data collection with some skepticism to ensure that it’s being done well and done correctly. I am pleased to see that psychological scientists are so actively engaged in improving scientific methods.

Keeping the Enterprise Going

AK: I understand that the average age of an investigator getting their first funding of an NIH grant continues to go up. What about encouraging the next generation of scientists? What do you do here at CSR that would keep this scientific enterprise going?

RN: We’re all very concerned about the next generation. At CSR, we developed the Early Career Reviewer (ECR) program to help jumpstart the careers of emerging researchers by giving them an insider’s view of the NIH peer review process. They get tremendous insights into how to write a successful application and they get to network with distinguished scientists.

Of course, there is a problem here bigger than CSR. Right now, graduate students and post docs are seeing their mentors really struggling. It doesn’t matter what area, I hear clinicians fretting, translational scientists fretting, and basic scientists certainly worried. NIH has worked with the institutes to make sure that early career scientists are given breaks in receiving their initial grants.

We need to be very aware that other countries have increased their level of investment in science, in terms of the percentage of GDP, faster than the United States. A number of Asian countries have gone past the United States, and several European counties have now done so as well. These are 30- to 50-year investments, so the effect won’t be noticed right away, and by the time the effect is noticed, it will be too late. It will affect the next generation of Americans, and not just scientists.

You and I grew up in a time when United States’ investment in science was second to none, and we’ve seen the benefits of those investments. It has been fascinating to watch the growing knowledge of how science and human psychology and human behavior interact, that is our understanding of, not only the molecular basis, but the organ basis and the social basis, and how all of these different levels interact through a structure of the body. And we’re just scratching the surface. There’s so much to learn and so much science to apply to human behavior and so many ways behavior may change biology and health. It is frustrating to know that many of those problems are deeply addressable through the science that we all love.

Supporting Clinical Psychological Science

AK: Back in your NIMH days, back when you were deputy director under Steve Hyman, you were very encouraging of the newly forming clinical science movement in psychology, and you supported the development of a science-based training program in clinical psychology.

RN: Yes. It was clear that while scientific psychology was advancing rapidly, there were also strands within psychology that were not only not scientifically based, they were antiscientific.

It’s true that some scientific trends — for instance, the viewpoint that all biology was ultimately deterministic and that human behavior really was irrelevant once we understood the molecular mechanisms underlying it — proved to be incorrect. But as the science broadened, as psychology, and I would say neuroscience, started to come together to show that complex behaviors — such as social behavior, affection, love — and the neural basis of those could be visualized, one of the great revelations that has affected both our understanding of psychology and our understanding of where science will take us ultimately, has been the understanding that those emotions and behaviors can affect the capacity and the structure of the brain through direct effects on the biology of the genome.

AK: In fact, these days, to be a leading clinical psychologist, you have to know cognitive science, you have to know the biological basis of behavior, you have to know neuroscience, you have to know a fair amount of genetics.

RN: Yes. It’s been great to see the development of a science of a human behavior that spans many disciplines. I think psychological scientists are in one of the most exciting areas of discovery. Psychology has been one of the most open disciplines, and I just hope it’ll stay open and a leader in science.