Publish or Perish? Grade Yourself and Persist

Brian Nosek

Upon receiving an article relevant to my research, one ego-fueled reaction I have is to flip immediately to the references. Did they cite me? I gave up having friends and a social life for this: a citation might at least let me feel like my research is having some impact on, well, other research.

Besides being a salve for fragile self-esteem, citations serve an important purpose. Articles use citations to acknowledge the prior work that influenced the present work. The accumulated citations provide the theoretical and empirical base on which the article builds or critiques. The network of citations in a field reflects the integration of scientific investigations and the accumulation of knowledge. As a consequence, counting citations is a popular method for gauging scientific impact of individuals, departments, universities, and even nations.

Of course, the use of citation counts is not without risk. In the field of scientometrics, there are heavily discussed issues about, for example, how best to count citations, whether to include self-citations (“I am so influential on myself!”), how to deal with co-authorship, and whether the type of contribution (theoretical, empirical, methodological) or the context of the citation (praising or criticizing, central or peripheral) matters. There is not a simple answer for any of these questions. However, the most important consideration for using citation counts in decisions big or small is the fact that impact is not the same thing as quality. Very high quality work can have very little citation impact if no one makes use of it. It can be tempting to simplify a decision that is supposedly based on quality evaluation into one of impact evaluation. Assessing quality is a lot more work. Nonetheless, gauging impact can be a useful contribution to evaluation, and it can be helpful for understanding the trends and directions of the field.

The most popular use of citations is to examine cumulative impact — the total contribution of a corpus of scientific works. Figure 1 plots the total times 611 individual scientists were cited as a function of the years since they earned their PhD. These scientists were culled from 97 social-personality programs in the US and Canada, and they composed the sample for our article in Personality and Social Psychology Bulletin (Nosek et al., 2010). In this article, we examined citation trends of social-personality programs and their members and introduced an approach to assess career-stage impact — how much impact has a scientist had, taking into account how long he or she has been a scientist?

We indexed cumulative impact as an aggregation of multiple indicators, including total citation count and the h-index — which is the largest number h for which the scientist has h scientific reports that have been cited at least h times each. Cumulative impact increases with time. In this sample, nearly half of the variation in impact was accounted for by years since PhD. In other words, a person’s cumulative impact probably says a lot about how long ago they defended their dissertation. Cumulative impact is very relevant for identifying those scientists who have had remarkable, sustained impact over the course of a career, but it does not provide much insight into the impact of the rest of us. How is a 1998 PhD to understand his or her cumulative citation impact except to note that it is way less than John Cacioppo’s or Shelley Taylor’s? (Theirs really are ridiculously high.)

We created a career-stage impact indicator for assessing individuals across the career span. We first adjusted for the fact that there is more variability in citation impact among more senior scientists (see Figure 1) and then computed individual impact scores against a regression line of the expected impact given one’s seniority. The resulting index is uncorrelated with years since PhD. Also, both cumulative and career-stage impact indicators showed convergent validity with other indicators of impact, such as national awards. Cumulative impact was particularly related to lifetime-achievement awards, whereas career-stage impact was particularly related to early career awards.

With career-stage impact, one can glean how an individual compares to the expected value at his or her career stage and also compare people at different stages of their career. Of course, maintaining high impact across an entire career is very difficult, so one can expect a good deal more movement in career-stage impact for early career scientists than those who have already accumulated a large body of work. Even so, the indicators provide useful insight to early- and mid-career scholars for understanding their present impact and perhaps predicting their future contribution. For example, the two top-ranked scientists from our sample of 611 for our career-stage metric were Andrew Elliot (University of Rochester, 1994 PhD; score = 2.07, where 0 is the expected value and 1 is slightly less than a standard deviation) and Seth Kalichman (University of Connecticut, 1990 PhD; score = 1.92). Both have already had a major impact on their fields and are only about 20 years post-PhD.

Besides quantifying characteristics of individuals, career-stage indicators can be aggregated to provide novel information about programs. We combined cumulative and career-stage impact scores among the members of each of the 97 social-personality programs in our sample. The aggregated cumulative impact scores were strongly related to the most prominent ranking of programs — the U.S. News and World Report. Five of the top six for cumulative impact were in U.S. News’s Top 10. U.S. News uses a reputation-based method of assessing programs — surveyed scientists rate and rank programs in their field based on whatever criteria they choose. Sixty-five percent of the variation in cumulative impact scores across programs was accounted for by two variables — the number of faculty in the program and their average seniority. In a sense, this is exactly how it should be. Programs with more members who have been doing science for more time have had more impact than other programs. At the same time, reputation ratings and cumulative impact indicators are sometimes criticized (probably more by those that are in smaller and younger programs) as reflections of past achievement, not present strength. Career-stage impact might be more indicative of present strength, or at least provide another view on program impact. For example, with our approach, the top two social-personality programs on career-stage impact were University of Missouri and University of British Columbia. Neither appears in the top 20 for cumulative impact. Their faculty, however, were much younger than the top cumulative impact programs (averages of 16.5 and 14.9 years since PhD, respectively). None of the faculty in the top 20 programs in cumulative impact had an average of below 20 years since PhD. Also, career-stage impact is more weakly related to reputation-based rankings. Only five of the top 10 social-personality programs, according to U.S. News, appeared in the top 20 for career-stage impact. At the same time, cumulative impact and career-stage were related (correlation = .49). Places with more scholars having impact above their expected value also tended to have a higher cumulative impact.

Citation metrics are appealing indicators of impact because they are democratic — the universe of scientists “vote” by citing work that influences their own. The accumulation of those votes provides some information about how articles, scientists, and programs are shaping the field. But, critically, the distinctiveness between cumulative and career-stage impact between individuals and programs clarifies that there is no singular index of scientific contribution. Like many psychological constructs, each approach may provide a distinct perspective that has its own value and limitations for comprehending the health, variation, and progress in advancing scientific knowledge. Perhaps one of the most instructive lessons of evaluating citation impact will be to identify those articles or individuals that are doing very high quality work but are not (yet) being cited for it. Those cases might provide fertile ground for pursuing the next great advances in psychological science.

Author’s Note: For more information about the study of cumulative and career-stage impact of social-personality programs and their members, such as tables of high-impact individuals and programs and resources to replicate the methodology with the same or different samples, visit http://briannosek.com/papers/citations/. Using Publish or Perish software (http://www.harzing.com/pop.htm), you can calculate many citation-impact metrics relatively easily. Also, we created a Web calculator for computing cumulative and career-stage impact scores to compare with our sample (http://briannosek.com/papers/citations/computeimpactfactor.htm).