The Greatest Literature Never Published

Lurking in certain computers (and, in a bygone era, certain filing cabinets) lies a large body of fascinating psychological literature that has never been published and that is inaccessible in literature searches. This body of work is and has been critically important to the field despite its invisibility. It determines our scientific lives, and which papers are published and which are not. In addition, it shapes those papers that are published.

I refer to the great body of action letters and reviews that issue forth from our journal editors and their editorial boards (and ad hoc reviewers) every year. Think of how many reviews and action letters there must have been in the history of psychology; consider how many new ones are created every year. Many reviews and letters are ordinary, but some have great ideas and are really fine pieces of work. Every editor knows the feeling of wanting to reject a paper but to publish the reviews associated with it. Some are that good. Yet are there any studies of the editorial process making use of these materials (long after the fact, of course)? I know of none. How has the peer review process changed over the years? I suspect reviews have gotten longer and that the number of reviewers per manuscript more numerous, even in my time in the field, but I have no proof. Why are editorial correspondence and its attendant reviews so neglected?

I served as editor or associate editor of two journals for a total of 13 years. Editing is hard work but can also be quite rewarding. During those years, I sometimes felt that some of my best thinking and writing (at least in my better moments) lay in my action letters. Sure, many action letters are routine, summarizing the reviews and reaching a decision. And many reviews are similarly humdrum. However, at least once a month (and sometimes more often) I would see a review or set of reviews that provided deep insight into an issue and that would set the field ahead. When the reviewing process works best, the authors are greatly aided and so is the field.

Editing a journal is certainly one of the hardest jobs in academia. Most editors with whom I have spoken report that the job was worth it because they learned so much. When I was offered the editorship of the Journal of Experimental Psychology: Learning, Memory and Cognition in the 1980s, I sought advice from Endel Tulving on whether to take it. After all, editorships drain the editor’s time and can detract from one’s own research and writing. Endel told me I should definitely do it because I would learn so much, both about the field and about human nature (in the complex dance of negotiation, acceptance, and rejection with authors and reviewers). He was right. The psychology of reviewing and editing are overlooked subjects worthy of psychological investigation. (Some research does exist on these topics, but this is not the place to review it.)

Reviewers and editors put in huge amounts of work on manuscripts. OK, sometimes they completely miss the boat and sometimes (maybe most of the time) they are too picky. A favorite hobby of all scientists is regaling one another with our trials and tribulations at the hands of unenlightened reviewers; we all have horror stories of the peer review system. (Of course, while regaling others, we rarely pause to think about how many of our colleagues are howling over the reviews that we write about their work. I have observed the same people who are most offended by criticisms of their work often write the harshest reviews of others’ work.) However, when the system works well (and often it does), authors learn from knowledgeable editors and reviewers and papers get better in the process. My guess from my own editing experience is that perhaps 60 percent of accepted papers are improved in the editorial process, 25 percent stay about the same, and 15 percent get worse. Papers get worse, by and large, when reviewers are incredibly picky and the final manuscript becomes a bloated, footnote-encrusted tome that tries to refute all the criticisms. Another reason is that editors sometimes make authors write papers to their (the editors’) exact specifications, which can eviscerate what the authors wish to say and make the paper cumbersome in trying to communicate dual messages.

For the editor, the worst sin of all is bad judgment. Tulving warned me not to make the mistake of someone (he cited a name) who had edited a well-known journal and who rejected several papers that went on to be citation classics when published in other journals. An interesting study in any of our subfields would be to pick 25 papers from about 20 years ago, ones that are highly cited and greatly advanced the field, and to write to the authors for their recollections (or editorial correspondence) about acceptance of the papers. Were they easily accepted? Did prescient reviewers immediately see the great discoveries that lay within the paper? I would be surprised (but then I often am). How many of these great papers were initially rejected?

Curiously, I know of no studies in psychology of action letters and of the ability of editors or reviewers to predict the fate of papers. Can some reviewers or editors pick out, ahead of time, the papers that will generate great excitement and inspire future research directions? How well calibrated is an editor’s metacognition? To pick an example from my own field, did the editor of the Journal of Verbal Learning and Verbal Behavior (now the Journal of Memory and Language) know when he accepted Loftus and Palmer’s (1974) first study of eyewitness memory and the misinformation effect, that it was going to create great excitement and lead to a whole new research area? I suspect not, because the paper was not even the lead article in the issue (where the editor often puts what he or she thinks is most notable) but was buried next to the last in that issue (a great serial position for short-term recall but not for long-term recall, all else being equal).

Another question in this vein: What is the status of editorial correspondence after a paper has been accepted or rejected? Say I am sent a paper to review by an editor. The paper is to be reviewed in confidence under our rules (as spelled out in the APA Publication Manual). However, what is the status of the editorial correspondence (the reviews and the action letter) after the editor passes judgment? Are these confidential? The review process is over. Am I free to share editorial correspondence with others when the paper is no longer under review? If someone signed his or her review, might I quote the review in my own papers? (I sign all my reviews, so the question is of some interest.) At the very least, if I am the author of the paper, may I cite the editorial correspondence in revising my paper for resubmission? Could I quote someone else’s signed review in a book chapter I am writing?

I assume the answer should be “yes” to these last questions, but rarely do authors ever follow this practice, so the field seems to act as if the answer is “no.” As a result, reviews and action letters remain invisible even in published work. Reference to reviewers and editors is often oblique, at best. You never read:

“We are reporting Experiment 4 to rule out a wildly implausible artifact that Reviewer C thought up and the editor endorsed. We are sorry to have wasted our time and energy in doing the research, the next three pages of journal space in reporting it, and your time in reading it. As we expected, the experiment completely rules out this uninteresting idea and you can just skip ahead to the General Discussion.”

OK, the author would have to soften the tone, but you get the idea.

I believe that only once in my years of editing did authors quote an action letter in revising a paper. Gordon Hayman and Endel Tulving (JEP:LMC, 1989, pp. 229-230) quoted musings from my action letter, in which I expressed skepticism about their method (contingency analyses between memory tests) although I accepted their paper. After quoting parts of my letter at length in Footnote 2, they chided “their esteemed colleague” and explained that my assumptions were wrong. This all seemed fair to me and, in fact, led me to wonder why authors do not quote reviews and action letters more often. Some colleagues and I are currently resubmitting a paper and quoting reviews and the action letter to motivate the last of our experiments.

Although I think interesting studies could be done with editorial correspondence and reviews (probably long after the fact, for reasons of confidentiality), it may be harder to do longitudinal studies because the correspondence may not be available. No one wants to preserve this material for posterity. I was the founding editor of Psychonomic Bulletin & Review of the Psychonomic Society, and my term occurred in the mid-1990s, just before electronic submission and reviewing became common. My associate editors and I, with the help of reviewers, created several filing cabinets full of correspondence. Some of it was quite interesting; lawsuits were threatened in the case of two papers (for one reason or another; cooler heads prevailed in the end and no suits were filed). However, when I tried to find someone to take this correspondence after my term as editor, no one wanted it. The Archives for the History of American Psychology in Akron did not have room for it, even though (or maybe because) they had taken my earlier correspondence from JEP:LMC in the 1980s. (I assume the stuff is still there.) The Psychonomic Society didn’t want my records, either, but I thought they were worth saving. I imagined that some future historian of psychology might want to study them. (I sometimes have an overactive imagination.) I stashed all the correspondence in a storeroom in my department for seven years, but it became needed for other purposes. I didn’t have any place in my lab or office for it, so five years of editorial correspondence surrounding the founding of the journal wound up in the Great Dumpster of History behind our department. Sic transit gloria mundi.

Another interesting study of correspondence and reviews would be to find the classic rejection lines. One favorite I saw (written by someone else in the 1980s) told a young author (in paraphrase), “You are establishing a great reputation for yourself. This paper is not very good (see the reviews for details) and would harm your developing reputation. Therefore I am rejecting it.” Why didn’t I think of that line? “I’m rejecting your paper to save your reputation.” It would have been so handy.

Other favorite editorial rejection lines are probably apocryphal. “We would like to use your paper in connection with our journal, but unfortunately you wrote on it.” Or the more subtle approach: “Your paper fills a much needed gap in our knowledge.” I know, I know, don’t write me: The same could be said of this column.

HENRY L. ROEDIGER, III, is James McDonnell Distinguished University Professor of Psychology at Washington University in St. Louis.