Becoming a Cumulative Science

In this space, I have been discussing urban legends in psychological science about our multiple roles within academic life. Assuming you are reading (and remembering) these Observer columns the way I did when I didn’t have to write them, I’ll briefly recapitulate some earlier points as they bear on today’s column.

Undermining Cumulative Science

Our urban legends, and the reality that underlies them, directly reward research and theorizing that emphasizes novelty and newsworthiness in publishing, grant-getting, and tenure requirements, especially in highly competitive universities. You get high marks for autonomy and brand new theories, unconnected to anybody else’s ideas, especially those of one’s mentors. That’s great, when the contribution is really new and solid, and adds to building a cumulative science. It’s not great when it turns out to be mostly recycled —  repackaged with a more newsworthy sexy label.

These legends and realities seem part of a culture that perpetuates the toothbrush problem, as I discussed in the last column. Namely, psychologists tend to treat other peoples’ theories like toothbrushes; no self-respecting individual wants to use anyone else’s. As one psychologist nearing the tenure point at a distinguished university commented after reading that column: “Young people are not only expected to make their own toothbrush, they are also expected to use it separately on every tooth. That is, we need to make small twists and tweaks and publish each in a separate paper.” In sum, the toothbrush culture undermines the building of a genuinely cumulative science, encouraging more parallel play and solo game playing, rather than building on each other’s directly relevant best work. In this column, I turn to the factors that may help build an increasingly cumulative, integrative psychological science, as seen in much exciting work already visible in our field.

Toward a Cumulative Science: Changing Values in Our Urban Legends

Common Tools. I sent my columns to an outstanding researcher-physician in the biological sciences at the University of California, Los Angeles (UCLA), who studies basic mechanisms in brain cancer. I asked him about the degree (if any) to which the concerns I raised apply also in his science and academic life. He writes:

There is considerable relevance to my world, but with some salient differences. Like yours, our world is based on competition for scarce resources that leads to vicious and irrelevant grant and journal reviews for which your comments are exactly on target. I think the key difference is that the biological sciences have done an extremely good job of being a cumulative science. This may in part be a function of the incredible cost and difficulty of creating molecular reagents and tools (i.e., modified genes, cells expressing modified genes, transgenic and knockout mice, etc). Thus, we tend to share tools because the opportunity cost of not doing so is overwhelming. This has an exceptionally favorable impact on developing a cumulative science. It is quite unlike the non-bench-based medical research, in which, development of a cumulative science is held back by the needs of each individual group to show their brilliance and to differentiate themselves (and for which there is no gain for sharing tools). I would argue that the best thing to do for the psychological sciences is to work with the community to develop a set of shared tools that can meet the needs of multiple investigators. I think as your world goes more molecular, this will be a natural process.

These comments about the fundamental importance of common tools get to the essence of what psychological science needs to become: increasingly cumulative and integrative. And while some areas of our science now have such tools, and some are rapidly getting them, others still sorely lack them. Without common tools, replication becomes difficult if not impossible, it cannot be demanded, and a toothbrush culture is perpetuated.

Common tools may be a necessary condition for becoming a cumulative science, but it’s worth remembering that many measures (e.g., the Rorschach, the TAT, the MMPI, the Szondi) were (and some still are) common tools widely used for decades, often in standard batteries in clinical psychology. Many also turned out to be disappointing. Building a cumulative science of course requires not just agreement to use particular tools, but establishing their standardized use, reliability, validity, and utility for deepening the science and moving it forward.

The exciting news is that the rapid development of increasingly sophisticated methods —  for example, for studying executive functions and working memory in cognitive science, brain imaging techniques in cognitive and social cognitive neuroscience, and the explosion of research and tools in behavioral genetics and epigenetics — is providing unprecedented opportunities, unimaginable in earlier years. There is good reason to think that psychological science is now at the brink of what can become a golden age, opening new windows into the links and reciprocal interactions between psychological and biological phenomena and processes. There were excellent examples presented at the 2008 APS meeting in Chicago, and there will be more at the upcoming May convention in San Francisco.

Robust, Replicable, Consequential Findings.  With common tools in hand, the urban legends — and the culture and values of our science community — hopefully will shift to encourage the building of a cumulative science, especially if we are deliberate about changing them. A cumulative science expects its practitioners to seek robust and replicable effects about important phenomena and processes. It puts high value on solid new findings and ideas that build on those already established, illuminating the nature and mechanisms underlying important phenomena with increasing precision and depth. It assumes speedy publication, short introductions, and minimal discussions, with replicable findings worth attention. It insists on publishing well-done failures to replicate results of other research, and takes them seriously. It makes such failures hot news, sometimes because it may reveal a serious weakness in earlier claims. But equally important, at least in some cases, examination of these replication failures brings out salient differences not explained by one-to-one relationships between variables, but rather by complex networks of interactions. A cumulative science might even welcome complex results that include such higher order interactions. Sometimes, they turn out to be remarkably informative, even if harder to process and evaluate than the simpler main effects or two-way interactions traditionally favored and expected in many areas of our science.

But What About Theory?  In a talk Amos Tversky gave, someone interrupted in a threatening voice, “But what about a theory?” Tversky’s answer was quick. Psychology should do what it does best: design good studies and describe interesting phenomena. “Theories,” he said, “I can offer a lot. But they are something that we should get to after we have a lot of data, and we should be very careful when we suggest them.”

Science thrives when important and interesting questions lead to clever data gathering and consequential findings, and ultimately to a testable (as best one can) theory.  Our field has wonderful examples of where this has been done well, and where it keeps happening. But it also seems time to rethink the expectation that tenure and professional recognition require everyone to come up with a brand new toothbrush within the first few years after training. In a cumulative science culture, one needs a different legend. One also needs graduate education and the nurturing of young faculty into competitive players within a collective science enterprise to which they can legitimately connect, and on which they can directly build, without jeopardizing their careers.

Boundary Crossing and Bridge Building.  Collaborative inter-disciplinary teams of researchers applying their distinctive tools and skills to important phenomena studied at different levels of analysis hold great promise for psychology. Such teams were described in John Cacioppo’s columns in these pages last year, and they are helping to make our field the “hub science” he documented, pursuing problems that are science-driven, rather than discipline-driven, in isolated studies by lone investigators.

The cognitive revolution was, and still is a prototype for such constructive problem-driven boundary crossing. The revolutionaries built bridges that connected psychologists with anthropologists, computer scientists, linguists, neuroscientists, and philosophers. The collaborations were natural, driven by common interests in understanding how knowledge acquisition and representation works when studied at various complementary levels. Such bridge building is now visible not just across the sub-disciplines within psychology, but stretches to other fields — from neuroscience to genetics and epigenetics, to economics, business, and more.

Bridge building opens phenomena that lie at the intersections among multiple disciplines, play out at multiple levels, and cannot be seen within the boundaries of any single discipline or captured in the work of any single investigator or lab. Such area-crossing collaborations allow the best tools and perspectives from different levels and disciplines to be focused on important questions. They also can lead to the development and implementation of new shared tools, driven by the questions that the team jointly wants to answer. (For a description of bridges in the social and personality areas see P. Van Lange (Ed.), Bridging Social Psychology: Benefits of Transdisciplinary Approaches. Mahwah, NJ: Erlbaum. 2006. In it about 60 leading personality and social psychologists describe the diverse bridges connecting their work in all directions.)

In short, problem driven interdisciplinary teams facilitate the study of interactions at nature’s natural joints, unconstrained by artificial traditional disciplinary boundaries rooted in training programs and department structures set a century ago. It is a step towards becoming the increasingly “big science” that psychology can be. As the culture of psychological science becomes increasingly cumulative and integrative, one hopes that our legends, publication practices, and tenure systems will also evolve to make such team work a viable option, not just for the well-established, but also for our young rising stars and potential future leaders.