Sharing Data: It’s Time to End Psychology’s Guild Approach

Psychological science is both a discipline and a guild. Sometimes the two realities rest uneasily with one another. Such is the case when psychologists try to wear both hats while addressing the question of how to optimize the output of psychological science.

Characteristic of a guild is that its members have undergone a process of induction into the guild. Psychology researchers have learned how to perform their work in a prescribed way through the apprenticeship program known as graduate school. Growing adaptation to the guild way is tested in the initial years of a new guild members’ working life. Through a series of steps, the young guildperson progresses to journeyman – in this instance, tenured – status, and a few become recognized as masters.

The strength of the guild system is that it sets and maintains standards. The quality of the guildperson’s work, and the integrity of the guild, are assured by adherence to standard practices. But as with the guilds of old, the weakness of the psychological science guild is that it is slow to recognize the need to change in the face of a changed environment.

Looking back through history, it is easy to find many instances when a guild recognized significant change too late, with the result that the guild went into decline as the need for the guild’s craft declined. Just beyond our own lifespans, the guilds responsible for crafting the accouterments of horse and buggy transportation went into decline when the guildsmen mistook the automobile for a passing fad. Today, some say that the guildpersons responsible for printing the printed word are in such jeopardy.

Changed electronic communication capabilities may be presenting a challenge of this scope to our guild. It is the guild standard to publish the results of research in peer-reviewed journals. For faster communication, guildpersons rely on societal meetings. Very rapid communication of essential information is accomplished through the Internet.

But it is not our practice to share raw data or even unpublished journal articles with others. Though our journals and federal granting agencies require preservation of data for a period of time, requests for raw data are rarely made and even more rarely fulfilled. And the sharing of unpublished manuscripts is considered prior publication by many of the guild’s journals. That deterrent has the salutary effect of preventing the spread of information that has not been checked independently for adherence to guild norms and for integrity.

Purposive as they are, our deterrents to what might be thought of as premature sharing are the quality control mechanisms of a system built around publication in paper journals. The guild’s reward system is built around the quantity and quality of reports of research in paper journals. The guild has no reward for sharing a useful dataset with others. It punishes those who share results formally prior to publication.

Sometimes, however, there is a point at which the safeguards of one era begin to restrain the innovations of a new era. We may be experiencing such a time in psychological science.

As the Internet began to grow out of BITNET, several sciences, including chemistry, geography, physics and genetics, began to adapt to the new medium by building databases and sharing data that was basic to further development within these disciplines. Chemistry began a database of molecular structures that has become indispensable to work in chemistry. The American Physical Society has built a variety of member information services around the Internet. The geography database anchored at the University of California at Santa Barbara but managed by a consortium of institutions is coming into wide use not just by geographers but by scientists from other disciplines with an interest in how geographic location affects other variables of interest. Similar multidisciplinary utility is being afforded by the database of the human genome.

Though there are individual exceptions, psychological science has largely taken a pass on optimizing knowledge production and integration through use of electronic communication. It can be argued that our hesitance is potentially positive in some ways. When the databases of the other sciences were created, the capabilities of electronic storage, processing and communications media were a shadow of what they are today. Sciences that waited to move foursquare into the age of the computer are the beneficiaries of the developments that have gone before. Tools and powers the early adopters did not have at their disposal as they carved their place in the computer age are available for the taking to the late adopters.

But does our guild really need to optimize its management of knowledge? Our cottage industry approach to research wherein we have our labs and carry out our experiments to obtain our data to analyze and our results to publish has served us well. Why change?

The answer is becoming clearer thanks in part to innovation that has been able to occur within the approved boundaries of the guild. Though some still regard it as voodoo, meta-analysis is slowly taking its place among the statistical tools of the science and of a few guild members. The putting together of results from many separate studies to glean knowledge that presents itself only when viewed in the aggregate has been carried out so far mainly on published results rather than on raw data.

Critics have argued that results of disparate studies cannot be put together in a reliably meaningful way. Different researchers may have used slightly different protocols, analyzed the data differently than would other researchers, and may have carried out the research under conditions that were not present when allegedly comparable studies were carried out. To these criticisms, some meta-analysts are beginning to say, “You’re right.” But the fault lies not with the techniques of meta-analysis. It lies with the reports of the research.

The cottage industry approach to research has served the guild very well, and has had its utility for the science as well, in that there is a body of knowledge on whose particulars many researchers would agree. But meta-analysts are now discovering that many published articles lack sufficient information to judge the appropriateness of the reported results, to say nothing of their comparability to the results obtained by others.

Is there not a problem in the field if we cannot ascertain the comparability of results from related studies nor judge from the report of the research whether the proper analyses were carried out in the proper way?

That there is widespread agreement on the core knowledge of the field may be a testament more to the robustness of those findings than to the ways in which they were obtained. That is, the things on which we agree survived the inconsistencies in the approaches by which they were discovered. The lesson of Chinese classical art is that excessive striving for sameness can be the death of innovation. Discovery is the making of new knowledge. For discovery to occur, the right degree of freedom from prescription has to exist. But the extreme – science as pure art – is as destructive of the making of new knowledge as is rigid adherence to formula.

Meta-analysis is telling us that we do not have the optimum balance of rigor and innovation. Our practices may still be serving the guild for the moment, but the world around us has obtained the means to make its own comparisons and evaluations of the knowledge produced by our guild members. If the reason for our work were merely sophistry, that is, being clever to please ourselves, then how we went about our work would be inconsequential because we would remain the masters of our closed system.

But self satisfaction is a byproduct, not the purpose, of psychological research. Psychology exists to help us know ourselves and to improve our existence through that knowledge. Psychology cannot fulfill its purpose for being if its focus is inward. That is why we give such reverence to the admonition to “give psychology away.” Our research, even our basic research, is good to the extent that it is, or can become, good for something. Our failure to make full use of the means to expand, authenticate, and, yes, give away our knowledge, will matter because those who eventually should be our consumers have the power of choice.

After the phonograph was invented, it is rumored that “experts” had trouble deciding on the use of the invention. Some thought it might be used to record the voices of famous men, but could see little other use for it. Thomas Edison himself didn’t see the potential of his invention. He is said to have made the phonograph because he felt telephone conversation was too fleeting. Once the words were spoken, they were gone. The phonograph made it possible to record one’s words and send them to the person for whom they were recorded so the recipient could hear the words again and again.

In view of the uses to which the ability to preserve sound has been put, the vision of the experts (and of the inventor himself) were limited. Today, psychology has before it a new arsenal of ways to preserve, process, and disseminate information. We can choose to look on that array of capabilities as a nice way to do with electrons what we have done for more than a hundred years with paper. Or we can look on these capabilities as a completely new medium with the capability to transform the science and invigorate the guild – if we choose to make it so.

Next time: The objections to databases answered.

This is the first of a series by David Johnson on databases in psychological research.