What Counts As Data?

There are times when data relevant to the truth need to be ruled out of court. Consider a doctor who has been accused of treating a patient with a practice that is now known to be associated with a morbid outcome — but was not at the time of treatment. Data that clearly establish a connection between the practice and the morbid outcome are deemed inadmissible in court, which seems perfectly reasonable given that the court’s goal is to establish the doctor’s guilt or innocence, not to establish the truth.

By contrast, our goal as scientists is to establish the truth. Yet we too have constraints on what we admit as data in pursuing that truth. Data are often inadmissible because of concerns about bias. But what I find interesting is that different fields worry about different types of bias and, as a result, rule different types of data out of court.

Every field constrains the data that it is willing to take seriously. For example, in psychological science, we typically favor situations where we have sufficient data on each individual and sufficient numbers of individuals to insure that we have enough power to detect an effect of the size we are testing. In contrast, in linguistics, a single counterexample — if it directly violates a prediction — not only counts as data but can substantially weaken a linguistic theory. Psychological scientists typically don’t give single observations much weight, and even case studies of a single individual are less acceptable. But note that if there are enough observations of a single individual to analyze that individual’s data as a system unto itself (as in psychophysics), data from a small number of individuals are taken seriously. Moreover, many observations of a single individual can be theoretically important if the point of the research is not to generalize, but to make an existential claim. For example, we only need data from one child to argue that a child who is not exposed to language can introduce linguistic structure into his or her communications (although we need enough data from that child to be certain that there is linguistic structure in his communications — a single instance in this case won’t do, e.g., Goldin-Meadow, Butcher, Mylander & Dodge, 1994).

In 2006, Tanya Luhrmann, an anthropologist now at Stanford University, and I taught a course at the University of Chicago called “What counts as data?” Our goal was to explore the systematic differences in the kinds of data that anthropologists and psychologists think need to be accounted for. We were ideal co-teachers for this task: Luhrmann is an anthropologist who uses psychological methods to test her hypotheses, and I am a psychological scientist who devotes much of my research life to observing behavior in naturalistic and unconstrained contexts (although I do feel it essential to develop coding schemes that allow me to make quantitative assessments of the behaviors I observe). The course focused on specific topics that the two fields have approached differently. For example, we looked at memory, which, in psychological science, is typically considered an individual phenomenon that happens largely inside our heads, and the data relevant to memory research typically stay within these bounds (e.g., Roediger & Gallo, 2005). In contrast, Cole (2001), an anthropologist, focused on why a community in Madagascar might appear to forget a punishing part of its history and, in so doing, enlarged the phenomenon of memory — and the relevant data — to include its social dimension.

More generally, an anthropological approach produces findings that reflect the informant’s perspective and are grounded in cultural validity. The resulting thick description of human behavior in context is an excellent way to generate grounded hypotheses that have face validity. But a psychological approach provides a precision of measurement and control that makes broader generalizations and comparisons possible across groups and across studies, and enables the exploration of implicit knowledge that violates cultural understandings (for discussion, see Gaskins, 1994; Astuti & Harris 2008). Moreover, documenting a quantifiable basis for our claims sets the stage for being able to use statistics to examine the strength of cross-group similarities or differences, and to explore mechanism.

Luhrmann’s research on the Evangelical relationship with God is a good example. Using anthropological tools in her book, When God Talks Back, Luhrmann (2012) talks with people who say that they hear God speak to them. Some even develop an intimate relationship with God and put out an extra cup of coffee for Him. Having described the phenomenon of close personal relationships with God using anthropological methods, Luhrmann then turned to psychological methods to explore a mechanism by which people can achieve this intimacy with God, a mechanism that underlies prayer and that she calls absorption (Luhrmann, Nusbaum & Thisted, 2010). Taking both an anthropological and a psychological approach to a problem adds a depth of significance, validity, reliability, and robustness to a research program that neither approach can guarantee on its own.

With the burgeoning tools available in psychological science, the issue of what counts as data comes to the fore even within our own field. Allowing different kinds of data to count in psychological science gives us ways to test a hypothesis using multiple approaches and thereby strengthen our conclusions.

What I am advocating is a respect for converging operations — for being open-minded and clear about the methods we use, and recognizing that different fields may, at times, come to different conclusions because they are looking at different data.

References:

Astuti, R., & Harris, P. L. (2008). Understanding mortality and the life of the ancestors in rural Madagascar. Cognitive Science, 32, 713–740.

Cole, J. (2001). Forget Colonialism? Sacrifice and the Art of Memory in Madagascar. Berkeley, CA: University of California Press.

Gaskins, S. (1994). Integrating interpretive and quantitative methods in socialization research. Merrill-Palmer Quarterly, 40, 313–333.

Goldin-Meadow, S., Butcher, C., Mylander, C., & Dodge, M. (1994). Nouns and verbs in a self-styled gesture system: What’s in a name? Cognitive Psychology, 27, 259–319.

Luhrmann, T. M. (2012). When God Talks Back: Understanding the American Evangelical Relationship with God. New York, NY: Alfred A. Knopf.

Luhrmann, T. M., Nusbaum, H., & Thisted, R. (2010). The absorption hypothesis: Learning to hear God in Evangelical Christianity. American Anthropologist, 12, 66–78.

Roediger, H. L., III, & Gallo, D. A. (2005). Associative memory illusions. In R. F. Pohl (Ed.), Cognitive illusions: A handbook on fallacies and biases in thinking, judgment and memory (pp. 309–326). New York, NY: Psychology Press.