Up-and-Coming Voices: Methodology and Research Practices

As part of the 2021 APS Virtual Convention, researchers had the opportunity to connect with colleagues and present their work to the broader scientific community in 15-minute flash talks. In this collection, we highlight talks by students and early-career researchers related to methodology and research practices, which are fundamental building blocks of addressing any grand challenges of psychological science.

A Critical Review of the Utility of College Student Samples in Research on Trauma and Posttraumatic Stress Disorder   

Elizabeth L. Griffith, Ateka Contractor, Heidemarie Blumenthal, and Adriel Boals (University of North Texas) 

What did the research reveal that you didn’t already know? 

There is some stigma around using college student samples, and the area of trauma is no exception. Some journals are reluctant to publish papers that use a college student sample, and proposing to use such a sample in a grant proposal can be a pitfall for a researcher’s grant hopes. Our critical review examined results from trauma studies based on whether the sample was a college student or non-college-student sample, such as combat veterans or assault survivors. We found that obtained results did not drastically differ based on the sample type. Our results suggest that the reputation of college student samples as inferior—at least in trauma research—is unfounded. 

How might your findings improve methodology or other research practices in psychological science? 

College student samples are more convenient and easier to obtain that non-college-student samples. Our findings will hopefully encourage researchers to take advantage of this resource, which makes it much easier to obtain very large sample sizes, allowing for more statistical power to identify small effect sizes and interactions. Perhaps more importantly, college student samples are more feasible than other samples in regard to prospective longitudinal studies because such samples may minimize likelihood of attrition. Prospective studies in trauma research are rare, yet sorely needed. A prospective trauma study would involve obtaining a large sample of participants and assessing them both before and after trauma exposure; researchers would then follow this sample over a relatively long period of time, identify those who have experienced trauma since baseline, and then follow up with those participants. Such a study is more feasible with college student samples. 

A Scoping Review of Structural and Intermediary Determinants of Health and Health Inequities in the ACEs Literature: Where Does the Story Begin?

Bria Gresham (University of Minnesota) 

What did the research reveal that you didn’t already know? 

As a researcher examining the associations between adverse childhood experiences (ACEs) and outcomes, I had not considered how excluding predictors of ACEs in my research contributed to the implications of my findings. The process of quantifying the extent to which social and structural determinants of health and health inequities (i.e., both characteristics of the socioeconomic and political context and their influence on income, race, etc.) were included in the ACEs literature was illuminating. Further, I was unaware of how the way structural determinants of health inequities are included in research designs (i.e., as predictors, mediators, moderators, outcomes) impacts the narrative being told. Our findings highlighted the lack of inclusion of social and structural determinants of health inequities in the ACEs literature. Overall, the findings demonstrate that the ACEs literature is focused primarily on downstream effects of adversity, rather than upstream factors that lead to exposure to ACEs in the first place—which is critically important for the prevention of ACEs. 

How might your findings improve methodology or other research practices in the field of psychological science? 

We found that ACEs are predominantly treated as predictors of health, underemphasizing the role of structural determinants of health inequities. Now that a robust literature on the deleterious effects of ACEs has been established, I recommend moving toward a focus on prevention of ACE exposure in the first place, marking a return to the public health roots of ACEs as a construct. This involves not only including structural determinants of health inequities in research designs but also placing them at the start of the story. Future research on intermediary determinants of health (e.g., ACEs, stress, health-risk behaviors) more broadly should incorporate the causal role of structural determinants of health inequities. Policies aimed at intermediary determinants should view these within a broader framework and focus more on preventing the factors that cause their inequitable distribution than on mitigating their health impacts. 

A Systematic Review of the Literature on Measurement Invariance/Equivalence of Parenting Scales by Race and Ethnicity: Recommendations for Inclusive Parenting Research 

Violeta J. Rodriguez, Dominique L. La Barrie, Miriam C. Zegarac, and Anne Shaffer (University of Georgia) 

What did the research reveal that you didn’t already know?   

We knew that in general, there is a limited consideration of how measures of parenting function in racially and ethnically diverse groups. So although we expected that there would be some lack of evidence for measurement invariance/equivalence across diverse groups of parents, we were surprised by the extent of this omission in the literature, which has persisted essentially unchanged for decades. We also learned more about specific potential problems with the ways measurement invariance/equivalence by race and ethnicity of parenting measures is assessed. That is, we found not only that measurement invariance/equivalence of parenting measures by race and ethnicity is rarely evaluated but also that when it is evaluated, the methods are inconsistently used across studies (e.g., whether factor-analytic or item-response-theory approaches are used). There is also a striking lack of qualitative research to inform measure development, and we see this as an important strategy to future measure development that is informed by input from diverse groups of parents.  

How might your findings improve methodology or other research practices in psychological science?    

We hope that the findings will stimulate more research on the psychometrics of parenting measures with samples comprising greater racial and ethnic diversity. This research is needed to improve the validity and utility of parenting measures with groups that have been historically underrepresented in parenting research and will ensure that conclusions based on group comparisons are psychometrically sound. We also recommend that more multimethod (e.g., qualitative, mixed methods) approaches are used in either developing or refining measures to incorporate more diverse perspectives and more accurately reflect the evolving demographics of the United States. 

Historical Trends in (Mis)Reporting p Values and Statistics: A Meta-Analysis 

Yuyang Zhong (University of California, Berkeley) 

What did the research reveal that you didn’t already know? 

This project investigated the trend of p-value distributions over time and found that despite the American Psychological Association’s editorial guideline, as of 2009, to present reported statistics as equalities, many authors still chose to report p values as inequality thresholds (e.g., p < .05 instead of p = .035). The uptick in p values around .05, .01, and .001 did not decrease until very recently (after 2015), which speaks to the influence of a landmark paper on reproducibility in psychological science published by the Open Science Collaboration. This project also recalculated p values from reported test statistics to identify common errors authors have made—either rounding errors or outright inconsistent results. 

How might your findings improve methodology or other research practices in psychological science?   

This project complements open-source, manuscript cross-check software (i.e., statcheck) that has played an increasingly important role in the publication review and submission process. It also provides additional information for future research to look out for common errors. This project also provides a framework to continue this systematic review every few years to see whether trends of p-value distributions will drastically change. 

Identifying and Leveraging Social Norm Networks Guiding Energy Use in the US and India 

Rohini Majumdar, Gregg Sparkman (Princeton University), Radhika Khosla (University of Oxford), and Elke Weber (Princeton University) 

What did the research reveal that you didn’t already know? 

Although researchers have shown how the power of social norms can be harnessed to motivate prosocial behaviors, past studies have mostly focused on relationships between one or two norms and behaviors. Our work was partly inspired by the idea that behaviors are multiply determined by many norms (and attitudes), which we were able to visualize using norm networks. We found that closely interrelated norms cluster within a larger network of norms, and the ways in which norms cluster together differ across cultures. Comparing the networks in two countries (i.e., India and the United States) revealed that interventionists should target the same behavior differently in different places. For example, we found that willingness to pay more for energy-efficient air conditioners in India is related to thinking that air conditioners are important for social status and quality of life. In the United States, the same behavior is more related to injunctive norms about climate change mitigation and burden. Institutional signals from the government might be more effective in this context. 

How might your findings improve methodology or other research practices in psychological science?   

Non-WEIRD (Western, educated, industrialized, rich, and democratic) populations remain underrepresented in psychological science even though we have no reason to believe that they think or behave in the same ways as people in WEIRD cultures, or that the strategies known to be effective in one context will also be effective in another. By studying a non-WEIRD population in India, we demonstrate the value of designing culturally sensitive behavioral interventions.  Methodologically, the network approach allows us to consider norms, attitudes, and behaviors as part of a complex system of mutually influencing variables. 

One-Way-ANOVA Within-Subjects Data: The Case of Nonrandom Missingness in Skewed Distributions 

Cristian Avila (The University of Texas at Austin), Rick Sperling, Destiny Lucero, and Pedro Gonzalez Aboyte (St. Mary’s University) 
 
What did the research reveal that you didn’t already know? 

Like most students, I was taught that paired samples t tests and repeated measures analyses of variance (ANOVAs) are the appropriate tests to use when data from the same participants are collected on multiple occasions. It wasn’t until I started learning about the problems associated with missing data that I began to wonder about the trade-off between the statistical power that comes from correlations across time points (repeated-measures ANOVA) and preservation of sample size (one-way ANOVA). Previous research on this topic didn’t account for missingness and high skew simultaneously. That’s what my study did. The results were mostly consistent with my expectations, but being able to identify the specific conditions in which the one-way ANOVA did and did not outperform the repeated measures ANOVA was gratifying. 

How might your findings improve methodology or other research practices in psychological science?  

Those of us studying this topic are motivated by practical applications that could lead to greater social justice. Underresourced schools tend to have lower attendance rates and higher student mobility, so finding ways to deal with missingness is critical if educators are to produce accurate assessments of student growth over time. We also are sensitive to the fact that teacher education programs typically do not require advanced statistics courses, and many teachers feel apprehensive about conducting complex statistical analyses on their own. More sophisticated ways of managing missingness, such as multiple imputation, are unattractive options for them. If the one-way ANOVA preserves statistical power at nominal or better Type I error rates, it represents a far more accessible method of addressing missingness than advanced approaches. Hopefully, my continuing program of research will tell us more about the appropriateness of one-way ANOVA under various conditions.

The Unintentional Dilution of Voices of Color in Traditional Qualitative Analyses 

Morgan D. Mannweiler (Yale Center for Emotional Intelligence), Tse Yen Tan, Jennifer Seibyl, and Christina Cipriano (Yale University) 

What did the research reveal that you didn’t already know?   

Our research illuminated the potential harm traditional qualitative research methods can propagate—specifically, the potential to dilute responses from minority groups in our samples and the resulting lack of identity representation when informing policy. Our pilot sample was composed of Connecticut school personnel, who are primarily White women (79% female and 83% White). We applied traditional qualitative coding and random sampling methodology to our data set. Upon analyzing the full sample of educators’ sources of stress, we found that responses from educators of color were significantly more likely to fall into our “miscellaneous” category than responses from White school personnel, potentially indicating missed themes within the larger sample and compromising our goal of informing educational leaders of the causes of stress among educators. To meaningfully evolve the generalizability of results to sociodemographic subpopulations, we built upon the lessons learned from our initial study and revised our research methods. We are currently analyzing data from a national sample of educators to explore the strengths of oversampling minority groups in qualitative analyses. 

How might your findings improve methodology or other research practices in psychological science? 

Traditional qualitative methods call for representative samples when considering participant race. However, deriving codes from random samples primarily representing dominant groups in the data set can result in coding schemes that unintentionally wash out the experiences of minority members. We have taken time to critically reflect on our research biases, intentionally center voices of color, and include educators as experts in their experience throughout the coding process. Such considerations are vital to promote equitable research practices that elicit truly representative policy changes.