New Content from Advances in Methods and Practices in Psychological Science

Why Bayesian “Evidence for H1” in One Condition and Bayesian “Evidence for H0” in Another Condition Does Not Mean Good-Enough Bayesian Evidence for a Difference Between the Conditions
Bence Palfi and Zoltan Dienes

Palfi and Dienes illustrate how the application of Bayes factors, like null hypothesis testing, can create inferential errors about differences between groups. This happens if the researchers compare simple effects in the groups instead of comparing the groups directly (i.e., if they do not test the interactions). The authors provide an example of these problems and the R script of the analyses. They also provide an app that can be used to calculate the Bayes factor for each group separately and for the interaction between groups, helping researchers develop intuitions about potential inferential mistakes.

Laypeople Can Predict Which Social-Science Studies Will Be Replicated Successfully
Suzanne Hoogeveen, Alexandra Sarafoglou and Eric-Jan Wagenmakers

In reviewing key findings from the social-science literature that have not been successfully replicated, laypeople were able to accurately predict replication success 59% of the time. Participants without a Ph.D. in psychology evaluated 27 key findings and predicted whether the finding would be replicated. When participants were also informed about the strength of the studies’ evidence, their predictive accuracy went up to 67%. The authors apply signal detection theory and show that this accuracy above chance does not appear to be due to response bias but to reflect discrimination abilities.

Boundary Conditions for the Practical Importance of Small Effects in Long Runs: A Comment on Funder and Ozer (2019)
James D. Sauer and Aaron Drummond

Funder and Ozer (2019) argued for the importance of small effects that may accumulate to have practical importance. Sauer and Drummond suggest that open data and preregistration will make small effects easier to verify, but they caution researchers to take two considerations into account. The first consideration is restricted extrapolation; researchers must be careful not to overstate the importance of small effects by extrapolating to unmeasured consequences. The second is construct validity, as a small effect on an operationalized variable may be stronger or weaker than the effect on the construct of interest.

Persons as Effect Sizes
James W. Grice, Eliwid Medellin, Ian Jones, et al.

Grice and colleagues show how to compute and report the answer to the question “What percentage of people in my study behaved or responded in a manner consistent with theoretical expectation?” For many studies, they show, researchers can calculate the percentage of participants who matched the theoretical expectation. This percentage essentially treats people as effect sizes, a concept that scientists, professionals, and laypersons can understand. This percentage can reveal novel patterns of data that further advance theories in psychological science.

Commentary on Hussey and Hughes (2020): Hidden Invalidity Among 15 Commonly Used Measures in Social and Personality Psychology
Eunike Wetzel and Brent W. Roberts

Hussey and Hughes (2020) investigated four psychometric properties relevant to the structural validity of questionnaires and scales widely used in social and personality psychology. They found that when they considered test-retest reliability, factor structure, and measurement invariance for age and gender groups, only 4% of the measures demonstrated good validity. Here, Wetzel and Roberts argue that (a) these measurement issues had not been previously ignored, (b) the models Hussey and Hughes used to test validity did not match the constructs in many of the measures, and (c) their analyses and conclusions were limited by the use of a dichotomic measurement of invariance.