Student Notebook: Meta-Analyses, To Do or Not to Do

Meta-analysis is the statistical procedure for aggregating and analyzing multiple data sets on a specific topic to answer one or more questions. Meta-analyses are designed to synthesize data across studies and provide statistical evidence for a specific effect, or lack thereof. The evidence from a meta-analysis is typically stronger than any single study or literature review. Further, meta-analyses can test why effects might vary across the data sets by conducting moderator analyses.

You might be asking yourself, “Should I conduct a meta-analysis?” It’s a big question. I’ll share the pros and cons I learned from conducting my own (King, Katz, Thompson, & Macnamara, 2019).

The idea of publishing a meta-analysis can be very attractive. Well-done meta-analyses are almost a guaranteed publication, and you don’t have to worry about programming an experiment or recruiting participants and collecting their data. All you have to do is conduct a literature review that you would be doing regardless, copy over the previous results, and run a program to meta-analyze all of the data, right? Not exactly.

Conducting a meta-analysis is a long, meticulous process that may take months or even years to complete. It requires extreme attention to detail and tenacity. There are many roadblocks and issues that can arise during every step of meta-analysis that can make this process extremely frustrating. I will go through the process of conducting a meta-analysis and include some tips for overcoming these potential roadblocks based on my own experiences.

1. Framing a Question

Before you begin a meta-analysis, you first must know what question you want to answer. You should conduct a literature review on your topic of interest, see if there is a gap in the literature, determine whether or not a meta-analysis is actually needed, frame a question, and ask whether there are potential moderating variables. When you frame your question, it is critical to be as specific as possible; you do not want to make the mistake of posing a vague question. Being vague may result in an even lengthier literature-search stage.

For example, in our meta-analysis on spatial reasoning ability in twins (King et al., 2019), we framed our question in a way that made the search more manageable. We were interested in the extent to which spatial reasoning skills are determined by genetic factors and environmental factors. We decided the best way to determine this in a meta-analysis was to specifically look at the twin-study literature. We also decided that age, sex, and the type of spatial-reasoning measure used were potential moderating variables. Thus, we framed our question as, “What are the genetic and environmental contributions of spatial reasoning ability in twin studies, and do these contributions differ depending on age, sex, or type of spatial-reasoning test?” If the question was framed as “To what extent are spatial skills determined by genetic factors?”, then our literature search may have included not only twin studies, but also DNA studies, chromosome studies, and any other type of study investigating genetic contributions. If you are considering conducting a meta-analyses and find that your research question is rather broad or overly ambitious, you may want to narrow the scope of your question.

2. Completing the Search

This step in conducting a meta-analysis is perhaps the most frustrating and time-consuming. However, with proper planning and organization, lots of headaches may be avoided. Before you begin the search process, it is vital to have a search plan in place. A well-built search strategy is the basis of the meta-analysis and will be reported in your pre-registration as well as the Method section of your manuscript. The search strategy determines which studies you will evaluate and assess for eligibility and inclusion.

An important step in this process is determining what your search terms will be. They need to be specific enough to keep you from sifting through an endless number of papers, but broad enough to ensure you don’t miss relevant studies. In King et al. (2019), the search terms included twin, genetic, heritability, spatial reasoning, and spatial ability (p. 67). By including all of these terms, we decreased the likelihood of missing out on a paper because the authors used different terminology. Once you have your search terms set and have determined your inclusion and exclusion criteria (more on this soon), it is time to start the search. This can be a tedious process, and it’s easy to lose track of where you left off. Stay organized, make lots of notes, and be persistent. Use the PRISMA guide (http://www.prisma-statement.org/PRISMAStatement/) to help organize and track your search (Moher, Liberati, Tetzlaff, Altman, & The PRISMA Group, 2009). It is easy to be discouraged or exhausted by doing the same search every day for weeks or even months. It may help to bring a colleague in on your study. Many hands make light work.

3. Screening and Coding

Before conducting the search, it is crucial to determine your inclusion and exclusion criteria. In the King et al. (2019) example, we set our criteria so that each study needed to:

“be a twin study design; include data for same sex twin pairs; include a measure of spatial ability; measure heritability; report an effect size reflecting the genetic influence on spatial reasoning or enough information to compute this effect size; report the methods and results in English; and include only human participants without known disorders or disabilities” (p. 67).

Criteria such as “report the methods and results in English” may be something you wouldn’t think of until papers you can’t decipher have appeared in your search.

Screening the many potential papers to see if they meet your inclusion criteria can be exhausting and time-consuming. Once you have determined the papers that will be included in your meta-analysis, you have to organize information and code certain variables. Make sure to note the page where you found each piece of information for later reference. This part can be less straightforward than expected because every study reports things a little bit differently.

4. Running Analyses

Once the search is complete, and all of the data is organized in a spreadsheet, it is time to run the actual meta-analysis. The R programming language has packages available to run analyses for both fixed and random effects models, and it is free to use. Many free online resources can help you learn R (Harrer, Cuijpers, Furukawa, & Ebert, 2019). However, if you do not possess these skills, this would be another good time to bring in a collaborator to be on the study with you. Excel is another software option (Neyeloff, Fuchs, & Moreira, 2012).

You’ll need to learn about the decisions that go into conducting the analyses as well (e.g., selecting the type of model, adjusting for dependent samples, deciding which publication bias analyses to run).

5. Writing the Paper

You’re finally there. Know there is a good chance all this careful work has paid off.

So should you conduct a meta-analysis? Again, it is a big question. Take time to do your research, consult with colleagues, and take a hard look at your schedule for the next year. It takes a lot of time and effort, but your work will be appreciated and respected by the scientific community.

References

Harrer, M., Cuijpers, P., Furukawa, T.A, & Ebert, D. D. (2019). Doing meta-analysis in R: a hands-on guide. Retrieved from
https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/

King, M. J., Katz, D. P., Thompson, L. A., & Macnamara, B. N. (2019). Genetic and environmental influences on spatial reasoning: A meta-analysis of twin studies. Intelligence, 73, 65–77. https://doi.org/10.1016/j.intell.2019.01.001

Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the Prizma Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Annals of Internal Medicine, 151, 264–269.
http://doi.org/10.7326/0003-4819-151-4-200908180-00135

Neyeloff, J. L., Fuchs S. C., & Moreira, L. B. (2012) Meta-analyses and forest plots using a Microsoft Excel spreadsheet: Step-by-step guide focusing on descriptive data analysis. BMC Research Notes, 5, Article 52.
https://doi.org/10.1186/1756-0500-5-52