Meta-analysis in a digitalized world: A step-by-step primer

Abstract

In recent years, much research and many data sources have become digital. Some advantages of digital or Internet-based research, compared to traditional lab research (e.g., comprehensive data collection and storage, availability of data) are ideal for an improved meta-analyses approach.In the meantime, in meta-analyses research, different types of meta-analyses have been developed to provide research syntheses with accurate quantitative estimations. Due to its rich and unique palette of corrections, we recommend to using the Schmidt and Hunter approach for meta-analyses in a digitalized world. Our primer shows in a step-by-step fashion how to conduct a high quality meta-analysis considering digital data and highlights the most obvious pitfalls (e.g., using only a bare-bones meta-analysis, no data comparison) not only in aggregation of the data, but also in the literature search and coding procedure which are essential steps in any meta-analysis. Thus, this primer of meta-analyses is especially suited for a situation where much of future research is headed to: digital research. To map Internet-based research and to reveal any research gap, we further synthesize meta-analyses on Internet-based research (15 articles containing 24 different meta-analyses, on 745 studies, with 1,601 effect sizes), resulting in the first mega meta-analysis of the field. We found a lack of individual participant data (e.g., age and nationality). Hence, we provide a primer for high-quality meta-analyses and mega meta-analyses that applies to much of coming research and also basic hands-on knowledge to conduct or judge the quality of a meta-analyses in a digitalized world.

Keywords: Digital research; Internet-based research; Mega-analysis; Meta-analysis; Overview; Research synthesis.

Fig. 1

Database for Meta-Analysis Example Taken from Villar et al. (2013) (Although we use in the following the term “our database”, we emphasize that the data belong to the Villar et al. (2013) dataset, which we adapted for our analyses in R.). Note: Study = The study number; Author = The name of the study authors; Year = Publication year of the study; ai = Number of participants, who dropped off in the progress bar condition; n1i = Number of participants, who started the progress bar condition; ci = Number of participants, who dropped off in the control condition; n2i = Number of participants, who started the control condition

Fig. 2

Estimating the Odds Ratio in our Database

Fig. 3

Change the Odds Ratio to Correlation Values and Add the Total Sample Size to the Database

Fig. 4

The Database with the Added Column Sample Variance

Fig. 5

Commented Output of your Bare-Bones Meta-Analysis

Fig. 6

Output of your Psychometric Meta-Analysis

Fig. 7

The Process of Identifying the 15 Meta-Analyses for Our Overview on Internet-Based Research. Adapted PRISMA Flow Chart for Overviews by the Authors

Fig. 8

The Four Different Search Strategies (a–d) and Two Control Strategies (e, f) Used in Our Search for Meta-Analyses