Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Randomized Controlled Trial
. 2018 Jul 3;115(27):E6106-E6115.
doi: 10.1073/pnas.1711978115. Epub 2018 Jun 18.

Lack of group-to-individual generalizability is a threat to human subjects research

Aaron J Fisher  1 , John D Medaglia  2   3 , Bertus F Jeronimus  4
Affiliations
Randomized Controlled Trial

Lack of group-to-individual generalizability is a threat to human subjects research

Aaron J Fisher et al. Proc Natl Acad Sci U S A. .

Abstract

Only for ergodic processes will inferences based on group-level data generalize to individual experience or behavior. Because human social and psychological processes typically have an individually variable and time-varying nature, they are unlikely to be ergodic. In this paper, six studies with a repeated-measure design were used for symmetric comparisons of interindividual and intraindividual variation. Our results delineate the potential scope and impact of nonergodic data in human subjects research. Analyses across six samples (with 87-94 participants and an equal number of assessments per participant) showed some degree of agreement in central tendency estimates (mean) between groups and individuals across constructs and data collection paradigms. However, the variance around the expected value was two to four times larger within individuals than within groups. This suggests that literatures in social and medical sciences may overestimate the accuracy of aggregated statistical estimates. This observation could have serious consequences for how we understand the consistency between group and individual correlations, and the generalizability of conclusions between domains. Researchers should explicitly test for equivalence of processes at the individual and group level across the social and medical sciences.

Keywords: ecological fallacy; generalizability; idiographic science; replicability; research methodology.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Fig. 1.
Fig. 1.
Histograms for intraindividual (red) and interindividual (blue) correlations for four bivariate relationships in sample 1. Top depicts intraindividual correlations calculated from raw data. Bottom depicts intraindividual correlations calculated from data with temporal dependence removed.
Fig. 2.
Fig. 2.
Histograms for intraindividual (red) and interindividual (blue) correlations for RSA and HR. Left depicts intraindividual correlations calculated from raw data. Right depicts intraindividual correlations calculated from data with temporal dependence removed.
Fig. 3.
Fig. 3.
Histograms for intraindividual (red) and interindividual (blue) correlations between positive affect (PA) and negative affect (NA) in samples 3, 4, and 5. Top depicts intraindividual correlations calculated from raw data. Bottom depicts intraindividual correlations calculated from data with temporal dependence removed.
Fig. 4.
Fig. 4.
Density plots for the distributions of correlations in sample 6 between low-arousal positive affect (PA) and negative affect (NA) (Left) and high-arousal PA and NA (Right). Red indicates distributions related to the 535 individual respondents. Blue indicates the distributions of the 85 cross-sectional comparison datasets. Intraindividual correlations were calculated from raw data (Top), and data with temporal dependence removed (Bottom).
See this image and copyright information in PMC

Comment in

References

    1. Molenaar PCM. A manifesto on psychology as idiographic science: Bringing the person back into scientific psychology, this time forever. Measurement. 2004;2:201–218.
    1. Molenaar PCM, Campbell CG. The new person-specific paradigm in psychology. Curr Dir Psychol Sci. 2009;18:112–117.
    1. Quine WV. Natural Kinds. Essays in Honor of Carl G. Hempel. Springer; New York: 1969. pp. 5–23.
    1. Wilson EO. Consilience among the great branches of learning. Daedalus. 1998;127:131–149. - PubMed
    1. Carrier M. The Completeness of Scientific Theories: On the Derivation of Empirical Indicators Within a Theoretical Framework: The Case of Physical Geometry. Springer Science and Business Media; Dordrecht, The Netherlands: 2012.

Publication types