Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Jul 26;119(30):e2120377119.
doi: 10.1073/pnas.2120377119. Epub 2022 Jul 19.

Examining the generalizability of research findings from archival data

Andrew Delios  1 , Elena Giulia Clemente  2 , Tao Wu  3 , Hongbin Tan  4 , Yong Wang  5 , Michael Gordon  6 , Domenico Viganola  7 , Zhaowei Chen  1 , Anna Dreber  2   8 , Magnus Johannesson  2 , Thomas Pfeiffer  6 ; Generalizability Tests Forecasting Collaboration; Eric Luis Uhlmann  9
Affiliations

Examining the generalizability of research findings from archival data

Andrew Delios et al. Proc Natl Acad Sci U S A. .

Abstract

This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability-for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.

Keywords: archival data; context sensitivity; generalizability; reproducibility; research reliability.

PubMed Disclaimer

Conflict of interest statement

The authors declare no competing interest.

Figures

Fig. 1.
Fig. 1.
Reproductions and generalizability tests for 29 strategic management findings. Results of the generalizability tests initiative are presented separately by type of effect size estimate (eta square, coefficient, hazard or odds ratio). The leftmost column is the numeric indicator for the original finding (1 to 29) (Table 1 has detailed descriptions). The central column depicts the effect size estimates for the reproductions (same data, same analysis) and generalizability tests (different time period and/or geography, same analysis). Generalizability test estimates are based on pooled data across all new tests. Triangles (reproductions) and circles (generalizability tests) are a solid color if the effect was statistically significant at P < 0.05. Findings 25 to 29 were nonsignificant in the original report. The two rightmost columns display the sample sizes for each analysis.
See this image and copyright information in PMC

References

    1. Aguinis H., Cascio W. F., Ramani R. S., Science’s reproducibility and replicability crisis: International business is not immune. J. Int. Bus. Stud. 48, 653–663 (2017).
    1. Baker M., First results from psychology’s largest reproducibility test: Crowd-sourced effort raises nuanced questions about what counts as replication. Nature, 10.1038/nature.2015.17433 (2015). - DOI
    1. Bergh D. D., Sharp B. M., Aguinis H., Li M., Is there a credibility crisis in strategic management research? Evidence on the reproducibility of study findings. Strateg. Organ. 15, 423–436 (2017).
    1. Bohannon J., Psychology. Replication effort provokes praise—and ‘bullying’ charges. Science 344, 788–789 (2014). - PubMed
    1. Bosco F. A., Aguinis H., Field J. G., Pierce C. A., Dalton D. R., HARKing’s threat to organizational research: Evidence from primary and meta-analytic sources. Person. Psychol. 69, 709–750 (2016).