Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2025 Jan 9;25(1):4.
doi: 10.1186/s12874-025-02457-w.

Identify the underlying true model from other models for clinical practice using model performance measures

Yan Li  1
Affiliations

Identify the underlying true model from other models for clinical practice using model performance measures

Yan Li. BMC Med Res Methodol. .

Abstract

Objective: To assess whether the outcome generation true model could be identified from other candidate models for clinical practice with current conventional model performance measures considering various simulation scenarios and a CVD risk prediction as exemplar.

Study design and setting: Thousands of scenarios of true models were used to simulate clinical data, various candidate models and true models were trained on training datasets and then compared on testing datasets with 25 conventional use model performance measures. This consists of univariate simulation (179.2k simulated datasets and over 1.792 million models), multivariate simulation (728k simulated datasets and over 8.736 million models) and a CVD risk prediction case analysis.

Results: True models had overall C statistic and 95% range of 0.67 (0.51, 0.96) across all scenarios in univariate simulation, 0.81 (0.54, 0.98) in multivariate simulation, 0.85 (0.82, 0.88) in univariate case analysis and 0.85 (0.82, 0.88) in multivariate case analysis. Measures showed very clear differences between the true model and flip-coin model, little or none differences between the true model and candidate models with extra noises, relatively small differences between the true model and proxy models missing causal predictors.

Conclusion: The study found the true model is not always identified as the "outperformed" model by current conventional measures for binary outcome, even though such true model is presented in the clinical data. New statistical approaches or measures should be established to identify the casual true model from proxy models, especially for those in proxy models with extra noises and/or missing causal predictors.

Keywords: Cardiovascular disease; Clinical risk prediction model; Model performance measures; Outcome generation true model.

PubMed Disclaimer

Conflict of interest statement

Declarations. Ethics approval and consent to participate: The research protocol was approved by the University Presidential Scholarship team. Ethical approval was not applied here since this is a methodological simulation and case study aims to improve application of clinical risk prediction model on chronic diseases such as CVD (i.e., to better prevent CVD), and there is no direct involvement of identifiable human subjects in the study. Consent for publication: Not applicable. Competing interests: The authors declare no competing interests.

Figures

Fig. 1
Fig. 1
Boxplot of differences of C statistics from candidate models to the true model in univariate simulations. X axis: type of models. Y axis: Differences of C statistics from candidate models to the true model
Fig. 2
Fig. 2
Boxplot of differences of C statistics from candidate models to the true model in multivariate simulations. X axis: type of models. Y axis: Differences of C statistics from candidate models to the true model
See this image and copyright information in PMC

Similar articles

See all similar articles

References

    1. National Clinical Guideline Centre Lipid Modification Cardiovascular Risk Assessment and the Modification of Blood Lipids for the Primary and Secondary Prevention of Cardiovascular Disease Clinical Guideline Methods, Evidence and Recommendations Lipid Modification Contents. https://www.nice.org.uk/guidance/cg181; 2014.
    1. Hippisley-Cox J, Coupland C, Brindle P. Development and validation of QRISK3 risk prediction algorithms to estimate future risk of cardiovascular disease: prospective cohort study. Bmj. 2017;2099:j2099. - PMC - PubMed
    1. Li Y, Sperrin M, Ashcroft DM, Van Staa TP. Consistency of variety of machine learning and statistical models in predicting clinical risks of individual patients: longitudinal cohort study using cardiovascular disease as exemplar. BMJ. 2020;371. - PMC - PubMed
    1. Collins GS, Reitsma JB, Altman DG, Moons KG. M. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. Eur Urol. 2015;67:1142–51. - PubMed
    1. Wolff RF, et al. A tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med 170. 2019;PROBAST:51–8. - PubMed

LinkOut - more resources