Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2007 Jun-Jul;31(4-5):338-45.
doi: 10.1016/j.compmedimag.2007.02.004. Epub 2007 Mar 23.

Improvement of bias and generalizability for computer-aided diagnostic schemes

Affiliations

Improvement of bias and generalizability for computer-aided diagnostic schemes

Qiang Li. Comput Med Imaging Graph. 2007 Jun-Jul.

Abstract

Computer-aided diagnostic (CAD) schemes have been developed for assisting radiologists in the detection of various lesions in medical images. The reliable evaluation of CAD schemes is as important as the development of such schemes in the field of CAD research. In the past, many evaluation approaches, such as the resubstitution, leave-one-out, cross-validation, and hold-out methods, have been proposed for evaluating the performance of various CAD schemes. However, some important issues in the evaluation of CAD schemes have not been analyzed systematically, either theoretically or experimentally. Such important issues include (1) the analysis and comparison of various evaluation methods in terms of some characteristics, in particular, the bias and the generalization performance of trained CAD schemes; (2) the analysis of pitfalls in the incorrect use of various evaluation methods and the effective approaches to reduction of the bias and variance caused by these pitfalls; (3) the improvement of generalizability for CAD schemes trained with limited datasets. This article consists of a series of three closely related studies that address the above three issues. We believe that this article will be useful to researchers in the field of CAD research who can improve the bias and generalizability of their CAD schemes.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Generalization performance levels obtained with the resubstitution, leave-one-out, and hold-out methods for 100 trials of Monte Carlo experiments
Figure 2
Figure 2
Figure 2(a) Generalization accuracy and mean estimated accuracy for the resubstitution method Figure 2(b) Generalization accuracy and mean estimated accuracy for the leave-one-out method Figure 2(c) Generalization accuracy and mean estimated accuracy for the hold-out method
Figure 2
Figure 2
Figure 2(a) Generalization accuracy and mean estimated accuracy for the resubstitution method Figure 2(b) Generalization accuracy and mean estimated accuracy for the leave-one-out method Figure 2(c) Generalization accuracy and mean estimated accuracy for the hold-out method
Figure 2
Figure 2
Figure 2(a) Generalization accuracy and mean estimated accuracy for the resubstitution method Figure 2(b) Generalization accuracy and mean estimated accuracy for the leave-one-out method Figure 2(c) Generalization accuracy and mean estimated accuracy for the hold-out method
Figure 3
Figure 3
Average specificities estimated by a full (F) and three partial (P) leave-one-out evaluation methods at a fixed sensitivity of 0.84
Figure 4
Figure 4
Average specificities estimated by the three- and two-subset evaluation methods at a fixed sensitivity of 0.84
Figure 5
Figure 5
Average specificities (disks) and standard deviations (bars) estimated by the cross-validation and hold-out evaluation methods at a fixed sensitivity of 0.84
Figure 6
Figure 6
The underlying probability density function, a random sample of 25 data, 25 kernel functions, and the estimated probability density function
Figure 7
Figure 7
Mean specificities at a fixed sensitivity of 80% for the Monte Carlo simulation experiment
Figure 8
Figure 8
Mean numbers of false positives per case at a fixed sensitivity of 80% for the real CT cases

Similar articles

Cited by

References

    1. Chan HP, Doi K, Vyborny CJ, Schmidt RA, Metz CE, Lam KL, Ogura T, Wu Y, MacMahon H. Improvement in radiologists’ detection of clustered microcalcifications on mammograms: The potential of computer-aided diagnosis. Invest Radiol. 1990;25:1102–1110. - PubMed
    1. Kobayashi T, Xu X, MacMahon H, Metz CE, Doi K. Effect of a computer-aided diagnosis scheme on radiologists’ performance in detection of lung nodules on radiographs. Radiology. 1996;199:843–848. - PubMed
    1. Aoyama M, Li Q, Katsuragawa S, Doi K. Automated computerized scheme for distinction between benign and malignant solitary pulmonary nodules on chest images. Med Phys. 2002;29:701–708. - PubMed
    1. Aoyama M, Li Q, Katsuragawa S, Li F, Sone S, Doi K. Computerized scheme for determination of the likelihood measure of malignancy for pulmonary nodules on low-dose CT images. Med Phys. 2003;30:387–394. - PubMed
    1. D.J.C. MacKay. Bayesian methods for adaptive models. PhD Thesis from California Institute of Technology, 1992. http://wol.ra.phy.cam.ac.uk/mackay/thesis.pdf

Publication types