Fully Automated Deep Learning System for Bone Age Assessment

Abstract

Skeletal maturity progresses through discrete phases, a fact that is used routinely in pediatrics where bone age assessments (BAAs) are compared to chronological age in the evaluation of endocrine and metabolic disorders. While central to many disease evaluations, little has changed to improve the tedious process since its introduction in 1950. In this study, we propose a fully automated deep learning pipeline to segment a region of interest, standardize and preprocess input radiographs, and perform BAA. Our models use an ImageNet pretrained, fine-tuned convolutional neural network (CNN) to achieve 57.32 and 61.40% accuracies for the female and male cohorts on our held-out test images. Female test radiographs were assigned a BAA within 1 year 90.39% and within 2 years 98.11% of the time. Male test radiographs were assigned 94.18% within 1 year and 99.00% within 2 years. Using the input occlusion method, attention maps were created which reveal what features the trained model uses to perform BAA. These correspond to what human experts look at when manually performing BAA. Finally, the fully automated BAA system was deployed in the clinical environment as a decision supporting system for more accurate and efficient BAAs at much faster interpretation time (<2 s) than the conventional method.

Keywords: Artificial intelligence; Artificial neural networks (ANNs); Automated measurement; Automated object detection; Bone-age; Classification; Clinical workflow; Computer vision; Computer-aided diagnosis (CAD); Data collection; Decision support; Digital X-ray radiogrammetry; Efficiency; Machine learning; Structured reporting.

Fig. 1

Overview of a the conventional GP-BAA methodology and b our proposed fully automated BAA deep learning system

Fig. 2

Bone age distribution for radiographs of female and male left hands

Fig. 4

Overview of a deep CNN-based preprocessing engine to automatically detect a hand, generate a hand mask, and feed that into the vision pipeline to standardize images, making the trained automated BAA model invariant to differences in input images

Fig. 3

Examples of input radiographs utilized in this work. All images have varying sizes, but they were resized for the purposes of this figure

Fig. 5

Finding the optimal combination of image patch sizes and strides for optimal mask generation in the preprocessing engine. a mean Intersection over (mIoU) results were shown for all combinations of strides (2, 4, 8, 16) and image patch sizes (16 × 16, 24 × 24, 32 × 32, 40 × 40, 48 × 48, 56 × 56, 64 × 64). b Representative predicted and ground truth binary maps with the equation for Intersection over Union (IoU) for a single case. mIoU was calculated by taking the arithmetic mean of IoU values for all 280 test cases

Fig. 6

a GoogLeNet network topology that we used for this study. b The inception module, utilized in GoogLeNet, contains six convolutional layers with different kernel sizes and a pooling layer. All resultant outputs are concatenated into a single output vector

Fig. 7

Ten examples at each stage of preprocessing as described in the “Preprocessing engine” section a Input radiographs. The images have been transformed to a square shape for consistent layout. b Normalized images with consistent grayscale base and image size. c Label maps of hand (white) and non-hand (black) classes. d Generated masks for segmentation. e Final preprocessed images.

Fig. 8

CNN test accuracy with the real-time data augmentation using different styles of training. The “trained from scratch” method trains a CNN with a random weight initialization. Other methods fine-tune the ImageNet pretrained CNNs by incrementally updating weights of each fully connected (fc) layer from inception5 to conv1, detailed in Fig. 6

Fig. 9

Performance of four different methods (M1–M4) of training for female (a) and male (b) bone age assessments. M1 trains a CNN from scratch with a random weight initialization on original images down sampled to 224 × 224 pixels. M2 contains images from the automated preprocessing engine. M3 contains synthetically generated images for improving network generalization in addition to M2. M4 fine-tunes an ImageNet pretrained CNN on the preprocessed images with data augmentation turned on. “Correct” corresponds to the case where the prediction of the model is the same as the ground truth. “Within 1 year” and “within 2 years” include the cases where the network’s prediction is within 1 and 2 years, respectively. In addition, root mean squared error (RMSE) and mean average precision (mAP) were reported for the four different models to figure out how robust and well-performing each model is

Fig. 10

Selected examples of attention maps for female (upper rows) and male (lower rows) in the four major skeletal maturity stages: prepuberty, early-and-mid puberty, late puberty, and postpuberty stages [10]. Infant and toddler categories were excluded. Six representative attention maps were carefully chosen to represent the general trend for each category. a Prepuberty: BAAs from 2 to 7 years for females and 3–9 years for males. b Early-and-mid puberty: 7–13 years for females and 9–14 years for males. c Late-puberty: 13–15 years for females and 14–16 years for males. d Postpuberty: 15 and up for females and 17 years and up for males