Accuracy and self-validation of automated bone age determination

Abstract

The BoneXpert method for automated determination of bone age from hand X-rays was introduced in 2009 and is currently running in over 200 hospitals. The aim of this work is to present version 3 of the method and validate its accuracy and self-validation mechanism that automatically rejects an image if it is at risk of being analysed incorrectly. The training set included 14,036 images from the 2017 Radiological Society of North America (RSNA) Bone Age Challenge, 1642 images of normal Dutch and Californian children, and 8250 images from Tübingen from patients with Short Stature, Congenital Adrenal Hyperplasia and Precocious Puberty. The study resulted in a cross-validated root mean square (RMS) error in the Tübingen images of 0.62 y, compared to 0.72 y in the previous version. The RMS error on the RSNA test set of 200 images was 0.45 y relative to the average of six manual ratings. The self-validation mechanism rejected 0.4% of the RSNA images. 121 outliers among the self-validated images of the Tübingen study were rerated, resulting in 6 cases where BoneXpert deviated more than 1.5 years from the average of the three re-ratings, compared to 72 such cases for the original manual ratings. The accuracy of BoneXpert is clearly better than the accuracy of a single manual rating. The self-validation mechanism rejected very few images, typically with abnormal anatomy, and among the accepted images, there were 12 times fewer severe bone age errors than in manual ratings, suggesting that BoneXpert could be safer than manual rating.

Figure 1

Flowchart of the software architecture. The architecture is dominated by the six steps of the selfvalidation. The first two steps validate individual bone contributions, while the last four validate the entire image.

Figure 2

Posterior-anterior left-hand radiograph with bone age assessment by BoneXpert. This is the result file automatically placed in the same study in the digital archive. BA = bone age, averaged over the 21 tubular bones; BHI = Bone Health Index; SDS = standard deviation score; GP = Greulich-Pyle; TW3 = Tanner-Whitehouse Version 3.

Figure 3

The agreement between automated and manual rating in the three Tübingen studies of children with Short Stature, Congenital Adrenal Hyperplasia or Precocious Puberty. The manual ratings are the original manual ratings without any corrections. The lines at ± 1.8 years define the disputed cases to be rerated 3 times. On the horizontal axis is the average of manual and automated bone age rating.

Figure 4

Bland Altmann plot for the agreement between the automated rating and the manual rating for the 200 cases in the RSNA test set.

Figure 5

Bland Altman plot of the agreement between automated and manual bone age rating below 2 y.

Figure 6

Females 2707 and 3781 from the RSNA study. The automated method accepted only 9 and 10 of the 21 tubular bones, which was just above the 8 bones needed for the entire analysis to succeed. In the first image, the overlap between radius and lunate is likely to have caused radius to be rejected; and the pose of the middle and distal phalanges in finger 2–4 is incorrect due to flexion. In the second image, there is brachydactyly in the second and fifth middle phalanges, and the second and third proximal phalanges have thick epiphyses, suggesting the monogenic disorder Brachydactyly Type C. For clarity, the bone contours are drawn in alternating colours. The numbers indicate the GP bone ages.

Figure 7

Two females, 4210 and 13,824, from the RSNA study with 10 and 9 accepted tubular bones. Ray 2 is missing in the first image, which causes the Bone Health Index analysis to fail (indicated by the code 3). The radial epiphysis is abnormally thickened and there is pronounced positive ulnar variance. The first digit is likely to be a pollicised index finger following surgery for thumb aplasia or severe hypoplasia; a likely case of radial longitudinal reduction defect. The second image shows brachydactyly in the middle phalanges. Finger 1 has two epiphyses in the metacarpal, and either a thick epiphysis in the distal phalanx or an extra short phalanx; the thumb is somewhat ‘finger-like’. The second and fifth digits similarly appear biphalangeal with large epiphyses of the distal phalanges. Digits 3 and 4 show middle phalangeal hypoplasia. Appearances suggest an underlying genetic disorder of hand-development requiring further evaluation.