Towards automatic recognition of pure and mixed stones using intra-operative endoscopic digital images

Abstract

Objective: To assess automatic computer-aided in situ recognition of the morphological features of pure and mixed urinary stones using intra-operative digital endoscopic images acquired in a clinical setting.

Materials and methods: In this single-centre study, a urologist with 20 years' experience intra-operatively and prospectively examined the surface and section of all kidney stones encountered. Calcium oxalate monohydrate (COM) or Ia, calcium oxalate dihydrate (COD) or IIb, and uric acid (UA) or IIIb morphological criteria were collected and classified to generate annotated datasets. A deep convolutional neural network (CNN) was trained to predict the composition of both pure and mixed stones. To explain the predictions of the deep neural network model, coarse localization heat-maps were plotted to pinpoint key areas identified by the network.

Results: This study included 347 and 236 observations of stone surface and stone section, respectively; approximately 80% of all stones exhibited only one morphological type and approximately 20% displayed two. A highest sensitivity of 98% was obtained for the type 'pure IIIb/UA' using surface images. The most frequently encountered morphology was that of the type 'pure Ia/COM'; it was correctly predicted in 91% and 94% of cases using surface and section images, respectively. Of the mixed type 'Ia/COM + IIb/COD', Ia/COM was predicted in 84% of cases using surface images, IIb/COD in 70% of cases, and both in 65% of cases. With regard to mixed Ia/COM + IIIb/UA stones, Ia/COM was predicted in 91% of cases using section images, IIIb/UA in 69% of cases, and both in 74% of cases.

Conclusions: This preliminary study demonstrates that deep CNNs are a promising method by which to identify kidney stone composition from endoscopic images acquired intra-operatively. Both pure and mixed stone composition could be discriminated. Collected in a clinical setting, surface and section images analysed by a deep CNN provide valuable information about stone morphology for computer-aided diagnosis.

Keywords: #EndoUrology; #KidneyStones; #UroStone; #Urology; aetiological lithiasis; automatic recognition; deep learning; endoscopic diagnosis; morpho-constitutional analysis of urinary stones.

Fig. 1

Representative automatic endoscopic stone recognition results obtained before laser fragmentation (surface image). Examples of both correctly (left panel) and misclassified images (right panel; type reported on far left is not recognised by network) are shown. In situ surface images (left image of each panel) are reported for each stone composition. Ia/calcium oxalate monohydrate, IIb/ calcium oxalate dihydrate and IIIb/uric acid pure morphologies are reported in first three rows. For each mixed stone (last two rows), a mixture of the corresponding pure morphologies is visible. Activation maps (right image of each panel) show areas where network concentrates attention.

Fig. 2

Representative automatic endoscopic stone recognition results obtained after laser fragmentation (section images). Examples of both correctly (left panel) and misclassified images (right panel: type reported on far left is not recognised by network) are shown. In situ section images (left image of each panel) are reported for each stone composition. Ia/calcium oxalate monohydrate, IIb/calcium oxalate dihydrate and IIIb/uric acid pure morphologies are reported in first three rows. For each mixed stone (last two rows), a mixture of the corresponding pure morphologies is visible. Activation maps (right image of each panel) show areas where network concentrates attention.

Fig. 3

Confusion matrices for implemented deep convolutional neural network classifier obtained using the surface (A) and section (B) datasets. Each column of the matrices represents an actual stone type, while each line represents a predicted type. Green diagonal cells show number (averaged by cross‐validation) and percentage of correct predictions by trained network. Red off‐diagonal cells correspond to wrongly predicted observations. Column on far right shows positive predictive value (green numbers) and false discovery rate (red numbers). Bottom row shows sensitivity (green numbers) and the false‐negative rate (red numbers). Blue cell bottom right shows overall percentage of correct (green) and incorrect (red) predictions.