Automatic scoring of virtual mastoidectomies using expert examples

Abstract

Purpose: Automatic scoring of resident performance on a virtual mastoidectomy simulation system is needed to achieve consistent and efficient evaluations. By not requiring immediate expert intervention, the system provides a completely objective assessment of performance as well as a self-driven user assessment mechanism.

Methods: An iconic temporal bone with surgically important regions defined into a fully partitioned segmented dataset was created. Comparisons between expert-drilled bones and student-drilled bones were computed based on gradations with both Euclidean and Earth Mover's Distance. Using the features derived from these comparisons, a decision tree was constructed. This decision tree was used to determine scores of resident surgical performance. The algorithm was applied on multiple expert comparison bones and the scores averaged to provide reliability metric.

Results: The reliability metrics for the multi-grade scoring system are better in some cases than previously reported binary classification metrics. The two scoring methods given provide a trade-off between accuracy and speed.

Conclusions: Comparison of virtually drilled bones with expert examples on a voxel level provides sufficient information to score them and provide several specific quality metrics. By merging scores from different expert examples, two related metrics were developed; one is slightly faster and less accurate, while a second is more accurate but takes more processing time.

Fig. 1

The surgical simulator used in the study.

Fig. 2

The regions created from the 3D Voronoi tessellation of the segments in the 9413R dataset. This is a Right Lateral view of the regional anatomy.

Fig. 3

The resultant decision tree for Posterior canal wall thinned based on the composite feature vectors.

Fig. 4

2D histogram petal plot for complete saucerization scores using the composite method. The number of petals equal the number of items in that particular bin. A single dot represents one item. Most of the items fall along the diagonal, which means that the item’s computed score and expert-given subjective score are equal. More petal plots of the composite method can be found in Fig. 6.

Fig. 5

The decision trees determined from the data to give the best division between classes.

Fig. 6

The remainder of petal plots of the scores from the composite score method. A continuation of Fig. 4.