Ultrasound examiners' ability to describe ovarian cancer spread using preacquired ultrasound videoclips from a selected patient sample with high prevalence of cancer spread

Abstract

Objectives: To assess the ability, as well as factors affecting the ability, of ultrasound examiners with different levels of ultrasound experience to detect correctly infiltration of ovarian cancer in predefined anatomical locations, and to evaluate the inter-rater agreement regarding the presence or absence of cancer infiltration, using preacquired ultrasound videoclips obtained in a selected patient sample with a high prevalence of cancer spread.

Methods: This study forms part of the Imaging Study in Advanced ovArian Cancer multicenter observational study (NCT03808792). Ultrasound videoclips showing assessment of infiltration of ovarian cancer were obtained by the principal investigator (an ultrasound expert, who did not participate in rating) at 19 predefined anatomical sites in the abdomen and pelvis, including five sites that, if infiltrated, would indicate tumor non-resectability. For each site, there were 10 videoclips showing cancer infiltration and 10 showing no cancer infiltration. The reference standard was either findings at surgery with histological confirmation or response to chemotherapy. For statistical analysis, the 19 sites were grouped into four anatomical regions: pelvis, middle abdomen, upper abdomen and lymph nodes. The videoclips were assessed by raters comprising both senior gynecologists (mainly self-trained expert ultrasound examiners who perform preoperative ultrasound assessment of ovarian cancer spread almost daily) and gynecologists who had undergone a minimum of 6 months' supervised training in the preoperative ultrasound assessment of ovarian cancer spread in a gynecological oncology center. The raters were classified as highly experienced or less experienced based on annual individual caseload and the number of years that they had been performing ultrasound evaluation of ovarian cancer spread. Raters were aware that for each site there would be 10 videoclips with and 10 without cancer infiltration. Each rater independently classified every videoclip as showing or not showing cancer infiltration and rated the image quality (on a scale from 0 to 10) and their diagnostic confidence (on a scale from 0 to 10). A generalized linear mixed model with random effects was used to estimate which factors (including level of experience, image quality, diagnostic confidence and anatomical region) affected the likelihood of a correct classification of cancer infiltration. We assessed the observed percentage of videoclips classified correctly, the expected percentage of videoclips classified correctly based on the generalized linear mixed model and inter-rater agreement (reliability) in classifying anatomical sites as being infiltrated by cancer.

Results: Twenty-five raters participated in the study, of whom 13 were highly experienced and 12 were less experienced. The observed percentage of correct classification of cancer infiltration ranged from 70% to 100% depending on rater and anatomical site, and the median percentage of correct classification for the 25 raters ranged from 90% to 100%. The probability of correct classification of all 380 videoclips ranged from 0.956 to 0.975 and was not affected by the rater's level of ultrasound experience. The likelihood of correct classification increased with increased image quality and diagnostic confidence and was affected by anatomical region. It was highest for sites in the pelvis, second highest for those in the middle abdomen, third highest for lymph nodes and lowest for sites in the upper abdomen. The inter-rater agreement of all 25 raters regarding the presence of cancer infiltration ranged from substantial (Fleiss kappa, 0.68 (95% CI, 0.66-0.71)) to very good (Fleiss kappa, 0.99 (95% CI, 0.97-1.00)) depending on the anatomical site. It was lowest for sites in the upper abdomen (Fleiss kappa, 0.68 (95% CI, 0.66-0.71) to 0.97 (95% CI, 0.94-0.99)) and highest for sites in the pelvis (Fleiss kappa, 0.94 (95% CI, 0.92-0.97) to 0.99 (95% CI, 0.97-1.00)).

Conclusions: Ultrasound examiners with different levels of ultrasound experience can classify correctly predefined anatomical sites as being infiltrated or not infiltrated by ovarian cancer based on video recordings obtained by an experienced ultrasound examiner, and the inter-rater agreement is substantial. The likelihood of correct classification as well as the inter-rater agreement is highest for sites in the pelvis and lowest for sites in the upper abdomen. However, owing to the study design, our results regarding diagnostic accuracy and inter-rater agreement are likely to be overoptimistic. © 2025 The Author(s). Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

Keywords: diagnostic imaging; education; gynecology; inter‐rater agreement; ovarian cancer; reliability; staging; training; ultrasound; video recordings.

Figure 1

Marginal probability of correct classification of videoclips regarding ovarian cancer infiltration of four anatomical regions (pelvis (), middle abdomen (), lymph nodes () and upper abdomen ()), depending on: (a) image quality as estimated by rater using a numeric rating scale ranging from 0 to 10, with level of diagnostic confidence set at 10 (i.e. median value for all 380 videoclips); (b) rater's diagnostic confidence expressed using a numeric rating scale ranging from 0 to 10, with image quality level set at 9 (i.e. median value for all 380 videoclips). Shading shows 95% confidence bands of the probability of correct classification. Results obtained using a generalized linear mixed model with random effects.

Figure 2

Marginal probability of correct classification of videoclips regarding ovarian cancer infiltration of four anatomical regions (pelvis, middle abdomen, lymph nodes and upper abdomen), depending on rater's level of experience (highly experienced () or less experienced ()), as well as: (a) image quality as estimated by rater using a numeric rating scale ranging from 0 to 10, with level of diagnostic confidence set at 10 (i.e. median value for all 380 videoclips); and (b) rater's diagnostic confidence expressed using a numeric rating scale ranging from 0 to 10, with image quality level set at 9 (i.e. median value for all 380 videoclips). Shading shows 95% confidence bands of the probability of correct classification. Results obtained using a generalized linear mixed model with random effects. Highly experienced was defined as ≥ 10 years' experience in performing ultrasound examinations or ≥ 100 annual ultrasound examinations to estimate spread of ovarian cancer; less experienced was defined as < 10 years' experience in performing ultrasound examinations and < 100 annual ultrasound examinations to estimate spread of ovarian cancer.