Intra- and inter-operator reproducibility of automated cloud-based carotid lumen diameter ultrasound measurement

Abstract

Background: Common carotid artery lumen diameter (LD) ultrasound measurement systems are either manual or semi-automated and lack reproducibility and variability studies. This pilot study presents an automated and cloud-based LD measurements software system (AtheroCloud) and evaluates its: (i) intra/inter-operator reproducibility and (ii) intra/inter-observer variability.

Methods: 100 patients (83M, mean age: 68±11years), IRB approved, consisted of L/R CCA artery (200 ultrasound images), acquired using a 7.5-MHz linear transducer. The intra/inter-operator reproducibility was verified using three operator's readings. Near-wall and far carotid wall borders were manually traced by two observers for intra/inter-observer variability analysis.

Results: The mean coefficient of correlation (CC) for intra- and inter-operator reproducibility between all the three automated reading pairs were: 0.99 (P<0.0001) and 0.97 (P<0.0001), respectively. The mean CC for intra- and inter-observer variability between both the manual reading pairs were 0.98 (P<0.0001) and 0.98 (P<0.0001), respectively. The Figure-of-Merit between the mean of the three automated readings against the four manuals were 98.32%, 99.50%, 98.94% and 98.49%, respectively.

Conclusions: The AtheroCloud LD measurement system showed high intra/inter-operator reproducibility hence can be adapted for vascular screening mode or pharmaceutical clinical trial mode.

Keywords: Atherosclerosis; Carotid; Cloud-based; Reliability; Reproducibility.

Fig. 1

(a) – Illustration of plaque formation in the carotid artery (Courtesy of AtheroPointÔ, Roseville, CA, USA.

Fig. 2

(a) – Workflow of AtheroCloud and its components. The tower represents server in the cloud. The arrows represent the bi-directional flow of information (Courtesy of AtheroPointÔ, Roseville, CA, USA).

Fig. 3

Routine trial mode automated tracings (red) of the carotid LD region showing lumen-intima (LI) interface for the near wall and far wall using AtheroCloud software (courtesy of AtheroPointÔ, Roseville, CA, USA). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 4

(color image) – Manual tracings (red) of the carotid LD region showing lumen-intima (LI) interface for the near wall and far wall using ImgTracer software (Courtesy of AtheroPointÔ, Roseville, CA, USA). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 5

The Bland-Altman plot for intra-operator reproducibility. Figure a1, a2, a3, a4, a5, a6, a7, a8 and a9 shows Bland-Altman plot between (Auto 1a-Auto 1b), (Auto 1b-Auto 1c), (Auto 1a-Auto 1c), (Auto 2a-Auto 2b), (Auto 2b-Auto 2c), (Auto 2a-Auto 2c), (Auto 3a-Auto 3b), (Auto 3b-Auto 3c), and (Auto 3a-Auto 3c) of LD measurements using AtheroCloud software.

Fig. 6

The Bland-Altman plot for inter-operator reproducibility. Figure a1, a2, a3, a4, a5, a6, a7, a8 and a9 shows Bland-Altman plot between (Auto 1a-Auto 2a), (Auto 1a-Auto 2b), (Auto 1a-Auto 2c), (Auto 1b-Auto 2a), (Auto 1b-Auto 2b), (Auto 1b-Auto 2c), (Auto 1c-Auto 2a), (Auto 1c-Auto 2b), and (Auto 1c-Auto 2c) of LD measurements using AtheroCloud software.

Fig. 7

The Bland-Altman plot for inter-operator reproducibility. Figure a1, a2, a3, a4, a5, a6, a7, a8 and a9 shows Bland-Altman plot between (Auto 1a-Auto 3a), (Auto 1a-Auto 3b), (Auto 1a-Auto 3c), (Auto 1b-Auto 3a), (Auto 1b-Auto 3b), (Auto 1b-Auto 3c), (Auto 1c-Auto 3a), (Auto 1c-Auto 3b), and (Auto 1c-Auto 3c) of LD measurements using AtheroCloud software.

Fig. 8

The Bland-Altman plot for inter-operator reproducibility. Figure a1, a2, a3, a4, a5, a6, a7, a8 and a9 shows Bland-Altman plot between (Auto 2a-Auto 3a), (Auto 2a-Auto 3b), (Auto 2a-Auto 3c), (Auto 2b-Auto 3a), (Auto 2b-Auto 3b), (Auto 2b-Auto 3c), (Auto 2c-Auto 3a), (Auto 2c-Auto 3b), and (Auto 2c-Auto 3c) of LD measurements using AtheroCloud software.

Fig. 9

The Bland-Altman plot for intra- and inter-observer variability. Figure a1, a2 shows Bland-Altman plot for intra-observer variability between (Manual 1a-Manual 1b), (Manual 2a- Manual 2b), and figure a3, a4, a5, and a6 for inter-observer variability between (Manual 1a- Manual 2a), (Manual 1a- Manual 2b), (Manual 1b- Manual 2a), and (Manual 1b- Manual 2b) of LD measurements using AtheroCloud software.

Fig. 10

Comparative receiver operating characteristic curves for three auto readings (Auto 1a, Auto 1b, and Auto 1c) by operator 1, taking Manual 1a, Manual 1b, Manual 2a, and Manual 2b as ground truth are shown in figures a1, a2, a3, and a4, respectively.

Fig. 11

Comparative receiver operating characteristic curves for three auto readings (Auto 2a, Auto 2b, and Auto 2c) by operator 1, taking Manual 1a, Manual 1b, Manual 2a, and Manual 2b as ground truth are shown in figures a1, a2, a3, and a4, respectively.

Fig. 12

Comparative receiver operating characteristic curves for three auto readings (Auto 3a, Auto 3b, and Auto 3c) by operator 1, taking Manual 1a, Manual 1b, Manual 2a, and Manual 2b as ground truth are shown in figures a1, a2, a3, and a4, respectively.

Fig. A1

The regression plot for intra-operator reproducibility. Figure a1, a2, a3, a4, a5, a6, a7, a8 and a9 shows regression plot between (Auto 1a-Auto 1b), (Auto 1b-Auto 1c), (Auto 1a-Auto 1c), (Auto 2a-Auto 2b), (Auto 2b-Auto 2c), (Auto 2a-Auto 2c), (Auto 3a-Auto 3b), (Auto 3b-Auto 3c), and (Auto 3a-Auto 3c) of LD measurements using AtheroCloud software.

Fig. A2

The regression plot for inter-operator reproducibility. Figure a1, a2, a3, a4, a5, a6, a7, a8 and a9 shows regression plot between (Auto 1a-Auto 2a), (Auto 1a-Auto 2b), (Auto 1a-Auto 2c), (Auto 1b-Auto 2a), (Auto 1b-Auto 2b), (Auto 1b-Auto 2c), (Auto 1c-Auto 2a), (Auto 1c-Auto 2b), and (Auto 1c-Auto 2c) of LD measurements using AtheroCloud software.

Fig. A3

The regression plot for inter-operator reproducibility. Figure a1, a2, a3, a4, a5, a6, a7, a8 and a9 shows regression plot between (Auto 1a-Auto 3a), (Auto 1a-Auto 3b), (Auto 1a-Auto 3c), (Auto 1b-Auto 3a), (Auto 1b-Auto 3b), (Auto 1b-Auto 3c), (Auto 1c-Auto 3a), (Auto 1c-Auto 3b), and (Auto 1c-Auto 3c) of LD measurements using AtheroCloud software.

Fig. A4

The regression plot for inter-operator reproducibility. Figure a1, a2, a3, a4, a5, a6, a7, a8 and a9 shows regression plot between (Auto 2a-Auto 3a), (Auto 2a-Auto 3b), (Auto 2a-Auto 3c), (Auto 2b-Auto 3a), (Auto 2b-Auto 3b), (Auto 2b-Auto 3c), (Auto 2c-Auto 3a), (Auto 2c-Auto 3b), and (Auto 2c-Auto 3c) of LD measurements using AtheroCloud software.

Fig. A5

The regression plot for intra- and inter-observer variability. Figure a1, a2 shows regression plot for intra-observer variability between (Manual 1a-Manual 1b), (Manual 2a- Manual 2b), and figure a3, a4, a5, and a6 for inter-observer variability between (Manual 1a- Manual 2a), (Manual 1a- Manual 2b), (Manual 1b- Manual 2a), and (Manual 1b- Manual 2b) of LD measurements using AtheroCloud software.