Precision of manual two-dimensional segmentations of lung and liver metastases and its impact on tumour response assessment using RECIST 1.1

Abstract

Background: Response evaluation criteria in solid tumours (RECIST) has significant limitations in terms of variability and reproducibility, which may not be independent. The aim of the study was to evaluate the precision of manual bi-dimensional segmentation of lung, liver metastases, and to quantify the uncertainty in tumour response assessment.

Methods: A total of 520 segmentations of metastases from six livers and seven lungs were independently performed by ten physicians and ten scientists on CT images, reflecting the variability encountered in clinical practice. Operators manually contoured the tumours, firstly independently according to the RECIST and secondly on a preselected slice. Diameters and areas were extracted from the segmentations. Mean standard deviations were used to build regression models and 95% confidence intervals (95% CI) were calculated for each tumour size and for limits of progressive disease (PD) and partial response (PR) derived from RECIST 1.1.

Results: Thirteen aberrant segmentations (2.5%) were observed without significant differences between the physicians and scientists; only the mean area of liver tumours (p = 0.034) and mean diameter of lung tumours (p = 0.021) differed significantly. No difference was observed between the methods. Inter-observer agreement was excellent (intra-class correlation >0.90) for all variables. In liver, overlaps of the 95% CI with the 95% CI of limits of PD or PR were observed for diameters above 22.7 and 37.9 mm, respectively. An overlap of 95% CIs was systematically observed for area. No overlaps were observed in lung.

Conclusions: Although the experience of readers might not affect the precision of segmentation in lung and liver, the results of manual segmentation performed for tumour response assessment remain uncertain for large liver metastases.

Keywords: Computed tomography; Liver; Lung; Metasatses; Response evaluation criteria in solid tumours (RECIST); Segmentation.

Fig. 1

After preselection of the tumours, manual segmentations were performed independently by the operators according to the RECIST and then on a preselected slice. Aberrant segmentations were excluded from the analysis. a. Example segmentations performed in liver. The purple line corresponds to a segmentation performed by a physician, the inner line one performed by a scientist. The outer segmentation (arrow) was excluded. b Example segmentations performed in lung. The purple line corresponds to a segmentation performed by a physician, the inner line one performed by a scientist. The outer segmentation (arrow) was excluded

Fig. 2

Regression models of the standard deviation according to maximum diameter (in mm) or area (in cm²). a Mean standard deviation according to the mean maximum diameter of the seven segmentations performed in the lung. A relative dispersion of the mean standard deviation was observed for the maximum diameter but remained below 2.5 mm whatever the size of the segmented tumour. The relative uncertainty decreased with the size. b Mean standard deviation according to mean area for each tumour in the lung. c Mean standard deviation according to mean maximum diameter for each segmented tumour in the liver (n = 6). The relative uncertainty increased with the size of the lesion. d Mean standard deviation according to mean area for each tumour in the liver

Fig. 3

The 95% confidence intervals (95% CIs) obtained for the limits of RECIST 1.1 criteria of stable disease, progressive disease (PD), and partial response (PR) using diameter. In the lung, it appeared that standard deviation decreased as diameter or area of the segmented tumour increased. The opposite was observed in the liver. a The 95% CI of the stable disease (y = x) in the lung did not cross the calculated 95% CI of the lower bound of PD (y = 1.2x). b The 95% CI of the stable disease in the lung did not cross the calculated 95% CI of the upper bound of PR (y = 0.7x). c The 95% CI of the stable disease in liver shows an overlap (blue zone) with 95% CI of the lower bound of PD. The cut-off value was x₁ = 22.7 mm (dashed line). d The 95% CI of the stable disease in the liver did cross the calculated 95% CI of the upper bound of PR (blue zone). The cut-off value was x₂ = 37.9 mm (dashed line)

Fig. 4

The 95% confidence intervals (95% CIs) obtained for the limits of criteria of stable disease, progressive disease (PD), and partial response (PR) using area. In liver, the 95% CI of area systematically overlapped across all tumour sizes for both partial response and progressive disease. a The 95% CI of the stable disease (y = x) in the lung did not cross the calculated 95% CI of the lower bound of PD (y = 1.44 x). b The 95% CI of the stable disease in the lung did not cross the calculated 95% CI of the upper bound of PR (y = 0.47 x). c The 95% CI of the stable disease in liver systematically shows an overlap (blue zone) with the 95% CI of the lower bound of PD. d The 95% CI of the stable disease in the liver always crossed the calculated 95% CI of the upper bound of PR (blue zone)