A VISION Substudy of Reader Agreement on 68Ga-PSMA-11 PET/CT Scan Interpretation to Determine Patient Eligibility for 177Lu-PSMA-617 Radioligand Therapy

Abstract

[ ⁶⁸Ga]Ga-PSMA-11 ( ⁶⁸Ga-PSMA-11) is used to identify prostate-specific membrane antigen (PSMA)-positive tumors on PET scans. In the VISION study, ⁶⁸Ga-PSMA-11 was used to determine the eligibility of patients with metastatic castration-resistant prostate cancer for treatment with [¹⁷⁷Lu]Lu-PSMA-617 (¹⁷⁷Lu-PSMA-617), based on predefined read criteria. This substudy aimed to investigate the interreader variability and intrareader reproducibility of visual assessments of ⁶⁸Ga-PSMA-11 PET/CT scans using the VISION read criteria and evaluate the agreement between read results for this and the VISION study. Methods: In VISION, ⁶⁸Ga-PSMA-11 PET/CT scans were centrally read as inclusion cases if they had at least 1 PSMA-positive lesion and no PSMA-negative lesions that fulfilled the exclusion criteria. In this substudy, 125 PET/CT scans (75 inclusion and 50 exclusion cases) were randomly selected from VISION and retrospectively assessed by 3 independent central readers. A random subset of 20 cases (12 inclusion and 8 exclusion cases) was recoded for assessment of intrareader reproducibility. Classification of cases as inclusion or exclusion cases was based on the VISION read criteria. Overall interreader variability was assessed by Fleiss κ-statistics, and pairwise variability and intrareader reproducibility were assessed by Cohen κ-statistics. Results: For interreader variability, the readers agreed on 77% of cases (overall average agreement rate, 0.85; Fleiss κ, 0.60 [95% CI, 0.50-0.70]). The pairwise agreement rate was 0.82, 0.88, and 0.84, and the corresponding Cohen κ was 0.54 (95% CI, 0.38-0.71), 0.67 (95% CI, 0.52-0.83), and 0.59 (95% CI, 0.43-0.75), respectively. For intrareader reproducibility, the agreement rate was 0.90, 0.90, and 0.95, and the corresponding Cohen κ was 0.78 (95% CI, 0.49-0.99), 0.76 (95% CI, 0.46-0.99), and 0.89 (95% CI, 0.67-0.99), respectively. The number of actual VISION inclusion cases out of the total number of cases scored as inclusion in this substudy was 71 of 93 (agreement rate, 0.76; 95% CI, 0.66-0.85) for reader 1, 70 of 88 (0.80; 0.70-0.87) for reader 2, and 73 of 96 (0.76; 0.66-0.84) for reader 3. All readers agreed on 66 of 75 VISION inclusion cases. Conclusion: Moderate-to-substantial interreader agreement and substantial-to-almost perfect intrareader reproducibility for ⁶⁸Ga-PSMA-11 PET/CT scan assessment using the VISION read criteria were observed. The read rules applied in VISION can be readily learned and demonstrate good reproducibility.

Keywords: PET/CT; PSMA; prostate cancer.

Graphical abstract

FIGURE 1.

Pairwise interreader agreement (125 cases). (A) Pairwise concordance combinations. (B) Cohen κ-analysis. κ-statistic is calculated as (observed agreement – chance agreement)/(1 – chance agreement). Chance agreement is probability that readers randomly agree. Excl = cases assessed as exclusion; Incl = cases assessed as inclusion.

FIGURE 2.

Illustrative PSMA PET/CT and diagnostic CT scans for discordant cases. (A) In the left panel, transaxial slice from diagnostic CT with intravenous contrast at level of aortic arch shows left thoracic vertebral metastasis (arrow) with soft-tissue component greater than 1 cm in short axis. In the right panel, PSMA PET/CT transaxial and maximum-intensity-projection images from mint Lesion were captured from annotations by 2 of 3 readers who correctly identified PSMA-negative vertebral lesion. (B) In the left panel, transaxial slice from diagnostic CT with intravenous contrast medium at level of lung bases demonstrates left lung nodule measuring 1.5 cm in short axis. In the right panel, PSMA PET/CT transaxial and sagittal PET images from mint Lesion were captured from annotations by 2 of 3 readers who classified lung nodule as PSMA-negative. One reader may not have identified this lesion as PSMA-negative since other nodules (not shown) were PSMA-positive.

FIGURE 3.

Intrareader agreement (20 cases). (A) Concordance combinations. (B) Cohen κ-analysis. κ-statistic is calculated as (observed agreement – chance agreement)/(1 – chance agreement). Chance agreement is probability that readers randomly agree. Excl = cases assessed as exclusion; Incl = cases assessed as inclusion.

FIGURE 4.

Agreement between substudy read results and VISION eligibility read results (125 cases). (A) Concordance combinations between individual readers in substudy and eligibility assessments in VISION. (B) Proportion of actual inclusion and exclusion cases in VISION among total number of inclusion cases assessed by reader in this substudy. Excl = cases assessed as exclusion; Incl = cases assessed as inclusion.