Detailed statistical assessment of the characteristics of the ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS) threshold rules

Abstract

Background: The European Society for Medical Oncology (ESMO) has developed the ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS), a tool to assess the magnitude of clinical benefit from new cancer therapies. Grading is guided by a dual rule comparing the relative benefit (RB) and the absolute benefit (AB) achieved by the therapy to prespecified threshold values. The ESMO-MCBS v1.0 dual rule evaluates the RB of an experimental treatment based on the lower limit of the 95%CI (LL95%CI) for the hazard ratio (HR) along with an AB threshold. This dual rule addresses two goals: inclusiveness: not unfairly penalising experimental treatments from trials designed with adequate power targeting clinically meaningful relative benefit; and discernment: penalising trials designed to detect a small inconsequential benefit.

Methods: Based on 50 000 simulations of plausible trial scenarios, the sensitivity and specificity of the LL95%CI rule and the ESMO-MCBS dual rule, the robustness of their characteristics for reasonable power and range of targeted and true HRs, are examined. The per cent acceptance of maximal preliminary grade is compared with other dual rules based on point estimate (PE) thresholds for RB.

Results: For particularly small or particularly large studies, the observed benefit needs to be relatively big for the ESMO-MCBS dual rule to be satisfied and the maximal grade awarded. Compared with approaches that evaluate RB using the PE thresholds, simulations demonstrate that the MCBS approach better exhibits the desired behaviour achieving the goals of both inclusiveness and discernment.

Conclusions: RB assessment using the LL95%CI for HR rather than a PE threshold has two advantages: it diminishes the probability of excluding big benefit positive studies from achieving due credit and, when combined with the AB assessment, it increases the probability of downgrading a trial with a statistically significant but clinically insignificant observed benefit.

Keywords: Clinical Benefit; ESMO-MCBS; Evaluation of Clinical Benefit; Threshold Rule.

Figure 1

The three critical evaluation steps of ESMO-MCBS.

Figure 2

Maximum observed HR to achieve statistical significance (two-sided alpha=0.05) and maximum observed HR that satisfies the LL95%CI rule for maximal RB, for a range of design HRs. Note: design HR=true HR. (A) OS, median control ≤12 months; PFS, all medians: LL95%CI ≤0.65. (B) OS: median control >12 months: LL95%CI ≤0.70. Annotation: Symbols and shaded area: red for power 80%, blue for power 90%. Stars: maximum observed HR to achieve statistical significance (two-sided significance level 0.05). Circles: maximum observed HR that would satisfy the LL95%CI rule for maximal RB score for significant trials. Progressively smaller vertical linear segments: corresponding estimated 95% CIs (for observed HR). Dotted horizontal lines with median control/gain values listed on the right y-axis: observed HR that would correspond to required AB gains and control medians, that is, HR=0.80 broadly corresponds to a 3-month gain for a 12-month median control. Non-shaded area in the figures: range for design HR for which stars and circles are identical, representing trials targeting relatively big benefit. Shaded area in the figures: range for design HR starting above a certain value, beyond which circles are progressively lower than stars, representing trials targeting smaller benefit.

Figure 3

Comparison of the LL95%CI rule to a PE threshold for maximal RB classification: sensitivity and specificity for the MCBS LL95%CI ≤0.65 rule versus the ROC curve for PE rules ranging from 0.60 to 0.90 (time-to-event median control=6 months). (A–C) Power 80%. (D–F) Power 90%. Annotation: Left and centre panels: true HR ≤0.65 and HR≤0.70, are respectively classified as big benefit, contrasted to a small benefit classification of true HR>0.80. Right panels: true HR ≤0.75 is classified as a relatively big benefit, contrasted to a small benefit classification of true HR>0.75. Red star: the MCBS LL95%CI≤0.65 rule for HR. ROC curve (black): PE rules ranging from 0.60 to 0.90.

Figure 4

Comparison of the LL95%CI rule with PE rules with similar behaviour: %acceptance of maximal RB for power 80% and 90% over all trials, for LL95%CI ≤0.65 rule, PE <0.75 and PE <0.80. Note: Initially, for big benefit studies, the %acceptance rate of maximal RB is higher when designed with 90% power versus with 80% power. Crossing of 80% and 90% power lines for: LL95%CI ≤0.65 (red lines) occurs at HR=0.73 with %acceptance 80%; PE <0.75 (green lines) occurs at HR=0.75 with %acceptance 50%; PE <0.80 (blue lines) occurs at HR=0.80 with %acceptance 50%.

Figure 5

Comparison of the LL95%CI rule to PE rules with similar behaviour, when the true HR is different from the design HR: %acceptance of maximal relative benefit, over all trials, for true HR of 0.90, 0.75 and 0.65, over a range of design HRs. Annotation: Panel headers: true HR (0.65, 0.75  and 0.90) is indicated above design HR (0.65 to 0.90). Height of the bars: proportion out of the 50 000 simulated trials satisfying each condition. Grey bars: proportion of simulated trials found statistically significant (at two-sided significance level 0.05). Red, green and blue bars: %acceptance using LL95%CI ≤0.65, PE of 0.75 and 0.80, respectively. Grey-shaded areas: scenarios with design HR=true HR. Orange-shaded areas: implausible scenarios (eg, when true HR=0.65 and design HR≥0.85).

Figure 6

Comparison of the ESMO-MCBS dual rule and the dual rule with PE thresholds exhibiting similar behaviour: %acceptance of maximal RB (solid line: RB rule) and maximal preliminary grade (dotted line: dual rule) over significant trials for different true HRs. (A-B) PFS, 80% power - 90% power. Note: Median control PFS ≤6 months: LL95%CI ≤0.65 rule vs PE<0.75 and PE<0.80; and gain ≥1.5 months.  Median control PFS >6 months: LL95%CI ≤0.65 rule vs PE<0.75 and PE<0.80; and gain ≥3 months. (C-D) OS, 80% power - 90% power. Note:  Median control OS ≤12 months: LL95%CI ≤0.65 rule vs PE<0.75 and PE<0.80; and gain ≥3 months. Median control OS >12 months (24 months): LL95%CI ≤0.70 rule vs PE<0.80 and PE<0.83; and gain ≥5 months.

Figure 7

Field testing results relative to the maximal preliminary grade and maximum observed HR (95% CI) for primary outcome PFS—study power 80%. Annotation: Squares: actual study information (y-axis: study observed HR and 95%CI; x-axis: study design HR); Green squares: study meeting both maximal RB & AB criteria; Blue squares: study meeting maximal RB but not AB criteria; Red squares: study meeting neither criterion; Other symbols, lines and shaded areas as in Fig. 2.