Surgical trial design for incorporating the effects of learning: what is the current methodological guidance, and is it sufficient?

Abstract

Background: Surgical interventions are complex. Key elements of this complexity are the surgeon and their learning curve. They pose methodological challenges in the design, analysis and interpretation of surgical RCTs. We identify, summarise, and critically examine current guidance about how to incorporate learning curves in the design and analysis of RCTs in surgery.

Examining current guidance: Current guidance presumes that randomisation must be between levels of just one treatment component, and that the evaluation of comparative effectiveness will be made via the average treatment effect (ATE). It considers how learning effects affect the ATE, and suggests solutions which seek to define the target population such that the ATE is a meaningful quantity to guide practice. We argue that these are solutions to a flawed formulation of the problem, and are inadequate for policymaking in this setting.

Reformulating the problem: The premise that surgical RCTs are limited to single-component comparisons, evaluated via the ATE, has skewed the methodological discussion. Forcing a multi-component intervention, such as surgery, into the framework of the conventional RCT design ignores its factorial nature. We briefly discuss the multiphase optimisation strategy (MOST), which for a Stage 3 trial would endorse a factorial design. This would provide a wealth of information to inform nuanced policy but would likely be infeasible in this setting. We discuss in more depth the benefits of targeting the ATE conditional on operating surgeon experience (CATE). The value of estimating the CATE for exploring learning effects has been previously recognised, but with discussion limited to analysis methods only. The robustness and precision of such analyses can be ensured via the trial design, and we argue that trial designs targeting CATE represent a clear gap in current guidance.

Conclusion: Trial designs that facilitate robust, precise estimation of the CATE would allow for more nuanced policymaking, leading to patient benefit. No such designs are currently forthcoming. Further research in trial design to facilitate the estimation of the CATE is needed.

Keywords: Design; Learning curve; Learning effects; Surgical RCTs; Surgical trials.

Fig. 1

ATEs evaluating Procedure A vs B under populations with a fixed CATE but different distributions of operating surgeon experience

Fig. 2

ATEs evaluating Procedure A vs B under the experts-only design in the same populations shown in Fig. 1. The shaded region represents an exclusion criteria based on (lack of) operating surgeon experience