OpenKBP-Opt: an international and reproducible evaluation of 76 knowledge-based planning pipelines

Abstract

Objective.To establish an open framework for developing plan optimization models for knowledge-based planning (KBP).Approach.Our framework includes radiotherapy treatment data (i.e. reference plans) for 100 patients with head-and-neck cancer who were treated with intensity-modulated radiotherapy. That data also includes high-quality dose predictions from 19 KBP models that were developed by different research groups using out-of-sample data during the OpenKBP Grand Challenge. The dose predictions were input to four fluence-based dose mimicking models to form 76 unique KBP pipelines that generated 7600 plans (76 pipelines × 100 patients). The predictions and KBP-generated plans were compared to the reference plans via: the dose score, which is the average mean absolute voxel-by-voxel difference in dose; the deviation in dose-volume histogram (DVH) points; and the frequency of clinical planning criteria satisfaction. We also performed a theoretical investigation to justify our dose mimicking models.Main results.The range in rank order correlation of the dose score between predictions and their KBP pipelines was 0.50-0.62, which indicates that the quality of the predictions was generally positively correlated with the quality of the plans. Additionally, compared to the input predictions, the KBP-generated plans performed significantly better (P< 0.05; one-sided Wilcoxon test) on 18 of 23 DVH points. Similarly, each optimization model generated plans that satisfied a higher percentage of criteria than the reference plans, which satisfied 3.5% more criteria than the set of all dose predictions. Lastly, our theoretical investigation demonstrated that the dose mimicking models generated plans that are also optimal for an inverse planning model.Significance.This was the largest international effort to date for evaluating the combination of KBP prediction and optimization models. We found that the best performing models significantly outperformed the reference dose and dose predictions. In the interest of reproducibility, our data and code is freely available.

Keywords: automated planning; inverse optimization; inverse problem; knowledge-based planning; open data; optimization; radiotherapy.

Figure 1.

An overview of a complete knowledge-based planning pipeline.

Figure 2.

An overview of our methods. Afull description of each component is provided in its corresponding subsection.

Figure 3.

An overview of our process. First, dose prediction models were developed with training and validation data. Second, those models predicted dose for testing data that was used by the dose mimicking models to generate KBP plans.

Figure 4.

The distribution of dose error over all KBP-generated dose (n = 1900 points in each box). Boxes indicate median and interquartile range (IQR). Whiskers extend to the minimum of 1.5 times the IQR and the most extreme outlier.

Figure 5.

The distribution ofDVHpoint differences between the reference dose and each set of KBP-generated dose. Negative differences indicate cases where the KBP-generated dose had a lower DVHpoint than the reference dose, and arrows indicate the direction where KBP-generated dose is considered better than reference dose for each DVHpoint. Boxes indicate median and IQR. Whiskers extend to the minimum of 1.5 times the IQR and the most extreme outlier.

Figure 6.

The percentage of all (a) OAR, (b) target, and (c) ROI clinical criteria that were satisfied by each KBP pipeline, which are labeled by their prediction dose score rank. The points indicate the percentage of satisfied criteria for n = 100 patients. Adashed line indicates the percentage of criteria satisfied by reference plans.