In Silico Modeling Demonstrates that User Variability During Tumor Measurement Can Affect In Vivo Therapeutic Efficacy Outcomes

Abstract

User measurement bias during subcutaneous tumor measurement is a source of variation in preclinical in vivo studies. We investigated whether this user variability could impact efficacy study outcomes, in the form of the false negative result rate when comparing treated and control groups. Two tumor measurement methods were compared; calipers which rely on manual measurement, and an automatic 3D and thermal imaging device. Tumor growth curve data were used to create an in silico efficacy study with control and treated groups. Before applying user variability, treatment group tumor volumes were statistically different to the control group. Utilizing data collected from 15 different users across 9 in vivo studies, user measurement variability was computed for both methods and simulation was used to investigate its impact on the in silico study outcome. User variability produced a false negative result in 0.7% to 18.5% of simulated studies when using calipers, depending on treatment efficacy. When using an imaging device with lower user variability this was reduced to 0.0% to 2.6%, demonstrating that user variability impacts study outcomes and the ability to detect treatment effect. Reducing variability in efficacy studies can increase confidence in efficacy study outcomes without altering group sizes. By using a measurement device with lower user variability, the chance of missing a therapeutic effect can be reduced and time and resources spent pursuing false results could be saved. This improvement in data quality is of particular interest in discovery and dosing studies, where being able to detect small differences between groups is crucial.

Keywords: 3D-TI; Tumor models; efficacy study; in silico; in vivo; study reproducibility; tumor imaging.

Figure 1.

Flow chart describing the analysis process.

Figure 2.

(a) Comparison of study average tumor volume against model predicted average volumes. Model used to predict rodent volumes averaged across the 5 users. Error bars are 95% CIs. (b) Average tumor growth of representative synthetic study data. Using the growth rate obtained from the model, synthetic study data was created in which a treatment was evaluated and initial group volumes across the groups were equal. This is referred to as the “ground truth study” and was the basis for which the impact of user variability was investigated. Error bars are 95% CIs.

Figure 3.

(a) Percentage difference from the mean (PDFM) for all users. Shown as a violin plot, users taken from valid longitudinal studies for 3D-TI (Left) and calipers (right) for a range of tumors sizes. (b) Mean of PDFM of each user. Violin plot represents average of each user’s measurement distribution, split by tumor size range. (c) Standard deviation of PDFM of each user. Violin plot represents standard deviation of each user’s measurement distribution, split by tumor size range.

Figure 4.

Diagram illustrating user bias (user 1) and inconsistency of bias compared with other users (users 2 and 3).

Figure 5.

Flow chart detailing the simulation algorithm.

Figure 6.

Percentage difference from the mean (PDFM) for large tumors. Data plotted as violin plots for the 5 users with the lowest (top) and highest (bottom) incorrect result rates for 3D-TI (left) and calipers (right).

Figure 7.

Standard deviation versus mean of PDFM. Mean and standard deviation PDFM colored by incorrect result rate for all users using 3D-TI and calipers. Data for large tumors only.