Performance of four centralized statistical monitoring methods for early detection of an atypical center in a multicenter study

Abstract

Background: Ensuring the quality of data is essential for the credibility of a multicenter clinical trial. Centralized Statistical Monitoring (CSM) of data allows the detection of a center in which the distribution of a specific variable is atypical compared to other centers. The ideal CSM method should allow early detection of problem and therefore involve the fewest possible participants.

Methods: We simulated clinical trials and compared the performance of four CSM methods (Student, Hatayama, Desmet, Distance) to detect whether the distribution of a quantitative variable was atypical in one center in relation to the others, with different numbers of participants and different mean deviation amplitudes.

Results: The Student and Hatayama methods had good sensitivity but poor specificity, which disqualifies them for practical use in CSM. The Desmet and Distance methods had very high specificity for detecting all the mean deviations tested (including small values) but low sensitivity with mean deviations less than 50%.

Conclusion: Although the Student and Hatayama methods are more sensitive, their low specificity would lead to too many alerts being triggered, which would result in additional unnecessary control work to ensure data quality. The Desmet and Distance methods have low sensitivity when the deviation from the mean is low, suggesting that the CSM should be used alongside other conventional monitoring procedures rather than replacing them. However, they have excellent specificity, which suggests they can be applied routinely, since using them takes up no time at central level and does not cause any unnecessary workload in investigating centers.

Keywords: Centralized Statistical monitoring; Data quality; Multicenter clinical trial; Sensitivity; Specificity.

Fig. 1

Sensitivity and specificity of each centralized statistical monitoring method to detect the atypical center under base case scenario Footnotes to Fig. 1 This figure explores the sensitivity and specificity of each of the four methods to detect that a center is atypical in the distribution of a continuous variable Y, in a simulated trial including 10 centers with the same number of participants (50) in each center. The results are shown with: • (${\bar{y}}_{a.}$- ${\bar{y}}_{na.})$/ ${\bar{y}}_{na.}$ varying from 10 to 100% (horizontal axis); With ${\bar{y}}_{na.}$ = mean of the Y values in the non-atypical centers and ${\bar{y}}_{a.}$ = mean of the Y values in the atypical center. • ${\bar{y}}_{na.}$ absolute values of 10, 100, 1000 and 10,000 (coloured curves), to ensure that the model is robust and shows similar results irrespective of the absolute value of the continuous variable studied. In these simulations, the ratio N_a/N remains constant (1/10).

Fig. 2

Sensitivity and specificity of each centralized statistical monitoring method to detect the atypical center when different number of participants are included in the atypical center Footnotes to Fig. 2 This figure explores the sensitivity and specificity of each of the four methods to detect that a center is atypical in the distribution of a continuous variable Y, in a simulated trial including 10 centers with the same number of participants in each center. The results are shown with: • (${\bar{y}}_{a.}$- ${\bar{y}}_{na.})$/ ${\bar{y}}_{na.}$ varying from 10 to 100% (horizontal axis); With ${\bar{y}}_{na.}$ = mean of the Y values in the non-atypical centers and ${\bar{y}}_{a.}$ = mean of the Y values in the atypical center. • the number of participants per center varying from 10 to 300 (coloured curves). In these simulations, the ratio N_a/N remains constant (1/10).

Fig. 3

Sensitivity and specificity of each centralized statistical monitoring method to detect the atypical center when varying the ratio $N_a/N$ of the number of participants in the atypical center on the overall number of participants in the study Footnotes to Fig. 3 This figure explores the sensitivity and specificity of each of the four methods to conclude that a center is atypical in the distribution of a continuous variable Y in a simulated trial including a varying number of centers with the same number of participants in each center. The results are shown with: • (${\bar{y}}_{a.}$- ${\bar{y}}_{na.})$/ ${\bar{y}}_{na.}$ varying from 10 to 100% (horizontal axis); With ${\bar{y}}_{na.}$ = mean of the Y values in the non-atypical centers and ${\bar{y}}_{a.}$ = mean of the Y values in the atypical center. • The number of trial centers varies from 4 to 20, and therefore the ratio N_a/N varies from ¼ to 1/20 (coloured curves).

Fig. 4

Sensitivity and specificity of each centralized statistical monitoring method to detect low values of the deviation of the mean (10%, 20%, 30%), when different number of participants are included in the atypical center and when the ratio $N_a/N$ varies Footnotes to Fig. 4 This figure explores the sensitivity and specificity of the Desmet and Distance methods to conclude that a center may be atypical in the distribution of a continuous variable Y for low deviations of the mean and for different N_a and N_a/N ratios. The results are shown with the number of participants in the atypical center $N_a$ varying from 10 to 300 (horizontal axis) and the overall number of centers of the same size varying from 4 to 20 (coloured curves). For each method, the sub-figure on the left (A) shows the result for (${\bar{y}}_{a.}$- ${\bar{y}}_{na.})$/ ${\bar{y}}_{na.}$ = 10%; the sub-figure on the middle (B) shows the result for (${\bar{y}}_{a.}$- ${\bar{y}}_{na.})$/ ${\bar{y}}_{na.}$ = 20%; and the sub-figure on the right (C) shows the result for (${\bar{y}}_{a.}$- ${\bar{y}}_{na.})$/ ${\bar{y}}_{na.}$ = 30%; With ${\bar{y}}_{na.}$ = mean of the Y values in the non-atypical study centers, and ${\bar{y}}_{a.}$ = mean of Y values in the atypical center (a). In these simulations, the ratio N_a/N varies from 1/4 to 1/20.

Fig. 5

Sensitivity and specificity of each centralized statistical monitoring method to detect an atypical center when other centers are atypical Footnotes to Fig. 5 This figure explores the sensitivity and specificity of the Desmet and Distance methods to conclude that a center may be atypical in the distribution of a continuous variable Y in the scenario where there are other atypical centers. The results are shown with the number of participants in the atypical center $N_a$ varying from 10 to 300 (horizontal axis) and the number of centers of the same size varying from 4 to 20 (coloured curves). All analyses are performed with (${\bar{y}}_{a.}$- ${\bar{y}}_{na.})$/ ${\bar{y}}_{na.}$ = 40%. With ${\bar{y}}_{na.}$ = mean of the Y values in the non-atypical study centers, and ${\bar{y}}_{a.}$ = mean of the Y values in the atypical center (a). For each method, the sub-figure on the left (A) shows the result when the only atypical center is that being analysed; the sub-figure on the middle (B) shows the results when there is another atypical center with the same number of participants; the sub-figure on the right (C) shows the results when there is another atypical center with twice the same number of participants. In these simulations, the ratio N_a/N varies from 1/4 to 1/20.