Validation of laboratory-developed molecular assays for infectious diseases

Abstract

Molecular technology has changed the way that clinical laboratories diagnose and manage many infectious diseases. Excellent sensitivity, specificity, and speed have made molecular assays an attractive alternative to culture or enzyme immunoassay methods. Many molecular assays are commercially available and FDA approved. Others, especially those that test for less common analytes, are often laboratory developed. Laboratories also often modify FDA-approved assays to include different extraction systems or additional specimen types. The Clinical Laboratory Improvement Amendments (CLIA) federal regulatory standards require clinical laboratories to establish and document their own performance specifications for laboratory-developed tests to ensure accurate and precise results prior to implementation of the test. The performance characteristics that must be established include accuracy, precision, reportable range, reference interval, analytical sensitivity, and analytical specificity. Clinical laboratories are challenged to understand the requirements and determine the types of experiments and analyses necessary to meet the requirements. A variety of protocols and guidelines are available in various texts and documents. Many of the guidelines are general and more appropriate for assays in chemistry sections of the laboratory but are applied in principle to molecular assays. This review presents information that laboratories may consider in their efforts to meet regulatory requirements.

FIG. 1.

Plot of results from a linearity experiment to determine reportable range. Seven concentrations of analyte prepared by dilution of a high-concentration standard were tested in triplicate. Assigned values, (converted to log₁₀) were plotted on the x axis versus measured values (converted to log₁₀) on the y axis using Microsoft Excel. Linear regression analysis gave the equation y = 0.9613x + 0.1286 (r² = 0.9932). A second-order polynomial trendline gave the equation y = −0.028x²+1.1937x − 0.2667 (r² = 0.9954). A third-order polynomial trendline gave the equation y = 0.009x³ + 0.1388x² + 1.5994x − 0.6948 (r² = 0.9958). The second- and third-order polynomials are not significantly better (P > 0.05) than the linear equation, which indicates that the linear equation is the best fit for the data. The fitted regression line shows the slope to be significantly different from zero and the intercept to be not significantly different from zero. The regression coefficient of 0.9973 verifies the linearity throughout the range tested. The reportable range in this example translates to 30 copies/ml (LLOQ) through 3,000,000 copies/ml (ULOQ). Because of imprecision at the low end, more replicates in a precision experiment may need to be tested to adequately determine the LLOQ before accepting the reportable range.

FIG. 2.

Bland-Altman bias plots and xy scatter plots for two different quantitative real-time PCR assays for which a new extraction method was being evaluated. (Data courtesy of Charles Hill, Emory University, Atlanta, GA; used with permission.) (A) Seventy-two patient specimens were tested using both a new (test) and old (comparison) extraction method for a cytomegalovirus viral load assay. Forty-three specimens that had numerical results by both methods are plotted. Visual inspection of the xy scatter plot showed no obvious outliers. Linear regression analysis gives the equation y = 0.9834x + 0.0576 with a correlation coefficient (r²) of 0.9449 to describe the line that fits the data most closely. The correlation coefficient of 0.9449 indicates good correlation. The slope of 0.9834 is very close to 1.00, indicating no proportional bias, and the y intercept at 0.0576 is very close to the origin (0.00), indicating no constant systematic bias. (B) Bland-Altman bias plot of the data in panel A. The mean bias was determined to be −0.01 log unit, which indicates no systematic bias. The plot shows that bias is greater at higher viral loads, but the bias is in both directions and within acceptable limits. Both extraction methods were determined to be equivalent, and the new extraction method could be used without need to rebaseline patients. (C) In this comparison study, 46 patient specimens positive for BK virus in a real-time quantitative PCR assay were tested using both a new extraction method and an old extraction method. No obvious outliers are seen in the xy scatter plot. Linear regression analysis gives the equation y = 0.9698x + 0.3398 with a correlation coefficient of 0.9543. The correlation coefficient of 0.9543 indicates good correlation, and the slope of 0.9698 is very close to 1.00, indicating no proportional bias. However, the y intercept at 0.3398 is significantly away from the origin (0.00), indicating the presence of some constant systematic bias. (D) Bland-Altman bias plot showing that the mean bias was determined to be 0.41 log unit, indicating a systematic bias of about 2.5-fold. The bias is not considered to be statistically significant since the 95% confidence interval contains zero (no difference), but it needs to be decided if the bias is clinically significant before the new extraction system is put into use for clinical testing.

FIG. 3.

Continuous monitoring of controls using Levey-Jennings plots. The low-positive control presented in Table 6 was monitored on each day of use using a Levey-Jennings plot. The plot shows the target value (threshold cycle value [C_T] = 37.90, solid red line) as well as expected limits (hatched lines) (two standard deviations = 34.7 to 41.1; three standard deviations = 33.1 to 42.7). All control results are within two standard deviations of the mean target value and are therefore in range using Westgard's multirule analysis.