Estimation in medical imaging without a gold standard

Abstract

Rationale and objectives: In medical imaging, physicians often estimate a parameter of interest (eg, cardiac ejection fraction) for a patient to assist in establishing a diagnosis. Many different estimation methods may exist, but rarely can one be considered a gold standard. Therefore, evaluation and comparison of different estimation methods are difficult. The purpose of this study was to examine a method of evaluating different estimation methods without use of a gold standard.

Materials and methods: This method is equivalent to fitting regression lines without the x axis. To use this method, multiple estimates of the clinical parameter of interest for each patient of a given population were needed. The authors assumed the statistical distribution for the true values of the clinical parameter of interest was a member of a given family of parameterized distributions. Furthermore, they assumed a statistical model relating the clinical parameter to the estimates of its value. Using these assumptions and observed data, they estimated the model parameters and the parameters characterizing the distribution of the clinical parameter.

Results: The authors applied the method to simulated cardiac ejection fraction data with varying numbers of patients, numbers of modalities, and levels of noise. They also tested the method on both linear and nonlinear models and characterized the performance of this method compared to that of conventional regression analysis by using x-axis information. Results indicate that the method follows trends similar to that of conventional regression analysis as patients and noise vary, although conventional regression analysis outperforms the method presented because it uses the gold standard which the authors assume is unavailable.

Conclusion: The method accurately estimates model parameters. These estimates can be used to rank the systems for a given estimation task.

Figure 1

A graphic, two-modality example of the method studied where a shows the results for M = 1 and b shows the results for M = 2. The dotted lines represent ±σ̂_m. The slope, intercept, and noise terms were estimated by using RWT. Although the x coordinates are plotted, they were not used in estimating the linear model parameters.

Figure 2

A comparison of the true gold-standard density, pr(Θ), and the parameterized density, $\widehat{\mathit{\text{pr}}}(\mathbf{\Theta }|\widehat{\overrightarrow{r}})$. The shape of the density, as characterized by r⃗, was determined with RWT but without previous information. The gold-standard density shown here is a truncated normal density, whereas the parameterized density used in the likelihood expression is a beta-density function. In a sense, this illustrates a beta density imitating a given truncated normal density. Note that the parameter of interest is limited to a finite domain.

Figure 3

(a) The $\overline{\text{RMSE}}$ for three different modalities versus the number of patients. As the number of patients increases, RMSE_m converges to σ_m/a_m by Equations (1) and (5). (b) A comparison between RWT and linear-regression analysis with a gold standard. Note that the RMSE is also averaged over the three modalities. As expected, conventional regression analysis has lower RMSE, but the performances of the two methods converge as the number of patients increases. For these experiments, a⃗ = [0.6,0.7,0.8], b⃗ = [−0.1,0.0,0.1], σ⃗ = [0.05,0.03,0.08], and the error bars represent the standard error calculated over 50 independent experiments.

Figure 4

The $\overline{\text{RMSE}}$ (averaged across simulations and modalities) versus the number of modalities used in a RWT experiment. A sharp decline in $\overline{\text{RMSE}}$ is seen from one to two modalities, followed by a slow decline. One might expect this, especially because RWT cannot work properly with only one modality. The performance of conventional regression analysis is independent of the number of modalities. The same model parameters were used for all modalities in all experiments (a_m = 1, b_m = 0.1, σ_m = 0.05, P = 100).

Figure 5

(a) The $\overline{\text{RMSE}}$ for three different modalities versus variance of the noise σ_m. The $\overline{\text{RMSE}}$ increases in accordance with 1/a_m by Equations (1) and (5). (b) A comparison between RWT and linear-regression analysis with a gold standard. Note that the RMSE is also averaged over the three modalities. The $\overline{\text{RMSE}}$ does not converge to zero for RWT as σ_m tends to zero. The parallel nature of the two graphs indicates that the comparative performance of RWT is independent of σ_m. For these experiments, a⃗ = [0.6,0.7,0.8], b⃗ = [−0.1,0.0,0.1], P = 100, and the error bars represent the standard error calculated over 50 independent experiments.

Figure 6

An application of RWT with a quadratic model. (a) For modality 1, a strong, nonlinear relationship with the gold standard and a relatively large variance were discovered qualitatively. (b) Modality 2 was slightly nonlinear with a small variance, whereas (c) modality 3 was linear with a large variance. Both were fit well by the quadratic RWT.