Convex optimization algorithms in medical image reconstruction-in the age of AI

Abstract

The past decade has seen the rapid growth of model based image reconstruction (MBIR) algorithms, which are often applications or adaptations of convex optimization algorithms from the optimization community. We review some state-of-the-art algorithms that have enjoyed wide popularity in medical image reconstruction, emphasize known connections between different algorithms, and discuss practical issues such as computation and memory cost. More recently, deep learning (DL) has forayed into medical imaging, where the latest development tries to exploit the synergy between DL and MBIR to elevate the MBIR's performance. We present existing approaches and emerging trends in DL-enhanced MBIR methods, with particular attention to the underlying role of convexity and convex algorithms on network architecture. We also discuss how convexity can be employed to improve the generalizability and representation power of DL networks in general.

Keywords: artificial intelligence; convex optimization; deep learning (DL); first order methods; inverse problems; machine learning (ML); model based image reconstruction.

Figure 5.

(a) When $\tau ⩽1$ and $\tilde{x}⩽\tau$, the objective $\varphi$ continuously increases as a function of $x$. There is a global minimizer at $x=0$. (b) When $\tau ⩽1$ and $\tilde{x}>\tau$ there is a unique intersection point (the filled red marker) between the two gradient lines $f^{\prime }\left(x\right)$ and $-q^{\prime }\left(x\right)$.

Figure 6.

(a) If $\tau >1$ and $\tilde{x}>\tau$, there is a unique intersection between $f^{\prime }\left(x\right)$ (blue curve) and $-q^{\prime }\left(x\right)$ (green line), indicated by the filled red marker. (b) If $\tau >1$ and $\tilde{x}<{\tilde{x}}_t$, there is no intersection between the $f^{\prime }\left(x\right)$ and $-q^{\prime }\left(x\right)$. The solution to (8.19) is $x=0$. Here $x_t=\sqrt{\tau }-1,{\tilde{x}}_t=2\sqrt{\tau }-1$.

Figure 7.

Two cases when ${\tilde{x}}_t<\tilde{x}⩽\tau$. The intersections between the blue curve $f^{\prime }\left(x\right)$ and the green line $-q^{\prime }\left(x\right)$ are marked by the open and the filled red markers. The former indicates a local maximum, the latter indicates a local minimum. There is another local minimum at $x=0$. (a) When area $\mathrm{A}>$ area $\mathrm{B}$, the global minimizer of (8.19) is at $x_{\ast }=0$. (b) When area $\mathrm{A}<$ area $\mathrm{B}$, the global minimizer is at $x_{\ast }=x_{\text{. }}$, the second (larger) intersection point. The critical point $\tilde{x}={\tilde{x}}_c$ separating the two cases is when area $\mathrm{A}=$ area $\mathrm{B}$.

Figure 8.

The thresholding solution given by (8.20). Here we append by symmetry the solution for $\tilde{x}<0$ as well. (a) If $\tau ⩽1$, the objective (8.19) is convex, the solution $x_{\ast }$ is a continuous function of $\tilde{x}$. (b) If $\tau >1$, the objective (8.19) is nonconvex, the solution $x_{\ast }$ has a jump at $\tilde{x}={\tilde{x}}_c$, given by (8.22).

Figure 1.

(a) An iterative algorithm where the data consistency (DC) term and the regularizer (Reg) connects in serial. The loop sign (green) indicates the recurrent nature of the iterations. (b) Variational network (VN) unrolls an iterative algorithm and replaces the regularizers by CNNs. The multiple CNNs can share weights $\left({\theta }_k=\theta \right.$, for all $k$) or have different weights, although the former adheres more to the recurrent nature of an iterative algorithm. The serial connection in (a) can model algorithms such as proximal gradient or alternating update schemes (Liang et al 2019). Parallel connection is also possible, e.g., as in gradient descent, which gives rise to different VN architectures (Liang et al 2019).

Figure 2.

Using the CNN to parametrize the unknown image $x$ as proposed in (Gong et al 2018a). The output of the CNN, which is pretrained to perform image denoising, is the reconstructed image. Image reconstruction is formulated to minimize the loss function with respect to $z$ or $\theta$.

Figure 3.

The hyperparameter learning framework proposed in (Xu and Noo 2021). The CNN, parametrized by $\theta$, generates patientspecific and spatially variant hyperparameter $\eta$ needed for optimization-based image reconstruction. End-to-end learning requires backpropagating the gradient from the loss to the CNN parameter $\theta$. During testing/inference, the image reconstruction module can run outside of a DL library.

Figure 4.

A convex optimization layer (COL) outputs the solution of a convex optimization problem $f(x;\theta )$, where $\theta$ lumps both the input $y$ and nuisance parameters $\eta$. A COL can be embedded as a component in a larger network. End-to-end training of such networks requires differentiation through argmin.