Matrix-R Theory: A Simple Generic Method to Improve RGB-Guided Spectral Recovery Algorithms

Abstract

RGB-guided spectral recovery algorithms include both spectral reconstruction (SR) methods that map image RGBs to spectra and pan-sharpening (PS) methods, where an RGB image is used to guide the upsampling of a low-resolution spectral image. In this paper, we exploit Matrix-R theory in developing a post-processing algorithm that, when applied to the outputs of any and all spectral recovery algorithms, almost always improves their spectral recovery accuracy (and never makes it worse). In Matrix-R theory, any spectrum can be decomposed into a component-called the fundamental metamer-in the space spanned by the spectral sensitivities and a second component-the metameric black-that is orthogonal to this subspace. In our post-processing algorithm, we substitute the correct fundamental metamer, which we calculate directly from the RGB image, for the estimated (and generally incorrect) fundamental metamer that is returned by a spectral recovery algorithm. Significantly, we prove that substituting the correct fundamental metamer always reduces the recovery error. Further, if the spectra in a target application are known to be well described by a linear model of low dimension, then our Matrix-R post-processing algorithm can also exploit this additional physical constraint. In experiments, we demonstrate that our Matrix-R post-processing improves the performance of a variety of spectral reconstruction and pan-sharpening algorithms.

Keywords: Matrix-R; pan-sharpening; spectral image fusion; spectral reconstruction; spectral super-resolution.

Figure 1

We demonstrate how the Matrix-R post-processing method works. First, either RGB images alone (in the spectral reconstruction case) or RGB images combined with low-resolution hyperspectral images (in pan-sharpening) are fed into existing spectral recovery algorithms. The output images are then enhanced by the Matrix-R post-processing algorithm. These refined images consistently achieve greater accuracy and are closer to the ground-truth compared to the initial estimates.

Figure 2

A spectrum of light measured by a camera system $\mathbf{Q}$ results in an RGB. In SR, an estimated spectrum is returned directly from analyzing the RGB image. In PS, low-res hyperspectral image can also guide spectral estimation. We group PS and SR algorithms in the single “spectral recovery algorithm” box. The estimated spectrum is decomposed into estimated metameric black and fundamental metamer components. Combining the correct fundamental metamer, calculated directly from the RGB [31], with this estimated metameric black returns a refined estimate of the spectrum. Refining a spectral estimate in this way is called “Matrix-R post-processing”. See the text for a description of the mathematical notation, while this figure serves as a glossary of important notations in this paper.

Figure 3

An illustration of the RGB-based hyperspectral pan-sharpening (PS; green box) and spectral reconstruction (SR; red box). The images are generated from the ICVL hyperspectral image database [40]. Left: The demonstration of the low-resolution hyperspectral image. Center: The high-resolution RGB image. Right: The target high-resolution hyperspectral image.

Figure 4

The Original, Matrix-R, and Matrix-R with a lower-dimensional spectral assumption results in RMSE error heat maps for the tested hyperspectral pan-sharpening algorithms.

Figure 5

The original upsampling-only, post-processing by Matrix-R, and Matrix-R with a lower-dimensional spectral assumption results in RMSE error heat maps for multispectral pan-sharpening.

Figure 6

The correlation plots of spectral and color improvements ($\Delta$RMSE (Spectral) and $\Delta$RMSE (RGB), respectively) for spectral reconstruction (SR; left plot) and hyperspectral pan-sharpening (PS; right plot). Note that the RMSEs are scaled by $\times {10}^3$ to be consistent with the numbers in result tables.