Digital restoration of colour cinematic films using imaging spectroscopy and machine learning

Abstract

Digital restoration is a rapidly growing methodology within the field of heritage conservation, especially for early cinematic films which have intrinsically unstable dye colourants that suffer from irreversible colour fading. Although numerous techniques to restore film digitally have emerged recently, complex degradation remains a challenging problem. This paper proposes a novel vector quantization (VQ) algorithm for restoring movie frames based on the acquisition of spectroscopic data with a custom-made push-broom VNIR hyperspectral camera (380-780 nm). The VQ algorithm utilizes what we call a multi-codebook that correlates degraded areas with corresponding non-degraded ones selected from reference frames. The spectral-codebook was compared with a professional commercially available film restoration software (DaVinci Resolve 17) tested both on RGB and on hyperspectral providing better results in terms of colour reconstruction.

Figure 1

Optical RGB images captured by Canon EOS 5D Mark IV camera of the film samples (S1–S6) considered in this work.

Figure 2

Schematic overview of the vector quantization algorithm. (a) k is the spectral wavelength number from the flattened image, N and M represent the total pixel number respectively in the reference image and target image, and (j,b) represents the index of the best representative codeword. (b–e) RGB representation of fade reference (b), good reference (c), target frame (d) to be restored, and the restoration result (e).

Figure 3

The conventional digital restoration strategy. (a) Comparison of RGB images before and after the hand restoration achieved by DaVinci Resolve 17. (b) Illustration of the processing pipelines. (c) Comparison of the histograms before and after restoration with the reference S1.

Figure 4

Originals and restoration results obtained via DaVinci Resolve software and RGB-codebook approach. (a, b) Optical RGB images of references S6 and S1. (c) Optical RGB images of target frames S2, S3, and S5. (d) Manually restored frames D2, D3, and D5 using DaVinci Resolve software. (e) Restoration results R2, R3, and R5 obtained via RGB codebook approach.

Figure 5

Evaluation of the results obtained via RGB images based approaches. (a) Colour difference (ΔE) map of the original target frames S2, S3, and S5 compared to the reference S1. (b) Colour difference (ΔE) map of the restoration results D2, D3 and D5 using DaVinci software. (c) Colour difference (ΔE) map of the restoration results R2, R3 and R5 obtained via RGB triplet codebook approach.

Figure 6

Datacube processing and selected spectra. (a) Greyscale illustration of original datacube before and after initial processing. (b) Comparison of spectrum extracted from the same x and y coordinates on S1 and S6, respectively.

Figure 7

Results and evaluation of simple codebook approach. (a) Digital restoration outcomes R2, R3, and R5 were obtained via a codebook approach visualized in RGB format. (b) Colour difference map of the above restoration results in a simple codebook approach.

Figure 8

Schematic overview of constructing multi-codebook, using spectra hand-selected and combined from multiple samples S2, S3, S5, and S6.

Figure 9

Results and evaluation of the multi-codebook approach. (a) Digital restoration outcomes R2, R3, and R5 were obtained via a multi-codebook approach visualized in RGB format. (b) Colour difference map of the above restoration results in a simple codebook approach.