SpecSeg Network for Specular Highlight Detection and Segmentation in Real-World Images

Abstract

Specular highlights detection and removal in images is a fundamental yet non-trivial problem of interest. Most modern techniques proposed are inadequate at dealing with real-world images taken under uncontrolled conditions with the presence of complex textures, multiple objects, and bright colours, resulting in reduced accuracy and false positives. To detect specular pixels in a wide variety of real-world images independent of the number, colour, or type of illuminating source, we propose an efficient Specular Segmentation (SpecSeg) network based on the U-net architecture that is expeditious to train on nominal-sized datasets. The proposed network can detect pixels strongly affected by specular highlights with a high degree of precision, as shown by comparison with the state-of-the-art methods. The technique proposed is trained on publicly available datasets and tested using a large selection of real-world images with highly encouraging results.

Keywords: image segmentation; specular highlights.

Figure A1

Segmentation results of SpecSeg network as compared to manually labelled ground truths in the Whu-Specular dataset [57].

Figure 1

The dichromatic reflection model for inhomogeneous materials.

Figure 2

Real-world examples of specular reflection in images.

Figure 3

Variation in specularity with the variation of polarisation angle (orange areas) in uncontrolled environments. Note that unpolarised light causes specular reflection regardless of polarisation filter angle (blue areas).

Figure 4

SpecSeg configuration based on the U-net architechture.

Figure 5

Segmentation results of SpecSeg network as compared to manually labelled ground truths in the Whu-Specular dataset [57].

Figure 6

Segmentation results of SpecSeg network as compared to manually labelled Ground Truths (GT) in the SIHQ dataset [49].

Figure 7

Segmentation results of SpecSeg on real world images from various sources and self acquired images. Sub-images from left to right (a) generated from [63], (b) image from [64], (c–g) taken by authors, (h,i) video frames from iRoads dataset [65].

Figure 8

Zoomed-in ground truth (GT) and prediction (Pred) views of the marked sections in RGB images from [57]. SpecSeg network is successfully able to detect regions that are (a) on light-coloured objects, (b) small in size, (c) in multiple blocks with cavities inside specular regions, (d) clipped around the edges of the image, (e) detect specularity correctly from images on a white background.

Figure 9

Specular segmentation results on outdoor images acquired on a sunny day and under clear sky conditions. Specular reflections detected under extreme conditions are plausible and significantly better than any other state-of-the-art technique. Note that brightly lit regions such as the sky or water puddles are not detected as specular regions.

Figure 10

(a) A summary of the metrics over the entire dataset. (b) Training and validation losses after 200 epochs. The training was stopped after 200 epochs to avoid overfitting by the network.