An augmented-reality edge enhancement application for Google Glass

Abstract

Purpose: Google Glass provides a platform that can be easily extended to include a vision enhancement tool. We have implemented an augmented vision system on Glass, which overlays enhanced edge information over the wearer's real-world view, to provide contrast-improved central vision to the Glass wearers. The enhanced central vision can be naturally integrated with scanning.

Methods: Google Glass' camera lens distortions were corrected by using an image warping. Because the camera and virtual display are horizontally separated by 16 mm, and the camera aiming and virtual display projection angle are off by 10°, the warped camera image had to go through a series of three-dimensional transformations to minimize parallax errors before the final projection to the Glass' see-through virtual display. All image processes were implemented to achieve near real-time performance. The impacts of the contrast enhancements were measured for three normal-vision subjects, with and without a diffuser film to simulate vision loss.

Results: For all three subjects, significantly improved contrast sensitivity was achieved when the subjects used the edge enhancements with a diffuser film. The performance boost is limited by the Glass camera's performance. The authors assume that this accounts for why performance improvements were observed only with the diffuser filter condition (simulating low vision).

Conclusions: Improvements were measured with simulated visual impairments. With the benefit of see-through augmented reality edge enhancement, natural visual scanning process is possible and suggests that the device may provide better visual function in a cosmetically and ergonomically attractive format for patients with macular degeneration.

Figure 1

(A) Schematic (top-down view) of Google Glass hardware configuration: 2D translation of the displayed image is required depending on distance to the object of interest (parallax). (B) Schematic (side view) of Google Glass hardware configuration with the eye in primary position of gaze and the head upright: 10° angular compensation between the virtual display orientation and the camera aiming direction is needed for visual alignment. (C) The Google Glass camera lens’ distortion presented by photographing a grid (top), and the distortion corrected grid (bottom). Dashed line rectangle is the image portion to be displayed on the virtual display. Note that only a small portion of the captured image needs to be rendered. A color version of this figure is available online at www.optvissci.com.

Figure 2

Block diagram of the image processing flow. Lens distortion correction is applied first, then image clipping, which is selected based on disparity setting, is applied next. Additional scaling and rotation is applied to compensate vertical angular difference between camera and the display. Edge extraction and enhancement is only applied to the clipped image area (defined by series of vertex operations).

Figure 3

Illustrations of edge enhanced scene with (A) a positive Laplacian edge detection method and (B) a negative Laplacian edge detection. Note that with positive Laplacian method, the location of edge is whitened, but with negative Laplacian method, surrounding of edges are whitened. (C–E) Illustrations of the effect of selective (positive) contrast enhancement of a Pelli-Robson virtual contrast sensitivity chart (F), calculated with high, medium, and low level contrast ranges, respectively. Note that only the edge information is shown in (C–E), while the overlaid view of the edges and see-through views are shown in (A) and (B). Different scaling levels applied to the images in the top and bottom rows cause the apparent thickness difference of enhanced edges. A color version of this figure is available online at www.optvissci.com.

Figure 4

Alignments of enhanced edges at different distances photographed through Google Glass. Indoor scenes (A) without AR edge enhancement, and (B) with AR edge enhancement (desk lamp is about 10ft away) showing slight misalignment. Outdoor scenes on a cloudy Boston day (C) with a person standing at 50ft away, and (D) two people approaching from about 20ft and 40ft away. Note that alignment of the eye (in this case, a camera to take pictures) and the virtual display position are not exactly aligned (can be observed by the position of the virtual display within the frame, especially in (B), but the AR alignment is robust enough to provide ‘on top’ edges. A color version of this figure is available online at www.optvissci.com.

Figure 5

(A) Photo of the Pelli-Robson contrast sensitivity chart used. (B) The same chart photographed through the Glass. Strong edges are enhanced when the high contrast edges are targeted. (C) When low contrast edges are targeted, camera and other system noise are also enhanced, but still the structured edge formation can be observed. Compare that to the ideal conditions calculated in Fig. 2, which is free of camera noise. Note that the edge free portion of the display is not completely dark due to leakage of the LCD display backlight. A color version of this figure is available online at www.optvissci.com.

Figure 6

Illustration of the utility of the image enhancement app on the Glass by a patient with AMD. (A) A retinal image of a patient (presented upside down to correspond to the visual field orientation of the other figures) with superimposed perimetry polar grid centered at the PRL. The location of the PRL, as measured by the Nidek MPI, and the calculated bivariate ellipse including it are shown with a frame, the size of the field of the Glass, centered on it (Virtual Display). A (doted orange) circle of 36° diameter centered at the PRL, illustrates the retinal span of a 12° field-of-view of a 3.0× bioptic telescope, which provides a distal field-of-view similar to that of the Glass display. (B) A room scene taken with the Glass, illustrating a series of possible objects to be examined by the patient using the image enhanced Glass. (C–F) Sequential scene scanning with edge enhancement at the selected gazed locations. The space variant imaging (eccentric-wise blurring) and HDR conversions are applied to a scene image to illustrate perceptual view of the patient. Note that unlike the bioptic telescope, all surrounding parts of the scene remain visible throughout the scanning process. In these images the illustrated blur is applied over the enhancement to illustrate the improved visibility provided by the enhanced edges. A color version of this figure is available online at www.optvissci.com.