CultReal-A Rapid Development Platform for AR Cultural Spaces, with Fused Localization

Abstract

Virtual and augmented reality technologies have known an impressive market evolution due to their potential to provide immersive experiences. However, they still have significant difficulties to enable fully fledged, consumer-ready applications that can handle complex tasks such as multi-user collaboration or time-persistent experiences. In this context, CultReal is a rapid creation and deployment platform for augmented reality (AR), aiming to revitalize cultural spaces. The platform's content management system stores a representation of the environment, together with a database of multimedia objects that can be associated with a location. The localization component fuses data from beacons and from video cameras, providing an accurate estimation of the position and orientation of the visitor's smartphone. A mobile application running the localization component displays the augmented content, which is seamlessly integrated with the real world. The paper focuses on the series of steps required to compute the position and orientation of the user's mobile device, providing a comprehensive evaluation with both virtual and real data. Pilot implementations of the system are also described in the paper, revealing the potential of the platform to enable rapid deployment in new cultural spaces. Offering these functionalities, CultReal will allow for the fast development of AR solutions in any location.

Keywords: augmented reality; beacons; computer vision; localization.

Figure 1

Deployment diagram of the CultReal System.

Figure 2

Workflows of the CultReal applications: location manager application (top) and visitor application (bottom).

Figure 3

Workflow of the location manager application displaying the guidelines and several prominent planes: the floor and several planar surfaces of a bed (a), displaying detected feature points (b), displaying the user-defined coordinate system (c), and displaying a message that a photo has been taken (d).

Figure 4

iB004N beacons used for beacon-based localization.

Figure 5

Estimation of transformation matrices between images I_b and I_a.

Figure 6

Correspondences detected with matching algorithms: results of the FLANN algorithm (top) and of the BFMatcher (bottom).

Figure 7

Evaluation of the beacon-based localization: static scenario (top) and dynamic scenario (bottom).

Figure 8

Piloting the beacon-based localization: a screenshot from an external video, filming the real scene and the visitor’s phone (left) and a screen capture of the phone, displaying a video with general information about that room, in the center, superimposed on the video stream from the phone’s camera (right).

Figure 9

Piloting the augmented images functionality: a screen capture of the visitor’s phone, displaying the video stream acquired with the camera before identifying the book cover (left), a screenshot from an external video filming the real scene, and the visitor’s phone (middle), and a screen capture of the phone displaying the virtual image with the book’s table of contents superimposed on the video stream from the phone’s camera (right).

Figure 10

Piloting the augmented images functionality: a screen capture of the visitor’s phone, displaying the video stream acquired with the camera, before identifying the logo of the Gong Theatre (left), and a screen capture of the phone displaying the video promoting a cultural event superimposed on the video stream from the phone’s camera (right).

Figure 11

Piloting the computer vision-based localization: screen captures of the visitor’s phone, displaying two virtual images (left), a 3D model of a plane (middle), 3D text placed on the floor (right).

Figure 12

Piloting the computer vision-based localization with multiple users in a common augmented space: a screen capture of the first phone, displaying a 3D model of a bamboo (left), a screen capture of the second phone, displaying the same 3D model (middle), a screenshot from an external video filming the real scene and the two phones of the visitors (right).