Automated Page Turner for Musicians

Abstract

An increasing number of musicians are opting to use tablet devices instead of traditional print media for their music sheets since the digital medium offers the benefit of storing a lot of music in a compact space. The limited screen size of the tablet devices makes the music difficult to read and musicians often opt to display part of the music page at a time. With fewer music lines on display, the musician will then have to resort to scrolling through the music to read the entire score. This scrolling is annoying since the musicians will need to remove their hands from the instrument to interact with the tablet, causing a break in the music if this is not done quickly enough, or if the tablet is not sufficiently responsive. In this paper, we describe an alternative page turning system which automates the page turning event of the musician. By actively monitoring the musician's on-screen point of regard, the system retains the musician in the loop and thus, the page turns are attuned to the musician's position on the score. By analysing the way the musician's gaze changes between attention to the score and the instrument as well as the way musicians fixate on different parts of the score, we note that musicians often look away from the score and toward their hands, or elsewhere, when playing the instrument. As a result, the eye regions fall outside the field-of-view of the eye-gaze tracker, giving rise to erratic page-turns. To counteract this problem, we create a gaze prediction model that uses Kalman filtering to predict where the musician would be looking on the score. We evaluate our hands-free page turning system using 15 different piano songs containing different levels of difficulty, various repeats, and which also required playing in different registers on the piano, thus, evaluating the applicability of the page-turner under different conditions. Performance of the page-turner was quantified through the number of correct page turns, the number of delayed page turns, and the number of mistaken page turns. Of the 289 page turns involved in the experiment, 98.3% were successfully executed, 1.7% were delayed, while no mistaken page turns were observed.

Keywords: Kalman filter; eye-gaze tracking; eye-hand span; half-page turns; page-turning.

Figure 1

Page turners described in the literature.

Figure 2

Illustrating a simple reading model. (1) When reading notes within the system, the point-of-regard moves with a horizontal displacement which is a function of the velocity v with which the piece is being played. (2) At the end of the system, the point-of-regard moves not only in the horizontal direction, but also in the vertical direction. Here the horizontal displacement is a function of the width w of the system, while the vertical displacement is a function of the separation s between the two systems.

Figure 3

Illustrating the region of interest on the second system. When the eye-gaze position exceeds a set threshold of ⅕ of the region of interest, the page turn can be effected without distracting the pianist.

Figure 4

Simulation of eye-gaze measurements while reading five systems and the Kalman filtered result. (A) The horizontal component of the eye-gaze movement and (B) the vertical component of the eye-gaze movement.

Figure 5

Simulating the loss of eye-gaze data, typical of brief instances when the pianist makes quick glances at the keyboard. (A) The horizontal component of the eye-gaze movement and (B) the vertical component of the eye-gaze movement.

Figure 6

Simulating the performance of the Kalman filter model under noisy measurement data, typical of deviations in eye-gaze due to micro-saccades and noisy sensors. The noise added has a normal distribution with zero mean and a standard deviation of 50 pixels. (A) The horizontal component of the eye-gaze movement and (B) the vertical component of the eye-gaze movement.

Figure 7

Comparing the performance of the Kalman filter under increasingly noisy data.

Figure 8

Simulating the performance of the Kalman filter model under combined sensor noise and short instances of measurement data loss. (A) The horizontal component of the eye-gaze movement and (B) the vertical component of the eye-gaze movement.

Figure 9

Simulating longer losses in eye-gaze measurements which are typical when the user moves away from the field-of-view of the eye-gaze tracker. The performance of the Kalman filter model (red) can be compared with proposed interpolation of the measurement values to adjust for measurement losses (green). (A) The horizontal component of the eye-gaze movement and (B) the vertical component of the eye-gaze movement.

Figure 10

Comparing the eye-gaze measurements obtained from the eye-gaze tracker and the Kalman filter results using the first two lines of Columbine Dances (Martinu) as an example.

Figure 11

Illustrating the instances where page turns occurred during the execution of Columbine Dances (Martinu). showing (A) the horizontal component and (B) the vertical component of the eye-gaze movement. Regions highlighted in yellow indicate the position of the region of interest of each system. The occurrence of a successful page turn is marked with black circles while delayed page turns are marked with red circles. Page turns occurring within the region of interest do not cause disturbance in the performance of the piece.

Figure 12

Illustrating the performance of the page-turning under conditions of re-starts and skips, showing (A) the horizontal eye-gaze position, (B) the vertical eye-gaze position (C) the measured and Kalman filtered eye-gaze values on the score. (1) The subject starts by reading the music normally but at (2) stops and restarts the performance from the beginning of the system. The Kalman filter eye-gaze tracking model responds in kind and restarts from the beginning of the system too. The current system remains visible for the subject, causing no interruptions in the flow other than those intentionally introduced by the subject. At (3) the subject proceeds to the next system and the Kalman filter model detects this change. The subject plays the first, second, and third bars of this system, but then skips the fourth bar and goes straight to the fifth bar. The Kalman filter treats such a skip as noise in the measurement model and lags behind. However, the page-turning mechanism can sense that the subject has moved to the second system and can update the first system (not shown here). Thus, when the subject completes the second system, the page is refreshed and can proceed with performing the next system which would be displayed on top.