Hearables: Multimodal physiological in-ear sensing

Abstract

Future health systems require the means to assess and track the neural and physiological function of a user over long periods of time, and in the community. Human body responses are manifested through multiple, interacting modalities - the mechanical, electrical and chemical; yet, current physiological monitors (e.g. actigraphy, heart rate) largely lack in cross-modal ability, are inconvenient and/or stigmatizing. We address these challenges through an inconspicuous earpiece, which benefits from the relatively stable position of the ear canal with respect to vital organs. Equipped with miniature multimodal sensors, it robustly measures the brain, cardiac and respiratory functions. Comprehensive experiments validate each modality within the proposed earpiece, while its potential in wearable health monitoring is illustrated through case studies spanning these three functions. We further demonstrate how combining data from multiple sensors within such an integrated wearable device improves both the accuracy of measurements and the ability to deal with artifacts in real-world scenarios.

Figure 1

Construction of the multimodal in-ear sensing device. (a) Detailed structure of the device, showing the placement of the microphone and the electrode on the substrate. (b) Construction of the multimodal sensor underneath one of the cloth electrodes. (c) Completed earpiece with electrodes and inward-facing microphone visible. (d) Placement of the earpiece in the user’s ear.

Figure 2

Mechanical tests comparing the viscoelastic foam substrate and silicone. (a) Stress-Strain curve for viscoelastic foam cylinders of varying density. (b) EEG signal captured with the foam-based earplug from the ear canal of a person with strong blood vessel pulsation, observe no pulsation in the recorded EEG. (c) Stress-Strain curve for silicone cylinders of varying hardness. (d) EEG signal captured using the silicone-based earplug from the ear canal of the same person with strong blood vessel pulsation, observe strong contamination of the EEG with pulsatile artifacts, indicated by red crosses.

Figure 3

Simulated electric potentials in the head at time 0 s, generated by dipoles which oscillated at 39 Hz, with positive potentials shown in red and negative potentials in blue. (a) Potentials on the whole scalp; (b) Potentials in the coronal plane; (c) Potentials in the right ear region seen from inside the head (note that (a–c) have different scales); (d) Waveforms of potential differences between electrode pairs on the scalp and inside the ear canal. The T8 and Cz positions are standard on-scalp EEG electrodes and further abbreviations are: RE D: Right ear canal, downward direction; RE U: Right ear canal, upward direction; RE H: Right ear, root of the helix. The positions of the ear electrodes are marked with black circles in panel (c).

Figure 4

Acquisition of EEG and respiration from inside the ear canal. (a) Stability of the electrode-skin interface over the course of 8 hours, for 5 subjects. (b) ASSR response measured from the ear, M1 and Cz scalp locations. (c) SSVEP response measured from the ear, M1 and Cz locations. (d) Visual evoked potential measured with the earpiece. (e) Alpha rhythm recorded from the ear electrodes, 30 s into the trial subject closed their eyes. (f,g) (Top and Middle) Spectrograms of acoustic signals measured by the two microphones (inward- and wall-facing) integrated within the earpiece when the subject was asked to breath at 16 and 28 breaths per minute, respectively. (f,g) (Bottom) Breathing signals extracted from the associated spectrograms. (h) EEG signals from scalp and ear canal during the N2 stage of sleep. (i) EEG signals from scalp and ear canal when the subject was awake, but with eyes closed.

Figure 5

Denoising of ear-EEG from mechanical jaw clench artifacts. (a) Best-case denoising scenario for which the artifact measurement with the embedded microphone was accurate. (b) Worst-case accuracy of artifact measurement with the embedded microphone, note that the artifact was nevertheless reduced in the Clean EEG trace. (Top panels) Raw EEG corrupted by a strong artifact. (Middle panels) Output of the mechanical sensor within the MMS. (Bottom panels) Denoised EEG using the mechanical MMS signal as reference.

Figure 6

Recording of speech from the multimodal earpiece, illustrated through a 10 s spectrogram. (Top) Speech signal recorded using a microphone placed on the tip of the earpiece looking towards the eardrum. (Bottom) Speech signal from an external microphone placed on a subject’s chest. The Y-axis for both spectrograms spans the 300–3400 Hz band, the so-called voice frequency band.

Figure 7

Cardiac activity captured from inside the ear canal using the multimodal sensor embedded within the earpiece. (Top) ECG signal from the arms. (Middle) PPG signal from the finger. (Bottom) Integral of the mechanical signal produced by the microphone within the multimodal sensor.