A digital mask to safeguard patient privacy

Abstract

The storage of facial images in medical records poses privacy risks due to the sensitive nature of the personal biometric information that can be extracted from such images. To minimize these risks, we developed a new technology, called the digital mask (DM), which is based on three-dimensional reconstruction and deep-learning algorithms to irreversibly erase identifiable features, while retaining disease-relevant features needed for diagnosis. In a prospective clinical study to evaluate the technology for diagnosis of ocular conditions, we found very high diagnostic consistency between the use of original and reconstructed facial videos (κ ≥ 0.845 for strabismus, ptosis and nystagmus, and κ = 0.801 for thyroid-associated orbitopathy) and comparable diagnostic accuracy (P ≥ 0.131 for all ocular conditions tested) was observed. Identity removal validation using multiple-choice questions showed that compared to image cropping, the DM could much more effectively remove identity attributes from facial images. We further confirmed the ability of the DM to evade recognition systems using artificial intelligence-powered re-identification algorithms. Moreover, use of the DM increased the willingness of patients with ocular conditions to provide their facial images as health information during medical treatment. These results indicate the potential of the DM algorithm to protect the privacy of patients' facial images in an era of rapid adoption of digital health technologies.

Fig. 1. Development of the DM system.

Our approach uses RGB images as input and outputs 3D reconstructed meshes. For a particular frame, the algorithm first extracts 2D face landmarks from the RGB image and fits a set of face model weights for 3D face reconstruction. Then, the algorithm extracts 2D eyelid landmarks and 2D semantic lines and fits eyelid model weights for 3D eyelid reconstruction. Finally, the algorithm extracts 2D iris landmarks and solves eyeball rotation for 3D eyeball reconstruction.

Fig. 2. Quantitative evaluation of the digital mask.

a, Schematic indicating how the Euclidean error was calculated. For both eyeball and eyelid reconstruction, we project the 3D points to 2D image space (yellow) and calculate the Euclidean pixel distance between them and landmarks in the original video (red). The Euclidean error was normalized by the pixel distance between the center of two eyes to exclude the influence of face size. b, The normalized pixel error of different ocular diseases for eyeball (left) and eyelid (right) reconstruction. Results were expressed as mean ± s.d. c, Heat map of the normalized pixel error for four cases. The frame number of each video (30 f.p.s.) is plotted on the y axis. Landmark numbers (238 in total, 38 landmarks for eyeballs and 200 landmarks for eyelids) are plotted in order on the x axis. Normalized pixel error (0–5%) is indicated by the collar bar at the right. The closer the color is to blue, the more accurate the performance is. OD, right eye; OS, left eye; f.p.s., frames per second.

Fig. 3. Clinical signs of the ocular diseases studied.

Main clinical signs (top) for diagnosis of each ocular disease studied are shown using schematic diagram, the original facial image, the image of the DM and the overlap between the original facial image and the DM. See Supplementary Video for details. More diverse secondary clinical signs of the four diseases are shown (bottom).

Fig. 4. Clinical validation of the DM.

a, Workflow of relevant diagnostic comparisons using the original videos and the corresponding DM-reconstructed videos. Participants were recruited from four outpatient departments after having been diagnosed by a specialist as having TAO, ptosis, strabismus, nystagmus or none of these. Once the participants were enrolled in the study, facial videos of appropriate ocular examinations were taken. Each video was independently used by three ophthalmologists from each of the four departments for making a diagnosis. A dichotomous diagnosis of abnormal or not was made for both the left eye and the right eye. Both the original video and DM video from the same participant were used by the same ophthalmologist for diagnosis, performed in a blinded fashion using the participant number (ID-1 to ID-n; TAO, n = 102; strabismus, n = 100; ptosis, n = 111; and nystagmus, n = 92). b, Line plots indicating diagnostic consistency for the indicated ocular diseases (Cohen’s κ ≥ 0.81 indicates perfect consistent). c, Left, workflow for identity removal validation in which the identity removal abilities of the DM and those of image cropping were compared. Respondents were given six options, including five facial images and an ‘other’ option. From these options, the respondents were asked to choose the original image corresponding to the DM-reconstructed image or cropped image. Red indicates an incorrect answer; green indicates a correct answer. Accuracy of identity removal validation (right). Results were expressed as mean ± s.d. Each scatter-point represents the score of one set calculated from one respondent, with ten questions per set and a total score of 100. For each disease, 20 sets of questions (10 of DM and 10 of cropped) were taken.

Fig. 5. Empirical investigation of the willingness of patients to share personal health information.

a, Schematic of the hypothesis. Patients’ trust in physicians and medical platforms was hypothesized to be positively affected by perceived benefits, such as health support of digital health information (H1) and negatively affected by perceived concerns, such as privacy concerns (H2). The DM was hypothesized to have a positive impact on such trust (H3) and further improved patients’ willingness to share information (H4). b, Questionnaire results. In the questionnaire, 16 questions were designed with respect to the five hypotheses; the responses were further measured using five-point scales ranging from ‘strongly disagree’ to ‘strongly agree’. The percentages of the responses to each item are shown (right).

Fig. 6. Validation of the DM using AI-powered re-identification algorithms.

a, Study workflow. The re-identification algorithms were used to find the ID of the patient from a database of 405 patients when given the original image, a cropped image or the DM-reconstructed image of a patient as a query image. b, Performance of the three re-identification algorithms tested, as assessed by TAR@FAR = 0.1, TAR@FAR = 0.01, Rank-1 (left) and ROC curves (right). The re-identification algorithms were trained on the CASIA-Webface dataset. TAR@FAR = X indicates the TAR when the FAR equals X. Rank-1 is the probability that the similarity score of the same identity ranks first among all the identities. Lower values of TAR@FAR = 0.1, TAR@FAR = 0.01 and Rank-1 indicate weaker performance of the re-identification algorithm and better performance of the privacy protection technology. TAR = TP/(TP + FN); FAR = FP/(FP + TN). TP, true positive; FP, false positive; TN, true negative; FN, false negative.