Eye-Tracking-Based Measurement of Social Visual Engagement Compared With Expert Clinical Diagnosis of Autism

Abstract

Importance: In the US, children with signs of autism often experience more than 1 year of delay before diagnosis and often experience longer delays if they are from racially, ethnically, or economically disadvantaged backgrounds. Most diagnoses are also received without use of standardized diagnostic instruments. To aid in early autism diagnosis, eye-tracking measurement of social visual engagement has shown potential as a performance-based biomarker.

Objective: To evaluate the performance of eye-tracking measurement of social visual engagement (index test) relative to expert clinical diagnosis in young children referred to specialty autism clinics.

Design, setting, and participants: In this study of 16- to 30-month-old children enrolled at 6 US specialty centers from April 2018 through May 2019, staff blind to clinical diagnoses used automated devices to measure eye-tracking-based social visual engagement. Expert clinical diagnoses were made using best practice standardized protocols by specialists blind to index test results. This study was completed in a 1-day protocol for each participant.

Main outcomes and measures: Primary outcome measures were test sensitivity and specificity relative to expert clinical diagnosis. Secondary outcome measures were test correlations with expert clinical assessments of social disability, verbal ability, and nonverbal cognitive ability.

Results: Eye-tracking measurement of social visual engagement was successful in 475 (95.2%) of the 499 enrolled children (mean [SD] age, 24.1 [4.4] months; 38 [8.0%] were Asian; 37 [7.8%], Black; 352 [74.1%], White; 44 [9.3%], other; and 68 [14.3%], Hispanic). By expert clinical diagnosis, 221 children (46.5%) had autism and 254 (53.5%) did not. In all children, measurement of social visual engagement had sensitivity of 71.0% (95% CI, 64.7% to 76.6%) and specificity of 80.7% (95% CI, 75.4% to 85.1%). In the subgroup of 335 children whose autism diagnosis was certain, sensitivity was 78.0% (95% CI, 70.7% to 83.9%) and specificity was 85.4% (95% CI, 79.5% to 89.8%). Eye-tracking test results correlated with expert clinical assessments of individual levels of social disability (r = -0.75 [95% CI, -0.79 to -0.71]), verbal ability (r = 0.65 [95% CI, 0.59 to 0.70]), and nonverbal cognitive ability (r = 0.65 [95% CI, 0.59 to 0.70]).

Conclusions and relevance: In 16- to 30-month-old children referred to specialty clinics, eye-tracking-based measurement of social visual engagement was predictive of autism diagnoses by clinical experts. Further evaluation of this test's role in early diagnosis and assessment of autism in routine specialty clinic practice is warranted.

Trial registration: ClinicalTrials.gov Identifier: NCT03469986.

Figure 1.. Participant Enrollment and Outcomes Comparing Eye-Tracking–Based Quantification of Social Visual Engagement With Expert Clinical Diagnosis of Autism

During a single visit at each clinical testing site, enrolled participants received expert clinical diagnosis using standardized assessments (reference standard diagnosis) as well as eye-tracking–based measurement of social visual engagement (index test). Clinical staff were blind to eye-tracking results and eye-tracking staff were blind to clinical results. For each participant, expert clinicians rated their certainty of diagnosis as in Klaiman et al and McDonnell et al.

Figure 2.. Test Performance of Measurement of Social Visual Engagement (Index Test) vs Reference Standard Diagnosis of Autism

Receiver operating characteristic (ROC) curves for comparison of index test performance relative to reference standard diagnosis in all participants (A), participants for whom expert clinicians rated their diagnoses as certain (B), and participants for whom expert clinicians rated their diagnoses as uncertain (C). Empirical area under the curve (AUC) metrics and 95% CIs are reported on each ROC plot. In panels A, B, and C, the prespecified test positivity threshold is marked with circles. The prespecified test positivity threshold was determined in the efficacy study, fixed, and applied here in the pivotal multisite trial. For comparison, a theoretical optimal threshold determined post hoc by Youden Index is marked with triangles in panels A and B. In panels D, E, and F, cross-tabulations of the achieved eye-tracking index test results vs reference standard diagnoses are given together with corresponding test performance estimates and 95% CIs. Performance results in D, E, and F all correspond to the circles marked in A, B, and C. Negative predictive value (NPV) and positive predictive value (PPV) estimates reported here are calculated based on study sample prevalence. In panels A, B, and C, the orange lines indicate the empirical ROC curve, while blue lines and shaded areas indicate the fitted ROC estimates with 95% CIs, respectively.

Figure 3.. Correlation Between Eye-Tracking–Based Measurement of Social Visual Engagement and Clinician-Administered, Standardized Assessments of Social Disability, Verbal Ability, and Nonverbal Cognitive Ability

A, Correlation between eye-tracking–based indices of social disability vs children’s total scores on the Autism Diagnostic Observation Schedule, Second Edition (ADOS-2, a standardized diagnostic assessment for autism, administered by a trained clinical specialist using a semistructured play session consisting of a set of presses for social and communication interaction intended to elicit behaviors relevant to autism diagnosis). B, Correlation between eye-tracking–based indices of verbal ability vs children’s verbal age equivalent scores as measured by the Mullen Scales of Early Learning (standardized developmental assessment, administered by a trained clinical specialist, to measure a child’s language, motor, perceptual, and cognitive abilities). C, Correlation between eye-tracking–based indices of nonverbal cognitive ability vs children’s nonverbal age equivalent scores as measured by the Mullen Scales of Early Learning. The blue lines in each panel are Deming regression fitted functions, with labels at each end that give directional interpretation for each measurement comparison. In all scatter plots, the blue data markers indicate individual data, and the triangles indicate regression outliers (bivariate outliers identified using Cook distance and difference-in-fits regression diagnostics). Adjusted R² values are adjusted for test-retest reliability of the reference standard (yielding percentage of reference standard nonerror variance explained by the index test). See the eMethods in Supplement 2 (Secondary Endpoint Analyses) for additional information.

Figure 4.. Index Test Predictive Value for Individual Children

A, Flowchart for utilization of index test eye-tracking metrics in specialty centers serving as referral hubs for children with developmental concerns (shown here as having prior probability of autism equal to observed study sample prevalence, 44.8%, for children with certain reference standard; referral center prevalence was expected to be higher than general prevalence due to referrals for concerns). Categorical test results provide post-test probabilities for either autism or nonautism diagnoses. B, Index test positive predictive values (orange line) and negative predictive values (dashed blue-gray line) vary continuously with diagnostic score. Example individual results for 4 children (JG, BC, AC, and KT; drawn from current study sample) are marked with arrows and circles, indicating the test score and the score’s corresponding positive or negative predictive value. C, Summary of test results, categorical post-test predictive value, and individual post-test predictive value for the 4 individual children plotted in panel B. Scores toward either end of the continuum increase probability of autism or nonautism diagnoses, while intermediate scores present intermediate probabilities. All positive and negative predictive values are adjusted for expected prevalence in the context of use. For additional examples of how positive and negative predictive values vary with prevalence, see eFigure 6 in Supplement 3.