Validation of a Mobile App for Remote Autism Screening in Toddlers

Abstract

Early detection of autism is important for timely access to diagnostic evaluation and early intervention services, which improve children's outcomes. Despite the ability of clinicians to reliably diagnose autism in toddlers, diagnosis is often delayed. SenseToKnow is a mobile autism screening application (app) delivered on a smartphone or tablet that provides an objective and quantitative assessment of early behavioral signs of autism based on computer vision (CV) and machine learning (ML). This study examined the accuracy of SenseToKnow for autism detection when the app was downloaded and administered remotely at home by caregivers using their own devices. The SenseToKnow app was administered by caregivers of 620 toddlers between 16 and 40 months of age, 188 of whom were subsequently diagnosed with autism by expert clinicians. The app displayed strategically designed movies and a bubble-popping game on an iPhone or iPad while recording the child's behavioral responses through the device's front-facing camera and touch/inertial sensors. Recordings of the child's behavior were then automatically analyzed using CV. Multiple behavioral phenotypes were quantified and combined using ML in an algorithm for autism prediction. SenseToKnow demonstrated a high level of diagnostic accuracy with area under the receiver operating characteristic curve of 0.92, sensitivity of 83.0%, specificity of 93.3%, positive predictive value of 84.3%, and negative predictive value of 92.6%. Accuracy of the app for detecting autism was similar when administered on either a caregiver's iPhone or iPad. These results demonstrate that a mobile autism screening app based on CV can be delivered remotely by caregivers at home on their own devices and can provide a high level of accuracy for autism detection. Remote screening for autism potentially lowers barriers to autism screening, which could reduce disparities in early access to services and support and improve children's outcomes.

Figure 1.

Illustration of the SenseToKnow App Workflow. First, movies and the bubble-popping game are displayed (Panel A). Then, raw features are extracted using computer vision (Panel B). Next, the computer vision analysis validity algorithm is implemented (Panel C). Then, behavioral features are computed using the raw behavioral features elicited in response to movies (Panel D). Finally, as illustrated in Panel E, classification modeling is performed including feature age adjustment — fitting a linear regression model on each of the individual 23 features with respect to the participant age and estimating the residuals for each feature, model training, cross-validation, and feature importance to provide an explainable prediction and interpretation using SHAP values. AUROC denotes area under the receiver operating characteristic curve; CVA, computer vision analysis; NPV, negative predictive value; PPV, positive predictive value; ROC, receiver operating characteristic; SHAP, SHapley Additive exPlanation; and XGBoost, Extreme Gradient Boosting.

Figure 2.

Accuracy of Remote Administration of SenseToKnow for Autism Detection Using Home Devices (iPad or iPhone). Panel A shows ROC curves with the respective AUROCs, sensitivities, and specificities based on the whole sample (Model 1), iPad only (Model 2), and iPhone only (Model 3). Panel B shows SHAP values representing behavioral features’ relative importance for Model 1. AUROC denotes area under the receiver operating characteristic curve; ROC, receiver operating characteristic; SHAP, SHapley Additive exPlanation; TNR, true negative rate; and TPR, true positive rate.

Figure 3.

Examples of Individual SHAP Values Plots for Four Different Participants. Panel A shows nonautism participants predicted as nonautism. Panel B shows nonautism participants predicted as autism. Panel C shows autism participants predicted as autism. Panel D shows autism participants predicted as nonautism. The direction and color of the arrows for each behavioral feature indicates whether the behavior is pointing in the autism (orange) or nonautism (blue) direction for the top 12 features. The remaining features were summed up as “other.”