Computational approaches to understanding interaction and development

Abstract

Audio-visual recording and location tracking produce enormous quantities of digital data with which researchers can document children's everyday interactions in naturalistic settings and assessment contexts. Machine learning and other computational approaches can produce replicable, automated measurements of these big behavioral data. The economies of scale afforded by repeated automated measurements offer a potent approach to investigating linkages between real-time behavior and developmental change. In our work, automated measurement of audio from child-worn recorders-which quantify the frequency of child and adult speech and index its phonemic complexity-are paired with ultrawide radio tracking of children's location and interpersonal orientation. Applications of objective measurement indicate the influence of adult behavior in both expert ratings of attachment behavior and ratings of autism severity, suggesting the role of dyadic factors in these "child" assessments. In the preschool classroom, location/orientation measures provide data-driven measures of children's social contact, fertile ground for vocal interactions. Both the velocity of children's movement toward one another and their social contact with one another evidence homophily: children with autism spectrum disorder, other developmental disabilities, and typically developing children were more likely to interact with children in the same group even in inclusive preschool classrooms designed to promote interchange between all children. In the vocal domain, the frequency of peer speech and the phonemic complexity of teacher speech predict the frequency and phonemic complexity of children's own speech over multiple timescales. Moreover, children's own speech predicts their assessed language abilities across disability groups, suggesting how everyday interactions facilitate development.

Keywords: Audio; Automated measurement; Deep learning; Development; Interaction; Language; Machine learning; Objective; Radio frequency identification; Social.

Figure 1.

ADOS-2 Assessment. A child’s face and expression are captured by examiner-worn video-enabled eyeglasses and detected by a machine learning program during administration of the ADOS-2.

Figure 2.

Phonemic Diversity and Assessed Language Ability. (A) Mediation model. The phonemic diversity of teachers’ speech was associated with the phonemic diversity of children’s speech, which, in turn, positively predicted children’s end-of-year expressive language abilities. (B) Scatterplot. Children who produced speech with a higher number of unique phonemes scored higher on a standardized measure of expressive language ability, the PLS-5. (Parallel results with receptive language ability not shown.) Credit: Mitsven, S. G., Perry, L. K., Tao, Y., Elbaum, B. E., Johnson, N. F., & Messinger, D. S. (2021). Objectively measured teacher and preschooler vocalizations: Phonemic diversity is associated with language abilities. Developmental science.

Figure 3.

Materials used to continuously track children’s and teachers’ location and orientation in the classroom. Ubisense sensors are arrayed, one in each corner of the classroom, and track the ultrawide RFID signal emitted by each of the two tags worn by children and by teachers.

Figure 4.

The radial distribution function. The radial distribution function, g(r), indicates distances at which the probability of child–child and child-teacher pairs being in contact exceeds chance (g(r) = 1). The grey area above between 0.2 and 2 m index illustrates the proximity criterion for social contact across cohorts (classes of children). Figure compliments of Chitra Banarjee, Yale University.

Figure 5.

Semi-log plots of the mean frequency of children’s social contacts by ordinal rank. Levels of social contact, aggregated for all children, are displayed for each child’s most to least contacted peer. Rank 1 indicates the most contacted peer, Rank 2 the second-most, and so on. Boys and girls’ mean contact frequency are shown separately. Both males and females consistently show levels of social contact with their most contacted peers that are ten to hundreds of time higher than levels of contact with their least contacted peers. Credit: Messinger, D. S., Prince, E. B., Zheng, M., Martin, K., Mitsven, S. G., Huang, S., … & Song, C. (2019). Continuous measurement of dynamic classroom social interactions. International Journal of Behavioral Development, 43(3), 263–270.

Figure 6.

Classroom visualization. A visualization of objective measurements of child and teacher location in an inclusion classroom. The tip of each triangle indicates the direction each child and teacher is facing (orientation). Orientation is calculated from the 2 Ubisense tags

Figure 7.

Mutual orientation. The heat map indicates the angle of each child to other children & teachers (θ₁ to θ₂) from .2–2m. Children tend to be shoulder to shoulder (~=90°), & face-to-face (within |45°| of 0°) with partners. The color bar indicates the ratio of observed mutual orientations to chance (=1, yellow line; black line =1.5). Chance relative orientation refers to the cumulative probability of relative orientations between all children over an entire observation.

Figure 8.

Peer speech transmission. Visualized associations between peer vocalizations in social contact with focal child at observation t and child vocalizations to those peers at t+1. The association did not differ by the focal child’s hearing status. The vocalization rate per hour is presented on a log10(x+1) scale. Each point represents one child’s vocalizations to one peer. Error bands show standard errors of the mean.

Figure 9.

Classroom speech network. Each node is one child in a classroom. Node size is proportional to total vocalization rate (input and output) in social contact with all peers. Tie width indexes the sum of vocalizations in social contact between two children (co-talk). For visualization purposes, the network was pruned such that only ties of greater than mean weight (both per observation and overall) are displayed. Credit: Fasano, R. M., Perry, L. K., Zhang, Y., Vitale, L., Wang, J., Song, C., & Messinger, D. S. (2021). A granular perspective on inclusion: Objectively measured interactions of preschoolers with and without autism. Autism Research.

Figure 10.

Mutual orientation and proximity. Mutual orientation is indexed by the relative orientation of two children (theta1 minus theta2) where 0 indicates a face-to-face orientation and ±180 indicates shoulder-to-shoulder orientation. R indicates the radius, the distance between the two children. The heat map indicates the likelihood of a given orientation and a given distance occurring with respect to chance (observed divided by chance). Reds indicate higher likelihoods probabilities and blues indicate lower likelihoods. The green line encloses all areas where the joint probability exceeds 1, and the black line encloses areas where the joint probability exceeds 2. Face-to-face contact tends to occur over relatively long distances (through 3 meters) but is concentrated in a small band around 1 meter. By contrast shoulder-to-shoulder contact (around plus and minus 180 degrees) tends to occur much more than chance in a relatively restricted range of distances between approximately .2 and 1.8 meters. Figure compliments of Chaoming Song and Yi Zhang, University of Miami.