Machine learning approaches to evaluate infants' general movements in the writhing stage-a pilot study

Abstract

The goals of this study are to describe machine learning techniques employing computer-vision movement algorithms to automatically evaluate infants' general movements (GMs) in the writhing stage. This is a retrospective study of infants admitted 07/2019 to 11/2021 to a level IV neonatal intensive care unit (NICU). Infant GMs, classified by certified expert, were analyzed in two-steps (1) determination of anatomic key point location using a NICU-trained pose estimation model [accuracy determined using object key point similarity (OKS)]; (2) development of a preliminary movement model to distinguish normal versus cramped-synchronized (CS) GMs using cosine similarity and autocorrelation of major joints. GMs were analyzed using 85 videos from 74 infants; gestational age at birth 28.9 ± 4.1 weeks and postmenstrual age (PMA) at time of video 35.9 ± 4.6 weeks The NICU-trained pose estimation model was more accurate (0.91 ± 0.008 OKS) than a generic model (0.83 ± 0.032 OKS, p < 0.001). Autocorrelation values in the lower limbs were significantly different between normal (5 videos) and CS GMs (5 videos, p < 0.05). These data indicate that automated pose estimation of anatomical key points is feasible in NICU patients and that a NICU-trained model can distinguish between normal and CS GMs. These preliminary data indicate that machine learning techniques may represent a promising tool for earlier CP risk assessment in the writhing stage and prior to hospital discharge.

Figure 1

Data pipeline using a two-step framework. (A) Model and data pipeline illustrating the two-step framework composed of two distinct models. Step 1 consists of a pose estimation model—which is a neural network trained to detect and localize anatomical key points on NICU specific data. Step 2 consists of a preliminary movement model trained to classify movement as normal or CS using the time series information generated by the pose estimation model. (B) Example of the data output from a representative patient. The key points form the infant’s skeleton from which movement is analyzed. (C) List of the 17 anatomic key points inferred by the pose estimation model.

Figure 2

Cosine similarity calculation method. The preliminary movement model (step 2) is trained to measure the cosine similarity between two vectors anchored at the joint. (A) Example of a left elbow anchor. The joint angle is calculated using the cosine similarity of the two vectors: (A) left elbow key point to left shoulder key point and (B) left elbow key point to left wrist key point. (B) List of the eight joint anchors and their respective endpoints used in the movement model. (C) Shows the spread of the autocorrelation values for k = 5, 7, 11, 13 for the eight joints between normal and CS.

Figure 3

Distribution of gestational age and birth weight of the cohort. The mean GA at birth was 29.9 ± 4.1 weeks (range 23–40 weeks) and the mean birth weight was 1803 ± 387 g.

Figure 4

Training of the key point detection model on NICU images. (A) Representative example in two patients of the performance of the NICU trained model versus the (MS COCO model) for key point detection. (B) OKS results in the NICU-trained versus the MS-COCO models demonstrating a maximum of 0.912 OKS. We performed an ablation analysis where the model was trained on 0% (open-source model), 25%, 50%, 75%, 100% of the available NICU training data and tested each models performance on a hold-out test set. The results show that OKS performance begins to saturate starting at 50% and OKS standard deviation reaches its minimum at 75%. This indicates that additional data would only marginally improve the pose estimation model. For this analysis the following parameters were used: $s=1$ and $k=0.001$ for all 17 anatomic key points. Legend: MS COCO (Microsoft Common Objects in Context); OKS (Object Key point Similarity).

Figure 5

Time series of cosine similarity in normal versus cramped synchronized GMs. Representative examples of time series of cosine similarity for each joint for normal (A) and CS GMs (B) are shown. The x-axis represents the frame number and the y-axis represents the cosine similarity. A cosine similarity of 0 indicates perpendicular vectors, + 1 indicates vectors with similar orientation and −1 indicates vectors in opposite direction. Because CS GMs lack variability and movements tend to be in extension with activation of limbs at the same time, the signal demonstrates increase in repeated patterns best shown in the left elbow and left knee as well as frequent occurrence of cosine similarity of −1.

Figure 6

Autocorrelation values in normal versus abnormal GMs. Average autocorrelation values of the eight anatomic anchors for all patients are shown at each lag level (CS GMs are shown in red, normal GMs are shown in green and individual patients are shown in light grey). CS GMs have a higher autocorrelation value at every lag level compared to normal GMs, particularly in the lower extremities.