Aligning syntactic structure to the dynamics of verbal communication: A pipeline for annotating syntactic phrases onto speech acoustics

Abstract

To investigate how the human brain encodes the complex dynamics of natural languages, any viable and reproducible analysis pipeline must rely on either manual annotations or natural language processing (NLP) tools, which extract relevant physical (e.g., acoustic, gestural), and structure-building information from speech and language signals. However, annotating syntactic structure for a given natural language is arguably a harder task than annotating the onset and offset of speech units such as phonemes and syllables, as the latter can be identified by relying on the physically overt and temporally measurable properties of the signal, while syntactic units are generally covert and their chunking is model-driven. We describe and validate a pipeline that takes into account both physical and theoretical aspects of speech and language signals, and operates a theory-driven and explicit alignment between overt speech units and covert syntactic units.

Keywords: Alignment; Annotation; Language; Neural tracking; Syntax.

Fig. 1

Word timestamps extraction. The acoustic signal is synthesized with gTTS (Durette, 2024) in Python, and then the annotation is performed manually in Praat (Boersma & Weenink, 2023). The temporal references of words are marked. Note that two words do not overlap on the time axis, i.e., their onsets and/or offsets never intersect, but follow one another sequentially

Fig. 2

Visualization of a syntactic tree. Example of the syntactic tree for the sentence “The idea of one employee improved the quality of the product”. The labeling scheme is as follows: Sentence (S); Noun Phrase (NP); Verb Phrase (VP); Prepositional Phrase (PP). All other labels refer to the Part-of-Speech (PoS) of the words: Determiner (Det); Adposition (Adp); Noun (Nouns); Verb (Verb)

Fig. 3

Alignment of word timestamps and syntactic structure. In this example, the higher-order phrases will share some of the onset/offset with lower-order phrases, as they are nested into each other. For example, the higher NP (the idea of one employee) has the same onset as the lower-left NP (the idea) and the same offset as the lower-right PP (of one employee) and NP (one employee). For this example, annotations at word level, as well as for the syntactic structure, were performed manually in Praat (Boersma & Weenink, 2023). The audio signal was synthesized using the gTTS (Durette, 2024) library in Python

Fig. 4

General overview of the pipeline. Given word timestamps and the parsed syntactic structure, the pipeline provides explicit alignment by tagging every word with all syntactic node labels necessary to integrate that word into the sentence. In order to uniquely identify each syntactic node label, its relative position (i.e., treeposition) is added. The final output is a dataframe that contains the word timestamps aligned with the syntactic structure, such that every word is tagged with all relevant information

Fig. 5

Example of using the treepositions to navigate a tree. For every word in a sentence, one loops through the structure and extracts the relative treeposition for every node in which the word appears. The indices of each label are used as a unique identifier for each node in the structure

Fig. 6

Comparison of durations of syntactic phrases extracted manually and automatically. a QQplot of the distributions of the durations of automatically extracted phrases and manually extracted phrases in seconds. b Empirical cumulative probability distribution of both automatic and manual annotations; c Equivalence prior and posterior for all phrases. d Equivalence Bayesian independent-samples t tests for all phrase labels. In case of mismatch in the number of items to compare, we resampled the data