Word embeddings quantify 100 years of gender and ethnic stereotypes

Abstract

Word embeddings are a powerful machine-learning framework that represents each English word by a vector. The geometric relationship between these vectors captures meaningful semantic relationships between the corresponding words. In this paper, we develop a framework to demonstrate how the temporal dynamics of the embedding helps to quantify changes in stereotypes and attitudes toward women and ethnic minorities in the 20th and 21st centuries in the United States. We integrate word embeddings trained on 100 y of text data with the US Census to show that changes in the embedding track closely with demographic and occupation shifts over time. The embedding captures societal shifts-e.g., the women's movement in the 1960s and Asian immigration into the United States-and also illuminates how specific adjectives and occupations became more closely associated with certain populations over time. Our framework for temporal analysis of word embedding opens up a fruitful intersection between machine learning and quantitative social science.

Keywords: ethnic stereotypes; gender stereotypes; word embedding.

Fig. 1.

Women’s occupation relative percentage vs. embedding bias in Google News vectors. More positive indicates more associated with women on both axes. $P<{10}^{-10}$, $r^2=0.499$. The shaded region is the $95\%$ bootstrapped confidence interval of the regression line. In this single embedding, then, the association in the embedding effectively captures the percentage of women in an occupation.

Fig. 2.

Average gender bias score over time in COHA embeddings in occupations vs. the average percentage of difference. More positive means a stronger association with women. In blue is relative bias toward women in the embeddings, and in green is the average percentage of difference of women in the same occupations. Each shaded region is the bootstrap SE interval.

Fig. 3.

Average ethnic (Asian vs. White) bias score over time for occupations in COHA (blue) vs. the average percentage of difference (green). Each shaded region is the bootstrap SE interval.

Fig. 4.

Pearson correlation in embedding bias scores for adjectives over time between embeddings for each decade. The phase shift in the 1960s–1970s corresponds to the US women’s movement.

Fig. 5.

Pearson correlation in embedding Asian bias scores for adjectives over time between embeddings for each decade.

Fig. 6.

Asian bias score over time for words related to outsiders in COHA data. The shaded region is the bootstrap SE interval.

Fig. 7.

Religious (Islam vs. Christianity) bias score over time for words related to terrorism in New York Times data. Note that embeddings are trained in 3-y windows, so, for example, 2000 contains data from 1999–2001. The shaded region is the bootstrap SE interval.