The origins of unpredictability in life outcome prediction tasks

Abstract

Why are some life outcomes difficult to predict? We investigated this question through in-depth qualitative interviews with 40 families sampled from a multidecade longitudinal study. Our sampling and interviewing process was informed by the earlier efforts of hundreds of researchers to predict life outcomes for participants in this study. The qualitative evidence we uncovered in these interviews combined with a mathematical decomposition of prediction error led us to create a conceptual framework. Our specific evidence and our more general framework suggest that unpredictability should be expected in many life outcome prediction tasks, even in the presence of complex algorithms and large datasets. Our work provides a foundation for future empirical and theoretical work on unpredictability in human lives.

Keywords: life course; limits to prediction; machine learning; mixed methods; prediction.

Fig. 1.

Life outcome prediction task. In the task we studied, the feature observation window is from the child’s birth to age 9. The intervening period follows from age 9 to 15. The outcome time is age 15. The chosen features are 12,942 survey responses. The chosen outcome is self-reported grade point average (GPA), which ranges from 1.00 (worst) to 4.00 (best).

Fig. 2.

Origins of unpredictability: irreducible error and learning error. See also figure 1.6 in ref. . Each dot represents a person; people who share a single value on the predictor have many outcomes (each vertical set of dots). This type of error is fixed by the task definition, so we call it irreducible error. Second, the estimated prediction function (regression line) is not equal to the true conditional expectation in each subgroup defined by $X$. The prediction is wrong in this case because the true relationship is not linear, but it could also be wrong due to sampling variability or bias. Because this component relates to the learning procedure, we call it learning error.

Fig. 3.

Inputs to irreducible error and learning error.

Fig. 4.

Sources of irreducible error.

Fig. 5.

Generalizing to other life outcome prediction tasks.