A deep-learning model of prescient ideas demonstrates that they emerge from the periphery

Abstract

Where do prescient ideas-those that initially challenge conventional assumptions but later achieve widespread acceptance-come from? Although their outcomes in the form of technical innovation are readily observed, the underlying ideas that eventually change the world are often obscured. Here, we develop a novel method that uses deep learning to unearth the markers of prescient ideas from the language used by individuals and groups. Our language-based measure identifies prescient actors and documents that prevailing methods would fail to detect. Applying our model to corpora spanning the disparate worlds of politics, law, and business, we demonstrate that it reliably detects prescient ideas in each domain. Moreover, counter to many prevailing intuitions, prescient ideas emanate from each domain's periphery rather than its core. These findings suggest that the propensity to generate far-sighted ideas may be as much a property of contexts as of individuals.

Fig. 1.

Prescient senators, minimum three congressional terms. Mean prescience is computed, standardized, and bootstrapped 10k times at the politician-quarter level.

Fig. 2.

Illustration of how prescience is computed based on a sentence from the legal dataset that the model deems highly prescient. This sentence rates highly in prescience because the RICO token better predicted the future period, when RICO’s statutory language was heavily contested by the courts (9).

Fig. 3.

Gingrich senators. Average standardized prescience for the Gingrich senators and all other Republicans by congressional term.

Fig. 4.

Prescience predicts success (A) prescience predicts political reelection. Marginal effects plot from panel linear probability models of political reelection on standardized prescience; politician-term unit of analysis, with political party × congressional term fixed effects (β = 0.00860, P < 0.05). (B) Prescience predicts congressional committee status. Mean committee status for the top tercile (high prescience) and bottom tercile (low prescience) Congressional term with 10k politician bootstrapped SEs. Congressional committee status defined by the committee transfer ratio (Supplementary Material Appendix). Please see Supplementary Material for panel regressions with fixed effects and other controls (β = 0.0155, P < 0.001). (C) Prescience predicts highly cited court decisions. Marginal effects plot of linear regression model of log total citations on standardized prescience; judicial decision unit of analysis, with judge, court, and year fixed effects (β = 0.0693, P < 0.001). (D) Prescience predicts landmark Supreme Court decisions. Marginal effects plot of linear probability models of landmark decisions on standardized prescience. Landmark decision is defined as the top 5% most highly cited US Supreme Court decisions by year, with the sample restricted to US Supreme Court decisions. Models include judge and year fixed effects (β = 0.687, P < 0.001). (E) Prescience predicts firm stock returns. Marginal effects plot of linear regression models of yearly stock returns from 2012–2015 on 2011 standardized prescience; North American Industry Classification System (NAICS) three-digit industry fixed effects (β = 0.0422, P < 0.001). (F) Prescience predicts elite firm performance. Total stock returns since 2012 by prescience quartile and year (with top and bottom 5%). The y-axis shifts from annual stock returns in panel (E) to cumulative stock returns in panel (F).

Fig. 5.

Highly prescient ideas come from the periphery. Marginal effects plots regressing the probability of having elite prescience (top 5% in standardized prescience) on alternatives measures of peripheral positions using logistic regression and 95% confidence intervals. All regressions include controls for the log number of sentences and are restricted to observations with at least 50 sentences given increased variance in prescience with small sample size. (A and B) Highly prescient politicians come from peripheral network positions. Politician network defined using bill consponsorship data (15). Network periphery measured by standardized eigenvector centrality (β = 0.292, P < 0.001) and standardized k-core centrality (β = 0.277, P < 0.001) with additional centrality measures in the Supplementary Material. (C and D) Highly prescient court decisions come from the lower courts. Panel (C) depicts the probability of a prescient decision using both state and federal courts and year fixed effects. Panel (D) adds judge fixed effects and restricts the sample to federal decisions (for which we have judge disambiguated decisions). (E and F) Highly prescient ideas come from small firms. NAICS two-digit industry fixed effects. Firm size measured by standardized total assets (β = 0.207, P < 0.01) and the standardized number of employees (β = 0.309, P < 0.05).