Computational Fact Checking from Knowledge Networks

Abstract

Traditional fact checking by expert journalists cannot keep up with the enormous volume of information that is now generated online. Computational fact checking may significantly enhance our ability to evaluate the veracity of dubious information. Here we show that the complexities of human fact checking can be approximated quite well by finding the shortest path between concept nodes under properly defined semantic proximity metrics on knowledge graphs. Framed as a network problem this approach is feasible with efficient computational techniques. We evaluate this approach by examining tens of thousands of claims related to history, entertainment, geography, and biographical information using a public knowledge graph extracted from Wikipedia. Statements independently known to be true consistently receive higher support via our method than do false ones. These findings represent a significant step toward scalable computational fact-checking methods that may one day mitigate the spread of harmful misinformation.

Fig 1. Using Wikipedia to fact-check statements.

(a) To populate the knowledge graph with facts we use structured information contained in the ‘infoboxes’ of Wikipedia articles (in the figure, the infobox of the article about Barack Obama). (b) Using the Wikipedia Knowledge Graph, computing the truth value of a subject-predicate-object statement amounts to finding a path between subject and object entities. In the diagram we plot the shortest path returned by our method for the statement “Barack Obama is a muslim.” Numbers in parentheses indicate the degree of the nodes. The path traverses high-degree nodes representing generic entities, such as Canada, and is assigned a low truth value.

Fig 2. Ideological classification of the US Congress based on truth values.

(a) Ideological network of the 112th US Congress. The plot shows a subset of the WKG constituted by paths between Democratic or Republican members of the 112th US Congress and various ideologies. Red and blue nodes correspond to members of Congress, gray nodes to ideologies, and white nodes to vertices of any other type. The position of the nodes is computed using a force-directed layout [33], which minimizes the distance between nodes connected by an edge weighted by a higher truth value. For clarity only the most significant paths, whose values rank in the top 1% of truth values, are shown. (b) Ideological classification of members of the 112th US Congress. The plot shows on the x axis the party label probability given by a Random Forest classification model trained on the truth values computed on the WKG, and on the y axis the reference score provided by dw-nominate. Red triangles are members of Congress affiliated to the Republican party and blue circles to the Democratic party. Histograms and density estimates of the two marginal distributions, color-coded by actual affiliation, are shown on the top and right axes.

Fig 3. Automatic truth assessments for simple factual statements.

In each confusion matrix, rows represent subjects and columns represent objects. The diagonals represent true statements. Higher truth values are mapped to colors of increasing intensity. (a) Films winning the Oscar for Best Movie and their directors, grouped by decade of award (see the complete list in the S1 Text). (b) US presidents and their spouses, denoted by initials. (c) US states and their capitals, grouped by US Census Bureau-designated regions. (d) World countries and their capitals, grouped by continent.

Fig 4. Receiver Operating Characteristic for the multiple questions task.

For each confusion matrix depicted in Fig 3 we compute ROC curves where true statements correspond to the diagonal and false statements to off-diagonal elements. The red dashed line represents the performance of a random classifier.

Fig 5. Real-world fact-checking scenario.

(a) A document from the ground truth corpus. (b) Statement to fact-check: Did Steve Tesich graduate from Indiana University, Bloomington? This information is not present in the infobox, and thus it is not part of the WKG. (c) Annotations from five human raters. In this case, the majority of raters believe that the statement is true, and thus we consider it as such for classification purposes. (d) Receiver operating characteristic (ROC) curve of the classification for subject-predicate-object statements in which the predicate is “institution” (e.g., “Albert Einstein,” “institution,” “Institute for Advanced Studies”). A true positive rate above the false positive rate (dashed line), and correspondingly an area under the curve (AUC) above 0.5, indicate better than random performance. (e) ROC curve for statements with “degree” predicate (e.g., “Albert Einstein,” “degree,” “University Diploma”).