Multimodal analysis of disinformation and misinformation

Abstract

The use of disinformation and misinformation campaigns in the media has attracted much attention from academics and policy-makers. Multimodal analysis or the analysis of two or more semiotic systems-language, gestures, images, sounds, among others-in their interrelation and interaction is essential to understanding dis-/misinformation efforts because most human communication goes beyond just words. There is a confluence of many disciplines (e.g. computer science, linguistics, political science, communication studies) that are developing methods and analytical models of multimodal communication. This literature review brings research strands from these disciplines together, providing a map of the multi- and interdisciplinary landscape for multimodal analysis of dis-/misinformation. It records the substantial growth starting from the second quarter of 2020-the start of the COVID-19 epidemic in Western Europe-in the number of studies on multimodal dis-/misinformation coming from the field of computer science. The review examines that category of studies in more detail. Finally, the review identifies gaps in multimodal research on dis-/misinformation and suggests ways to bridge these gaps including future cross-disciplinary research directions. Our review provides scholars from different disciplines working on dis-/misinformation with a much needed bird's-eye view of the rapidly emerging research of multimodal dis-/misinformation.

Keywords: literature review; machine learning; multimodal dis-/misinformation; qualitative analysis.

Figure 1.

PRISMA statement. The number of records included at each step are shown.

Figure 2.

(a) Number of records published each year (1900 to 31 March 2020) within scope of Stage 1. (b) The number of records for each type of document.

Figure 3.

The number of records by the same author (a) or records published in the same source (b).

Figure 4.

Subject breakdown. The breakdown of subjects for all eligible records (a) and the breakdown into sub-subjects for the records in the social sciences (b).

Figure 5.

Methodology. The distribution of subjects for content-specific eligible records (a), and the type of methodology used in these records (b).

Figure 6.

The 20 most common words in the abstracts after stop words were removed.

Figure 7.

Coherence scores for LDA models for different numbers of topics (minimum 2). The red-highlighted points indicate possible choices of topic numbers.

Figure 8.

The frequency with which each topic was the primary topic in an abstract (a) and the normalized frequency with which topics co-occurred with other topics (b).

Figure 9.

The frequency with which each topic was the primary topic in an abstract.

Figure 10.

This chart compares the number of CS versus non CS publications across both Stages 1 and 2 using the stricter selection criteria outlined in Stage 2. All papers were screened to check that they were applicable to dis-/misinformation; the earliest example dated from 2015.

Figure 11.

This is a general schematic of a bimodal disinformation detector. The blue boxes labelled 1 and 2 are modules that process each modality of the input data. The role of module 3 is to pass information, no matter how complex, between modalities. The outputs of these modules are first aggregated before then being passed into a black box classifier; concatenation and ‘multi-layer perceptrons’ were the most commonly observed ML methods, respectively.

Figure 12.

Shown here is where the data originates and how these sources mix. The breakdowns were chosen to highlight the sorts of disinformation studied. A source was counted if the paper’s data source was distinct (e.g. datasets ‘X’ and ‘Y’ count as two). We counted 91 instances of sources, hence an average mixing of 1.25 sources per paper—largely independent. (a) Depicting the mixing of data sources and (b) breakdown of ‘Miscellaneous’ in (a).