A multi-modal approach towards mining social media data during natural disasters - a case study of Hurricane Irma

Abstract

Streaming social media provides a real-time glimpse of extreme weather impacts. However, the volume of streaming data makes mining information a challenge for emergency managers, policy makers, and disciplinary scientists. Here we explore the effectiveness of data learned approaches to mine and filter information from streaming social media data from Hurricane Irma's landfall in Florida, USA. We use 54,383 Twitter messages (out of 784K geolocated messages) from 16,598 users from Sept. 10 - 12, 2017 to develop 4 independent models to filter data for relevance: 1) a geospatial model based on forcing conditions at the place and time of each tweet, 2) an image classification model for tweets that include images, 3) a user model to predict the reliability of the tweeter, and 4) a text model to determine if the text is related to Hurricane Irma. All four models are independently tested, and can be combined to quickly filter and visualize tweets based on user-defined thresholds for each submodel. We envision that this type of filtering and visualization routine can be useful as a base model for data capture from noisy sources such as Twitter. The data can then be subsequently used by policy makers, environmental managers, emergency managers, and domain scientists interested in finding tweets with specific attributes to use during different stages of the disaster (e.g., preparedness, response, and recovery), or for detailed research.

Keywords: data mining; machine learning; natural disaster; social media.

Figure 1:

The path of Hurricane Irma in September 2017 (orange line), the extent of tropical storm force winds (pink outline), and the location for all 784K geolocated tweets used as the basis for this study (black dots).

Figure 2:

Overall information flow model. The Metadata Extraction stage develops variables from the raw Twitter data, Filtering stage utilizes the developed 1) Geospatial, 2) User, 3) Image, and 4) Text analysis modules to score tweets, and the Visualization stage is used to observe the at location posted image along with Google Street View

Figure 3:

Precipitation and wind speed in relation to the distance from Hurricane Irma’s eye.

Figure 4:

Cumulative Distribution Function (CDF) for Min-Max Normalization, Log, and Box-Cox transformed geospatial scores for the nine models. The common legend of all three figures is shown in Figure c.

Figure 5:

Cumulative Distribution Function (CDF) and F1-Scores for the top five geospatial models.

Figure 6:

Area Under - Receiver Operating Characteristics (AU-ROC) Curves for V3, VGG net, ResNet architecture and Tuned Inception V3 models for binary classification of images (hurricane related versus non-hurricane related).

Figure 7:

Area Under - Receiver Operating Characteristics (AU-ROC) Curves for Tuned Inception V3 model for multi-label annotation for images - 1) ‘Flood’, 2) ‘Wind’, and 3) ‘Destruction’.

Figure 8:

AU-ROC Curves for Random Forest, Gradient Boosted, and Logistic Regression Classifiers in predicting Verified users.

Figure 9:

AU-ROC Curves for text — 1) Cosine Similarity of Tweet Vector Sum (CSTVS), 2) Dot Product of Search Term Vector and Tweet Vector Sum (DP), 3) Mean Cosine Similarity (MCS), 4) Sum of Cosine Similarity over Square Root of Token Count (SCSSC).

Figure 10:

Performance of the user Random Forest Classifier in the binary classes and feature importance metrics.

Figure 11:

CDF of Overall Model and percentage of tweets passing different model thresholds.