Transferability Based on Drug Structure Similarity in the Automatic Classification of Noncompliant Drug Use on Social Media: Natural Language Processing Approach

Abstract

Background: Medication noncompliance is a critical issue because of the increased number of drugs sold on the web. Web-based drug distribution is difficult to control, causing problems such as drug noncompliance and abuse. The existing medication compliance surveys lack completeness because it is impossible to cover patients who do not go to the hospital or provide accurate information to their doctors, so a social media-based approach is being explored to collect information about drug use. Social media data, which includes information on drug usage by users, can be used to detect drug abuse and medication compliance in patients.

Objective: This study aimed to assess how the structural similarity of drugs affects the efficiency of machine learning models for text classification of drug noncompliance.

Methods: This study analyzed 22,022 tweets about 20 different drugs. The tweets were labeled as either noncompliant use or mention, noncompliant sales, general use, or general mention. The study compares 2 methods for training machine learning models for text classification: single-sub-corpus transfer learning, in which a model is trained on tweets about a single drug and then tested on tweets about other drugs, and multi-sub-corpus incremental learning, in which models are trained on tweets about drugs in order of their structural similarity. The performance of a machine learning model trained on a single subcorpus (a data set of tweets about a specific category of drugs) was compared to the performance of a model trained on multiple subcorpora (data sets of tweets about multiple categories of drugs).

Results: The results showed that the performance of the model trained on a single subcorpus varied depending on the specific drug used for training. The Tanimoto similarity (a measure of the structural similarity between compounds) was weakly correlated with the classification results. The model trained by transfer learning a corpus of drugs with close structural similarity performed better than the model trained by randomly adding a subcorpus when the number of subcorpora was small.

Conclusions: The results suggest that structural similarity improves the classification performance of messages about unknown drugs if the drugs in the training corpus are few. On the other hand, this indicates that there is little need to consider the influence of the Tanimoto structural similarity if a sufficient variety of drugs are ensured.

Keywords: data mining; machine learning; medication noncompliance; natural language processing; pharmacovigilance; text classification; transfer learning.

Figure 1

Our approach, transfer learning based on chemical structures, assumes similarly structured corpora are transferable.

Figure 2

Value of F1 score for transfer learning.

Figure 3

The Tanimoto similarity between each drug.

Figure 4

Relationship between the Tanimoto similarity and F1 value for each drug.

Figure 5

Comparison of the accuracy of the 2 models. Sim is a model transfer learned from a data set of drugs with close similarity; Rnd is a model transfer learned from a data set of randomly selected drugs.

Figure 6

The Tanimoto similarity and F1 score pairs for each drug. OTC-rel type contains an over-the-counter (OTC) drug in one of the pairs; antipsycho type is a combination of antipsychotic medications such as sleeping pills, anxiolytics, and antischizophrenics; and other type is any other combination.).

Figure 7

Comparison of the results of transfer learning for drugs with similar structure and drugs with similar indications.

Figure 8

Scatterplot showing the relationship between the number of tweets and F1 value for each drug. G-m: general mention; G-u: general use; NC-s: noncompliant sales; NC-u/m: noncompliant use or mention.