Prompt-based fine-tuning with multilingual transformers for language-independent sentiment analysis

Abstract

In the era of global digital communication, understanding user sentiment across multiple languages is a critical challenge with wide-ranging applications in opinion mining, customer feedback analysis, and social media monitoring. This study advances the field of language-independent sentiment analysis by leveraging prompt-based fine-tuning with state-of-the-art transformer models. The performance of classical machine learning approaches, hybrid deep learning architectures, and multilingual transformer models is evaluated across eight typologically diverse languages: Arabic, English, French, German, Hindi, Italian, Portuguese, and Spanish. Baseline models are established using traditional machine learning approaches such as Support Vector Machines (SVM) and Logistic Regression, with feature extraction methods like TF-IDF. A hybrid deep learning model is introduced, combining Long Short-Term Memory (LSTM) and Convolutional Neural Networks (CNNs) to capture local and sequential text patterns. Building on these, pre-trained multilingual transformer models, specifically BERT-base-multilingual and XLM-RoBERTa, are fine-tuned for language-independent sentiment classification tasks. The key contribution lies in the implementation of prompt-based fine-tuning strategies for language independent sentiment analysis. Using (1) prefix prompts and (2) cloze-style prompts, a unified framework is established that employs templates designed in one language and evaluates their performance on data from the remaining $(n-1)$ languages. Experimental results demonstrate that transformer models, particularly XLM-RoBERTa with prompt-based fine-tuning outperform both classical and deep learning methods. With only 32 training examples per class, prefix prompts produce results comparable to standard fine-tuning, which typically uses 70-80% of the data for training. This highlights the potential of prompt-based learning for scalable, multilingual sentiment analysis in diverse language settings.

Fig. 1

Overview of the proposed approach: Tweets in eight languages (left) are paired with a prompt template in any (of the eight) languages. The model predicts the masked word based on context, mapping it to a sentiment class for consistent multilingual sentiment analysis.

Fig. 2

Graph visualizing the performance metrics (Precision, Recall, F1-Score, and Accuracy) for the Prefix Prompt approach using the two multilingual models (BERT-base-multilingual-cased and XLM-RoBERTa) across eight languages.

Fig. 3

Graph illustrating the performance metrics (Precision, Recall, F1-Score, and Accuracy) for two multilingual models (BERT and XLM-RoBERTa) across eight languages.