A novel technique for ransomware detection using image based dynamic features and transfer learning to address dataset limitations

Abstract

The increasing frequency of ransomware attacks necessitates the development of more effective detection methods. Existing image-based ransomware detection approaches have largely focused on static analysis, overlooking specialized ransomware behaviors such as encryption, privilege escalation, and system recovery disruption. Although dynamic and memory forensics-based visualization methods exist in the broader malware domain, they primarily target generic malware families and often rely on memory dumps or system snapshots without transforming behavioral features into spatially meaningful representations. Moreover, traditional machine learning methods such as Random Forest (RF), Support Vector Machine (SVM), and K-Nearest Neighbors (KNN) typically depend on manual feature engineering and large labelled datasets, limiting scalability and adaptability. To address these limitations, we propose a novel behavior-to-image ransomware detection framework that transforms dynamic behavioral features extracted from sandbox-generated JSON reports into two-dimensional (2D) grayscale and color image representations, optimized for transfer learning (TL), enabling effective classification under small-data conditions. Our approach integrates domain-specific feature filtering and impact analysis to ensure the selection of the most ransomware-relevant attributes. TL subsequently automates feature extraction and classification, eliminating the need for separate feature selection procedures and overcoming the time-consuming process of manual feature engineering. Furthermore, by leveraging prior knowledge from large-scale image datasets, TL significantly mitigates the need for extensive labelled data while maintaining high detection accuracy and strong generalization. Experimental results demonstrate that fine-tuned pretrained models, notably ResNet50, achieve up to 99.96% accuracy with a minimal loss factor of 0.0026, even with a small dataset of 500 ransomware and 500 benign samples. We further validated the model's interpretability through t-SNE visualizations and saliency maps, confirming its ability to focus on class-discriminative behavioral patterns. The low misclassification rate, along with the transparency of the model, highlights its potential for practical deployment in ransomware detection systems.

Keywords: Convolutional neural network; Dynamic analysis; Image classification; Portable executable (PE); Pretrained models; Ransomware; Transfer learning.

Fig. 1

Proposed ransomware detection workflow.

Fig. 2

Timeline showing the prevalence and family wise distribution of ransomware samples collected between 2019 and 2024. The ordering reflects attack frequency and global impact based on multi-source threat intelligence reports.

Algorithm 1

Extraction and structuring of dynamic behavioral features from JSON reports.

Algorithm 2

Generation of 2D image representations from structured CSV data.

Algorithm 3

Generation of 2D image representations from structured CSV data.

Fig. 3

Accuracy trends for various pretrained learning models. The ResNet50 model demonstrates superior accuracy as the number of epochs increases.

Fig. 4

The loss trends of pretrained models show that reveals that the ResNet50 model consistently demonstrates the lowest loss as the number of epochs increases.

Fig. 5

Comparative performance of non-pretrained models and ResNet50 over 15 epochs.

Fig. 6

Graph of the accuracy against the epoch across all models (pretrained and non-pretrained). The consistently lowest-performing model is ANN, whereas RsNet50 consistently ranks highest.

Fig. 7

Graph of the loss against the epoch across all models (pretrained and non-pretrained). The consistently lowest loss performing model is ResNet50, whereas ANN shows the highest loss.

Fig. 8

t-SNE feature visualization of used ransomware dataset. Orange points represent benign samples and green points represent ransomware. The two classes formed distinct clusters with limited overlap, illustrating strong feature separability and explaining the near-perfect classification accuracy of the model.

Fig. 9

Saliency maps of representative benign (left) and ransomware (right) images. The highlighted regions indicate the pixels with the highest impact on the prediction of the model. The model focuses on structured behavior-encoded areas and confirms the class-specific pattern learning.

Fig. 10

Confusion matrix for the best performing model ResNet50, evaluated on the 200-sample validation set (98.50% accuracy).