Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Review
. 2023 Aug 17:6:1179625.
doi: 10.3389/fdata.2023.1179625. eCollection 2023.

Discovering anomalies in big data: a review focused on the application of metaheuristics and machine learning techniques

Claudia Cavallaro  1 , Vincenzo Cutello  1 , Mario Pavone  1 , Francesco Zito  1
Affiliations
Review

Discovering anomalies in big data: a review focused on the application of metaheuristics and machine learning techniques

Claudia Cavallaro et al. Front Big Data. .

Abstract

With the increase in available data from computer systems and their security threats, interest in anomaly detection has increased as well in recent years. The need to diagnose faults and cyberattacks has also focused scientific research on the automated classification of outliers in big data, as manual labeling is difficult in practice due to their huge volumes. The results obtained from data analysis can be used to generate alarms that anticipate anomalies and thus prevent system failures and attacks. Therefore, anomaly detection has the purpose of reducing maintenance costs as well as making decisions based on reports. During the last decade, the approaches proposed in the literature to classify unknown anomalies in log analysis, process analysis, and time series have been mainly based on machine learning and deep learning techniques. In this study, we provide an overview of current state-of-the-art methodologies, highlighting their advantages and disadvantages and the new challenges. In particular, we will see that there is no absolute best method, i.e., for any given dataset a different method may achieve the best result. Finally, we describe how the use of metaheuristics within machine learning algorithms makes it possible to have more robust and efficient tools.

Keywords: anomaly detection; classification; deep learning; fault detection; machine learning; metaheuristics; recurrent neural network; security threats.

PubMed Disclaimer

Conflict of interest statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Figures

Figure 1
Figure 1
Structure of a recurrent layer.
Figure 2
Figure 2
Architecture of the recurrent neural network considered.
Figure 3
Figure 3
(A) Confusion matrix of the non-optimized NN. (B) Confusion matrix of the optimized NN.
See this image and copyright information in PMC

References

    1. Agrawal R., Imieliński T., Swami A. (1993). Mining association rules between sets of items in large databases. ACM SIGMOD Rec. 22, 207–216. 10.1145/170036.170072 - DOI
    1. Ahmed H., Traore I., Saad S. (2017). Detecting opinion spams and fake news using text classification. Secur. Privacy 1, e9. 10.1002/spy2.9 - DOI
    1. Bejoy B., Raju G., Swain D., Acharya B., Hu Y.-C. (2022). A generic cyber immune framework for anomaly detection using artificial immune systems. Appl. Soft Comput. 130:109680. 10.1016/j.asoc.2022.109680 - DOI
    1. Bock S., Weiß M. (2019). “A proof of local convergence for the Adam optimizer,” in 2019 International Joint Conference on Neural Networks (IJCNN) (Piscataway, NJ: IEEE; ), 1–8. 10.1109/IJCNN.2019.8852239 - DOI
    1. Bottou L. (2012). “Stochastic gradient descent tricks,” in Neural Networks: Tricks of the Trade, 2nd Edn, eds G. Montavon, G. B. Orr, and K. R. Müller (Berlin; Heidelberg: Springer Berlin Heidelberg; ), 421–436. 10.1007/978-3-642-35289-8_25 - DOI

LinkOut - more resources