Noisecut: a python package for noise-tolerant classification of binary data using prior knowledge integration and max-cut solutions

Abstract

Background: Classification of binary data arises naturally in many clinical applications, such as patient risk stratification through ICD codes. One of the key practical challenges in data classification using machine learning is to avoid overfitting. Overfitting in supervised learning primarily occurs when a model learns random variations from noisy labels in training data rather than the underlying patterns. While traditional methods such as regularization and early stopping have demonstrated effectiveness in interpolation tasks, addressing overfitting in the classification of binary data, in which predictions always amount to extrapolation, demands extrapolation-enhanced strategies. One such approach is hybrid mechanistic/data-driven modeling, which integrates prior knowledge on input features into the learning process, enhancing the model's ability to extrapolate.

Results: We present NoiseCut, a Python package for noise-tolerant classification of binary data by employing a hybrid modeling approach that leverages solutions of defined max-cut problems. In a comparative analysis conducted on synthetically generated binary datasets, NoiseCut exhibits better overfitting prevention compared to the early stopping technique employed by different supervised machine learning algorithms. The noise tolerance of NoiseCut stems from a dropout strategy that leverages prior knowledge of input features and is further enhanced by the integration of max-cut problems into the learning process.

Conclusions: NoiseCut is a Python package for the implementation of hybrid modeling for the classification of binary data. It facilitates the integration of mechanistic knowledge on the input features into learning from data in a structured manner and proves to be a valuable classification tool when the available training data is noisy and/or limited in size. This advantage is especially prominent in medical and biomedical applications where data scarcity and noise are common challenges. The codebase, illustrations, and documentation for NoiseCut are accessible for download at https://pypi.org/project/noisecut/ . The implementation detailed in this paper corresponds to the version 0.2.1 release of the software.

Keywords: Binary data; Hybrid mechanistic/data-driven modeling; Max-cut problem; Noise-tolerant classification; Overfitting.

Fig. 1

The visual representation of illustrative introduced in Eqs. (1–4)

Fig. 2

A schematic representation of the information flow from binary-represented input data to binary labels. This procedure has been used to generate the synthetic datasets

Fig. 3

A tree-structured FN $\mathcal{F}:x\in {\{0,1\}}^N⟼y\in \{0,1\}$, which maps binary-represented data to binary output. The FN has M first-layer boxes, operating on separate subsets of the input variables: $f_m={\text{F}}_m(\text{Decimal}({[x_m^i]}_{i=1}^{i=n_m}))$. The output box in the second layer processes the outcomes of the first-layer boxes towards the overall output of the FN: $y={\text{F}}_O(\text{Decimal}({[f_i]}_{i=1}^{i=M}))$

Fig. 4

Classifier accuracy on testing datasets comparison of NoiseCut with various ML models for classifying binary data across the entire spectrum of noise intensities, with a consistent 70% training data size. NoiseCut outperforms the others as noise intensifies, demonstrating superior overfitting mitigation across varying levels of noise compared to the early stopping approach used by the other ML models

Fig. 5

a. Comparison of ROC curves illustrating the classification performance of NoiseCut alongside other ML models on testing datasets. b. Comparison of computational time between NoiseCut and the other ML models across varying sample sizes. The evaluation is conducted with only 30% of the training data available and 5% noise intensity in the data labeling