Comparison of individual and ensemble machine learning models for prediction of sulphate levels in untreated and treated Acid Mine Drainage

Abstract

Machine learning was used to provide data for further evaluation of potential extraction of octathiocane (S₈), a commercially useful by-product, from Acid Mine Drainage (AMD) by predicting sulphate levels in an AMD water quality dataset. Individual ML regressor models, namely: Linear Regression (LR), Least Absolute Shrinkage and Selection Operator (LASSO), Ridge (RD), Elastic Net (EN), K-Nearest Neighbours (KNN), Support Vector Regression (SVR), Decision Tree (DT), Extreme Gradient Boosting (XGBoost), Random Forest (RF), Multi-Layer Perceptron Artificial Neural Network (MLP) and Stacking Ensemble (SE-ML) combinations of these models were successfully used to predict sulphate levels. A SE-ML regressor trained on untreated AMD which stacked seven of the best-performing individual models and fed them to a LR meta-learner model was found to be the best-performing model with a Mean Squared Error (MSE) of 0.000011, Mean Absolute Error (MAE) of 0.002617 and R² of 0.9997. Temperature (°C), Total Dissolved Solids (mg/L) and, importantly, iron (mg/L) were highly correlated to sulphate (mg/L) with iron showing a strong positive linear correlation that indicated dissolved products from pyrite oxidation. Ensemble learning (bagging, boosting and stacking) outperformed individual methods due to their combined predictive accuracies. Surprisingly, when comparing SE-ML that combined all models with SE-ML that combined only the best-performing models, there was only a slight difference in model accuracies which indicated that including bad-performing models in the stack had no adverse effect on its predictive performance.

Keywords: Acid Mine Drainage; Environmental chemistry; Machine learning; Regression; Stacking ensemble machine learning; Sulphate.

Fig. 1

Scatter matrix (below main diagonal) and Pearson’s correlation matrix (above main diagonal) of Pump A indicate the interrelationships between water quality parameters. Diagonal histogram and density plots indicate the distribution of each parameter. For the correlation matrix, red circles are positive correlations, blue circles are negative correlations and larger circles indicate more strongly correlated variables

Fig. 2

Dimensionality reduction and feature extraction results obtained from PCA for Pump A, a Bi-plot indicating the clustering of individual observations and its relation to the loadings plot. b Expanded view of the loadings plot indicating the relationship between parameters. c Elbow method plot indicating the optimal number of clusters. d Scree plot indicating the amount of variance explained by each principal component

Fig. 3

Regression algorithm NMSE comparison for Pump A showing a all individual baseline models, b all models and a stacking regressor containing all the models and c all well-performing models and a stacking regressor containing all the best-performing models

Fig. 4

Testing statistics (MSE, MAE and R²) accuracy comparison of regression models trained on Pump A. a, b Stacking regressor using all models. c, d Stacking regressor using only the best-performing models

Fig. 5

Scatter matrix (below main diagonal) and Pearson’s correlation matrix (above main diagonal) of Pump B indicate the interrelationships between water quality parameters. Diagonal histogram and density plots indicate the distribution of each parameter. For the correlation matrix, red circles are positive correlations, blue circles are negative correlations and larger circles indicate more strongly correlated variables

Fig. 6

Dimensionality reduction and feature extraction results obtained from PCA for Pump B. a Bi-plot indicating the clustering of individual observations and its relation to the loadings plot. b Expanded view of the loadings plot indicating the relationship between parameters. c Elbow method plot indicating the optimal number of clusters. d Scree plot indicating the amount of variance explained by each principal component

Fig. 7

Regression algorithm NMSE comparison for Pump B showing a all individual baseline models, b all models and a stacking regressor containing all the models and c all good-performing models and a stacking regressor containing all the best-performing models

Fig. 8

Testing statistics (MSE, MAE and R²) accuracy comparison of regression models trained on Pump B. a, b Stacking regressor using all models. c, d Stacking regressor using only the best-performing models

Fig. 9

Scatter matrix (below main diagonal) and Pearson’s correlation matrix (above main diagonal) of Treated Water indicate the interrelationships between water quality parameters. Diagonal histogram and density plots indicate the distribution of each parameter. For the correlation matrix, red circles are positive correlations, blue circles are negative correlations and larger circles indicate more strongly correlated variables

Fig. 10

Dimensionality reduction and feature extraction results obtained from PCA for the Treated Water. a Bi-plot indicating the clustering of individual observations and its relation to the loadings plot. b Expanded view of the loadings plot indicating the relationship between parameters. c Elbow method plot indicating the optimal number of clusters. d Scree plot indicating the amount of variance explained by each principal component

Fig. 11

Regression algorithm NMSE comparison for Treated Water showing a all individual baseline models, b all models and a stacking regressor containing all the models and c all good-performing models and a stacking regressor containing all the best-performing models

Fig. 12

Testing statistics (MSE, MAE and R²) accuracy comparison of regression models trained on Treated Water. a, b Stacking regressor using all models. c, d Stacking regressor using only the best-performing models