Benchmarking AutoML for regression tasks on small tabular data in materials design

Abstract

Machine Learning has become more important for materials engineering in the last decade. Globally, automated machine learning (AutoML) is growing in popularity with the increasing demand for data analysis solutions. Yet, it is not frequently used for small tabular data. Comparisons and benchmarks already exist to assess the qualities of AutoML tools in general, but none of them elaborates on the surrounding conditions of materials engineers working with experimental data: small datasets with less than 1000 samples. This benchmark addresses these conditions and draws special attention to the overall competitiveness with manual data analysis. Four representative AutoML frameworks are used to evaluate twelve domain-specific datasets to provide orientation on the promises of AutoML in the field of materials engineering. Performance, robustness and usability are discussed in particular. The results lead to two main conclusions: First, AutoML is highly competitive with manual model optimization, even with little training time. Second, the data sampling for train and test data is of crucial importance for reliable results.

Figure 1

Data processing pipeline for a typical data mining workflow in materials design, starting with a material dataset and ending with model testing. Steps within the highlighted space can be automated by use of AutoML tools.

Figure 2

Design space visualisation from chosen datasets. (a) Histogram of one target value normalized via standard scaler. (b) Representation of the standardized input feature space for the selected datasets via visualization of the first two principal components , the color of the points represents the target value.

Figure 3

Workflow for the evaluation process.

Figure 4

The mean relative scores of the four tested AutoML frameworks and the best AutoML aggregation per training time. (a) Mean relative score based on ${\mathrm{R}}^2$ (b) Mean relative score based on RMSE.

Figure 5

The relative score from the outer splits per task. Relative score means ${\mathrm{MAE}}_{\mathit{rel}}$ for Matbench-steels, ${\mathrm{MAPE}}_{\mathit{rel}}$ for Xiong and ${\mathrm{R}}_{\mathit{rel}}^2$ otherwise. For Hu and Koya the literature provides a performance range, represented by a black “error bar”.

Figure 6

The performance ${\mathrm{R}}^2$ of with respect to the dataset size and shape, one box represents all outer loop runs of one dataset. (a) ${\mathrm{R}}^2$ over dataset size, the boxes are slightly shifted to avoid overlapping, without affecting the interpretation of the graphic. (b) ${\mathrm{R}}^2$ over dataset size divided by number of features.