Lifelong Machine Learning Potentials

Abstract

Machine learning potentials (MLPs) trained on accurate quantum chemical data can retain the high accuracy, while inflicting little computational demands. On the downside, they need to be trained for each individual system. In recent years, a vast number of MLPs have been trained from scratch because learning additional data typically requires retraining on all data to not forget previously acquired knowledge. Additionally, most common structural descriptors of MLPs cannot represent efficiently a large number of different chemical elements. In this work, we tackle these problems by introducing element-embracing atom-centered symmetry functions (eeACSFs), which combine structural properties and element information from the periodic table. These eeACSFs are key for our development of a lifelong machine learning potential (lMLP). Uncertainty quantification can be exploited to transgress a fixed, pretrained MLP to arrive at a continuously adapting lMLP, because a predefined level of accuracy can be ensured. To extend the applicability of an lMLP to new systems, we apply continual learning strategies to enable autonomous and on-the-fly training on a continuous stream of new data. For the training of deep neural networks, we propose the continual resilient (CoRe) optimizer and incremental learning strategies relying on rehearsal of data, regularization of parameters, and the architecture of the model.

Figure 1

S_N2 reaction at a methyl carbon atom with leaving group X and nucleophile Y.

Figure 2

Convergence of the optimizers Adam, RPROP, and CoRe for training reference data set B. The test set RMSE values of (a) energies E^test and (b) atomic force components F_α,n^test are shown as a function of the training epoch n_epoch. RMSE values of individual HDNNPs are represented by dots, while their mean, which is unequal to the ensemble RMSE value, is shown by a solid line. Lifelong adaptive data selection was applied in the optimizations.

Figure 3

Training data set reduction of the optimizers Adam, RPROP, and CoRe for training reference data set B. The number of considered training conformations N^train is shown as a function of the training epoch n_epoch. The values of N^train of individual HDNNPs are represented by dots, while their mean is shown by a solid line. The black dashed line represents the number of training conformations which was used for fitting in each epoch.

Figure 4

Final accuracy of lMLPs for which a fraction of p_late training data of reference data set B was first available at a late training epoch. These data were either chosen randomly or a certain block was used. More detailed information about the procedure is provided in the main text. The test RMSE values of (a) energies E^test and (b) atomic force components F_α,n^test are shown as a function of the late data fraction p_late. RMSE values of individual HDNNPs are represented by dots, their mean by a solid line, and their range by a lighter colored band. The CoRe optimizer (β₁^b = 0.7 for “Block” with p_late > 0 and β₁^b = 0.725 otherwise) and lifelong adaptive data selection were applied for 1500 epochs.

Figure 5

Potential energy surface of the S_N2 reaction Br^– + CH₃Cl ⇌ BrCH₃ + Cl^–. The lMLP ensemble prediction energy E is referenced to the minimum DFT reference energy E_min^ref of the given conformation space spanned by DFT optimized structures with constrained distances r_Cl–C and r_Br–C. The color represents the error of E with respect to the DFT reference energy E^ref. Black dots show the explicit evaluations of the lMLP. The colors between black dots are interpolated.

Figure 6

Absolute values of the errors with respect to the DFT reference and the uncertainty quantification for the ensemble prediction of (a) energies |ΔE| and (b) atomic force components |ΔF_α,n| of the validation data. The order x sorts the validation conformations according to their uncertainty quantification.