Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Sep 21;11(1):18757.
doi: 10.1038/s41598-021-96723-8.

Universal activation function for machine learning

Brosnan Yuen  1 , Minh Tu Hoang  1 , Xiaodai Dong  1 , Tao Lu  2
Affiliations

Universal activation function for machine learning

Brosnan Yuen et al. Sci Rep. .

Abstract

This article proposes a universal activation function (UAF) that achieves near optimal performance in quantification, classification, and reinforcement learning (RL) problems. For any given problem, the gradient descent algorithms are able to evolve the UAF to a suitable activation function by tuning the UAF's parameters. For the CIFAR-10 classification using the VGG-8 neural network, the UAF converges to the Mish like activation function, which has near optimal performance [Formula: see text] when compared to other activation functions. In the graph convolutional neural network on the CORA dataset, the UAF evolves to the identity function and obtains [Formula: see text]. For the quantification of simulated 9-gas mixtures in 30 dB signal-to-noise ratio (SNR) environments, the UAF converges to the identity function, which has near optimal root mean square error of [Formula: see text]. In the ZINC molecular solubility quantification using graph neural networks, the UAF morphs to a LeakyReLU/Sigmoid hybrid and achieves RMSE=[Formula: see text]. For the BipedalWalker-v2 RL dataset, the UAF achieves the 250 reward in [Formula: see text] epochs with a brand new activation function, which gives the fastest convergence rate among the activation functions.

PubMed Disclaimer

Conflict of interest statement

The authors declare no competing interests.

Figures

Figure 1
Figure 1
The UAF’s approximations of the following activation functions: (a) step, (b) sigmoid, (c) tanh, (d) ReLU, (e) LeakyReLU, and (f) Gaussian. The black solid lines represent the UAF, while the green dashed lines represent the targeted activation functions, whose values can be obtained from the y axis on the left. The red solid lines represent the error E between the UAF and targeted activation function and the values can be read from the y axis on the right side.
Figure 2
Figure 2
The UAF evolution of the following datasets: (a) CIFAR-10 image classification, (b) CORA publication classification, (c) 9 gas concentration quantification, (d) ZINC molecular solubility quantification, and (e) BipedalWalker-V2 reinforcement learning.
See this image and copyright information in PMC

References

    1. He, X., Zhao, K. & Chu, X. AutoML: A survey of the state-of-the-art. arXiv:1908.00709 (2019).
    1. Floreano D, Dürr P, Mattiussi C. Neuroevolution: From architectures to learning. Evol. Intell. 2008;1(1):47–62. doi: 10.1007/s12065-007-0002-4. - DOI
    1. Yao, Q. et al. Taking human out of learning applications: A survey on automated machine learning. arXiv:1810.13306 (2018).
    1. Stanley KO, Miikkulainen R. Evolving neural networks through augmenting topologies. Evol. Comput. 2002;10(2):99–127. doi: 10.1162/106365602320169811. - DOI - PubMed
    1. Stanley KO, D’Ambrosio DB, Gauci J. A hypercube-based encoding for evolving large-scale neural networks. Artif. Life. 2009;15(2):185–212. doi: 10.1162/artl.2009.15.2.15202. - DOI - PubMed

Publication types