Adaptive-Tanh ReLU (AHReLU): A Novel Activation Function for Enhanced Convolutional Neural Network Performance

Abstract

 In this work, we propose a novel activation function, AHReLU, which demonstrates superior performance compared to traditional activa tion functions such as ReLU, GELU, and Mish. Replacing ReLU with AHReLU in the VGG-16 model improved Top-1 classification accuracy by 0.79%,reaching97.57%. In the CIFAR-100dataset, AHReLUoutper formed ReLU by 0.32%, achieving a Top-1 accuracy of 59.82%. For theSVHN dataset, AHReLU achieved a mean accuracy of 95.38%, slightly surpassing ReLU’s performance of 95.36%. In machine translation tasks, specifically on the WMT 2014 dataset, AHReLU achieved a BLEU score of 27.5, which is 0.2 points higher than ReLU and 1.2 points higher than GELU. These results highlight that AHReLU, with its learnable parame ters, outperforms traditional activation functions, leading to better model performance across various datasets and tasks. The introduction of learn able parameters into the activation function is key to the observed im provements, making AHReLU a promising candidate to replace widely used activation functions such as ReLU, GELU, and Mish in deep learn
ing models.