摘要
作为深度学习算法中重要的环节,激活函数可以为神经网络引入非线性因素。大量学者通过提出或改进激活函数的方法在一定程度上提高了算法的优化及泛化能力。研究了现阶段的激活函数,将激活函数大致分为S系激活函数和ReLU系激活函数,从不同激活函数的功能特点和存在的饱和性、零点对称和梯度消失及梯度爆炸的现象进行研究分析,针对Sigmoid,Tanh,ReL,P-ReLU,L-ReLU等典型激活函数分别应用在卷积神经网络(Covolutional Neural Network,CNN)和循环神经网络(Recurrent Neural Network,RNN)中测试。在CNN中使用MNIST,CIFAR-10经典数据集测试不同激活函数,并在RNN中使用大豆粮油数据集对大豆的产值进行预警,通过结果得到S系激活函数比ReLU系激活函数收敛更快,而ReLU系激活函数则在精度上优于S系激活函数,其中P-ReLU在大豆产值预测中达到93%的最高精度。
Since Deep Learning algorithm has attracted widespread attention,academia has made great efforts to improve algorithm’s optimization performance.As an important part of deep learning algorithm,activation function introduces non-linear factors to neural networks.A lot of authors have,to some extent,improved optimization and generalization of the algorithm by proposing or updating activation function methods.This article roughly divides activation functions into S-system activation function and ReLU-system activation function after a thorough research.Starting with researching and analyzing functional characteristics of different activation functions,such as the existence of saturation,zero symmetry,gradient disappearance and gradient explosion,the article focuses on the typical activation functions such as Sigmoid,Tanh,ReLU and P-ReLU,and their respective test results in Convolutional Neural Network(CNN)and Recurrent Neural Network(RNN).Classic data sets like MNIST and CIFAR-10 in CNN are used to test different activation functions.Soybean data set are used in RNN to give an early warning to the output value,which shows that the S-system activation function converges faster than the ReLU-system activation function while ReLU-system has an edge in accuracy,P-ReLU achieved the highest accuracy of 93%in soybean yield prediction.
作者
张有健
陈晨
王再见
ZHANG Youjian;CHEN Chen;WANG Zaijian(The School of Physics and Electronic Information,Anhui Normal University,Wuhu 241000,China;Anhui Provincial Engineering Laboratory on Information Fusion and Control for Intelligent Robot,Wuhu 241000,China)
出处
《无线电通信技术》
2021年第1期115-120,共6页
Radio Communications Technology
基金
粮食信息处理与控制教育部重点实验室开放基金(KFJJ-2018-205)。
