Analysis of function of rectified linear unit used in deep learning
作者： Kazuyuki Hara , Daisuke Saito , Hayaru Shouno
DOI: 10.1109/IJCNN.2015.7280578
关键词: Active learning (machine learning) 、 Artificial neural network 、 Unsupervised learning 、 Semi-supervised learning 、 Competitive learning 、 Artificial intelligence 、 Machine learning 、 Computational learning theory 、 Deep belief network 、 Deep learning 、 Instance-based learning 、 Stability (learning theory) 、 Rectifier (neural networks) 、 Computer science 、 Wake-sleep algorithm 、 Online machine learning 、 Generalization error
摘要: … To avoid this difficulty, a rectified linear unit (ReLU) is proposed to speed up the learning convergence. However, the reasons the convergence is speeded up are not well understood. In …
sci-hub.se 
researchgate.net 参考文章(10)
Kazuyuki Hara, Kentaro Katahira, Kazuo Okanoya, Masato Okada, Theoretical Analysis of Function of Derivative Term in On-Line Gradient Descent Learning Artificial Neural Networks and Machine Learning – ICANN 2012. pp. 9- 16 ,(2012) , 10.1007/978-3-642-33266-1_2
Scott E. Fahlman, An empirical study of learning speed in back-propagation networks ,(1988)
Shun-ichi Amari, Natural gradient works efficiently in learning Neural Computation. ,vol. 10, pp. 177- 202 ,(1998) , 10.1162/089976698300017746
David Saad, Sara A. Solla, On-line learning in soft committee machines Physical Review E. ,vol. 52, pp. 4225- 4243 ,(1995) , 10.1103/PHYSREVE.52.4225
Kenji Fukumizu, A regularity condition of the information matrix of a multilayer perceptron network Neural Networks. ,vol. 9, pp. 871- 879 ,(1996) , 10.1016/0893-6080(95)00119-0
M.D. Zeiler, M. Ranzato, R. Monga, M. Mao, K. Yang, Q.V. Le, P. Nguyen, A. Senior, V. Vanhoucke, J. Dean, G.E. Hinton, On rectified linear units for speech processing international conference on acoustics, speech, and signal processing. pp. 3517- 3521 ,(2013) , 10.1109/ICASSP.2013.6638312
M Biehl, H Schwarze, Learning by on-line gradient descent Journal of Physics A. ,vol. 28, pp. 643- 656 ,(1995) , 10.1088/0305-4470/28/3/018
Kazuyuki Hara, Kentaro Katahira, Theoretical Analysis of Learning Speed in Gradient Descent Algorithm Replacing Derivative with Constant Ipsj Online Transactions. ,vol. 7, pp. 14- 19 ,(2014) , 10.2197/IPSJTRANS.7.14
Geoffrey E. Hinton, Simon Osindero, Yee-Whye Teh, A fast learning algorithm for deep belief nets Neural Computation. ,vol. 18, pp. 1527- 1554 ,(2006) , 10.1162/NECO.2006.18.7.1527
Yoshua Bengio, Xavier Glorot, Antoine Bordes, Deep Sparse Rectifier Neural Networks international conference on artificial intelligence and statistics. ,vol. 15, pp. 315- 323 ,(2011)