Deep Neural Network Structures Solving Variational Inequalities.
作者： Patrick L. Combettes , Jean-Christophe Pesquet
DOI:
关键词:
摘要: Motivated by structures that appear in deep neural networks, we investigate nonlinear composite models alternating proximity and affine operators defined on different spaces. We first show a wide range of activation used networks are actually operators. then establish conditions for the averagedness proposed constructs their asymptotic properties. It is shown limit resulting process solves variational inequality which, general, does not derive from minimization problem.
arxiv.org参考文章(38)
John S. Bridle, Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition NATO Neurocomputing. pp. 227- 236 ,(1990) , 10.1007/978-3-642-76153-9_28
P. L. Combettes, Construction d'un point fixe commun à une famille de contractions fermes Comptes rendus de l'Académie des sciences. Série 1, Mathématique. ,vol. 320, pp. 1385- 1390 ,(1995)
Klaus Greff, Rupesh Kumar Srivastava, Jürgen Schmidhuber, Training very deep networks neural information processing systems. ,vol. 28, pp. 2377- 2385 ,(2015)
Geoffrey E. Hinton, Vinod Nair, Rectified Linear Units Improve Restricted Boltzmann Machines international conference on machine learning. pp. 807- 814 ,(2010)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification international conference on computer vision. pp. 1026- 1034 ,(2015) , 10.1109/ICCV.2015.123
Patrick L. Combettes, Jean-Christophe Pesquet, Proximal Thresholding Algorithm for Minimization over Orthonormal Bases Siam Journal on Optimization. ,vol. 18, pp. 1351- 1376 ,(2007) , 10.1137/060669498
Ken-Ichi Funahashi, On the approximate realization of continuous mappings by neural networks Neural Networks. ,vol. 2, pp. 183- 192 ,(1989) , 10.1016/0893-6080(89)90003-8
J.-B. Baillon, P.L. Combettes, R. Cominetti, There is no variational characterization of the cycles in the method of periodic projections Journal of Functional Analysis. ,vol. 262, pp. 400- 408 ,(2012) , 10.1016/J.JFA.2011.09.002
Radu Ioan Boţ, Ernö Robert Csetnek, A Dynamical System Associated with the Fixed Points Set of a Nonexpansive Operator Journal of Dynamics and Differential Equations. ,vol. 29, pp. 155- 168 ,(2017) , 10.1007/S10884-015-9438-X
Warren S. McCulloch, Walter Pitts, A logical calculus of the ideas immanent in nervous activity Bulletin of Mathematical Biology. ,vol. 52, pp. 99- 115 ,(1990) , 10.1007/BF02478259