A Dynamical System Perspective for Escaping Sharp Local Minima in Equality Constrained Optimization Problems
作者： Javad Lavaei , Han Feng , Haixiang Zhang
DOI: 10.1109/CDC42340.2020.9303907
关键词:
摘要: This paper provides a dynamical system perspective on the escape of sharp local minima in constrained optimization problems. The view models perturbed projected first-order algorithm and translates problem escaping problems to that regions attraction corresponding system. We develop notion biased perturbation show it gives quantitative small can be escaped. As counterpart, we explain why dynamics is stable wide region around strongly equilibrium. Numerical examples are provided illustrate usefulness developed concepts.
sci-hub.st 参考文章(18)
P. Massart, B. Laurent, Adaptive estimation of a quadratic functional by model selection Annals of Statistics. ,vol. 28, pp. 1302- 1338 ,(2000) , 10.1214/AOS/1015957395
Arthur Sard, The measure of the critical values of differentiable maps Bulletin of the American Mathematical Society. ,vol. 48, pp. 883- 890 ,(1942) , 10.1090/S0002-9904-1942-07811-6
J. Schropp, I. Singer, A dynamical systems approach to constrained minimization Numerical Functional Analysis and Optimization. ,vol. 21, pp. 537- 551 ,(2000) , 10.1080/01630560008816971
Revised January, Ernesto P. Adorio, MVF - Multivariate Test Functions Library in C for Unconstrained Global Optimization ,(2005)
Jason D Lee, Max Simchowitz, Michael I Jordan, Benjamin Recht, None, Gradient Descent Only Converges to Minimizers conference on learning theory. ,vol. 49, pp. 1246- 1257 ,(2016)
Stephen Boyd, Emmanuel J. Candès, Weijie Su, A differential equation for modeling Nesterov's accelerated gradient method: theory and insights Journal of Machine Learning Research. ,vol. 17, pp. 5312- 5354 ,(2016)
Dimitri Bertsekas, Nonlinear Programming ,(1995)
Ross Drummond, Stephen Duncan, Accelerated Gradient Methods with Memory arXiv: Optimization and Control. ,(2018)
Bin Shi, Simon S Du, Michael I Jordan, Weijie J Su, None, Understanding the Acceleration Phenomenon via High-Resolution Differential Equations arXiv: Optimization and Control. ,(2018)
Umut Simsekli, Mert Gurbuzbalaban, Levent Sagun, A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks arXiv: Learning. ,(2019)