The Physical Systems Behind Optimization Algorithms

摘要: We use differential equations based approaches to provide some {\it \textbf{physics}} insights into analyzing the dynamics of popular optimization algorithms in machine learning. In particular, we study gradient descent, proximal coordinate gradient, and Newton's methods as well their Nesterov's accelerated variants a unified framework motivated by natural connection physical systems. Our analysis is applicable more general problems \textbf{beyond}} convexity strong convexity, e.g. Polyak-\L ojasiewicz error bound conditions (possibly nonconvex).