Stochastic Variance Reduced Gradient Methods Using a Trust-Region-Like Scheme

摘要: Stochastic variance reduced gradient (SVRG) methods are important approaches to minimize the average of a large number cost functions frequently arising in machine learning and many other applications. In this paper, based on SVRG, we propose SVRG-TR method which employs trust-region-like scheme for selecting stepsizes. It is proved that linearly convergent expectation smooth strongly convex enjoys faster convergence rate than SVRG methods. order overcome difficulty tuning stepsizes by hand, combine Barzilai–Borwein (BB) automatically compute method, named as SVRG-TR-BB method. By incorporating mini-batching with SVRG-TR-BB, respectively, further two extended mSVRG-TR mSVRG-TR-BB. Linear complexity given. Numerical experiments some standard datasets show generally better or comparable best-tuned modern stochastic methods, while mSVRG-TR-BB very competitive mini-batch variants recent successful