Averaged Least-Mean-Squares: Bias-Variance Trade-offs and Optimal Sampling Distributions

摘要: We consider the least-squares regression problem and provide a detailed asymptotic analysis of performance averaged constant-step-size stochastic gradient descent. In strongly-convex case, we an expansion up to explicit exponentially decaying terms. Our leads new insights into approximation algorithms: (a) it gives tighter bound on allowed step-size; (b) generalization error may be divided variance term which is as O(1/n), independently step-size , bias that decays O(1/ 2 n ); (c) when allowing non-uniform sampling examples over dataset, choice good density depends trade-off between variance: dominates, optimal densities do not lead much gain, while can choose larger step-sizes significant improvements.