Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization

作者: H. Vincent Poor , Gauri Joshi , Jianyu Wang , Qinghua Liu , Hao Liang

DOI:

关键词:

摘要: In federated optimization, heterogeneity in the clients' local datasets and computation speeds results large variations number of updates performed by each client communication round. Naive weighted aggregation such models causes objective inconsistency, that is, global model converges to a stationary point mismatched function which can be arbitrarily different from true objective. This paper provides general framework analyze convergence heterogeneous optimization algorithms. It subsumes previously proposed methods as FedAvg FedProx first principled understanding solution bias slowdown due inconsistency. Using insights this analysis, we propose FedNova, normalized averaging method eliminates inconsistency while preserving fast error convergence.

参考文章(54)
Hao Yu, Sen Yang, Shenghuo Zhu, Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning national conference on artificial intelligence. ,vol. 33, pp. 5693- 5700 ,(2019) , 10.1609/AAAI.V33I01.33015693
Ananda Theertha Suresh, Mehryar Mohri, Gary Sivek, Agnostic Federated Learning arXiv: Learning. ,(2019)
Indranil Gupta, Cong Xie, Sanmi Koyejo, Asynchronous Federated Optimization arXiv: Distributed, Parallel, and Cluster Computing. ,(2019)
Guojing Cong, Fan Zhou, A Distributed Hierarchical SGD Algorithm with Sparse Global Reduction. arXiv: Learning. ,(2019)
Rong Jin, Hao Yu, Sen Yang, On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization. international conference on machine learning. pp. 7184- 7193 ,(2019)
Tian Li, Maziar Sanjabi, Ahmad Beirami, Virginia Smith, None, Fair Resource Allocation in Federated Learning arXiv: Learning. ,(2019)
Farzin Haddadpour, Viveck R. Cadambe, Mehrdad Mahdavi, Mohammad Mahdi Kamani, Trading Redundancy for Communication: Speeding up Distributed SGD for Non-convex Optimization international conference on machine learning. pp. 2545- 2554 ,(2019)
Angelia Nedic, Alex Olshevsky, Michael G. Rabbat, Network Topology and Communication-Computation Tradeoffs in Decentralized Optimization Proceedings of the IEEE. ,vol. 106, pp. 953- 976 ,(2018) , 10.1109/JPROC.2018.2817461
Sebastian Urban Stich, Local SGD Converges Fast and Communicates Little international conference on learning representations. ,(2018)