Performance Modeling and Scalability Optimization of Distributed Deep Learning Systems

摘要: Big deep neural network (DNN) models trained on large amounts of data have recently achieved the best accuracy hard tasks, such as image and speech recognition. Training these DNNs using a cluster commodity machines is promising approach since training time consuming compute-intensive. To enable extremely DNNs, are partitioned across machines. expedite very sets, multiple model replicas in parallel different subsets examples with global parameter server maintaining shared weights replicas. The correct choice for partitioning overall system provisioning highly dependent DNN distributed hardware characteristics. These decisions currently require significant domain expertise empirical state space exploration.This paper develops performance that quantify impact scalability. Also, we use to build scalability optimizer efficiently determines optimal configuration minimizes time. We evaluate our state-of-the-art framework two benchmark applications. results show estimate high estimation correctly chooses configurations, minimizing DNNs.