Proactive DP: A Multple Target Optimization Framework for DP-SGD

摘要: We introduce a multiple target optimization framework for DP-SGD referred to as pro-active DP. In contrast to traditional DP accountants, which are used to track the expenditure of privacy budgets, the pro-active DP scheme allows one to {\it a-priori} select parameters of DP-SGD based on a fixed privacy budget (in terms of and ) in such a way to optimize the anticipated utility (test accuracy) the most. To achieve this objective, we first propose significant improvements to the moment account method, presenting a closed-form -DP guarantee that connects all parameters in the DP-SGD setup. Generally, DP-SGD is -DP if with at least and , where is the total number of rounds, and is the total number of gradient computations where measures in number of epochs of size of the local data set. We prove that our expression is close to tight in that if is more than a constant factor smaller than the lower bound , then the -DP guarantee is violated. Our enhanced DP theory allows us to create a utility graph and DP calculator. These tools link privacy and utility objectives and search for optimal experiment setups, efficiently taking into account both accuracy and privacy objectives, as well as implementation goals. We furnish a comprehensive implementation flow of our proactive DP, with rigorous experiments to showcase the proof-of-concept.