BIMSA >
BIMSA Optimization Seminar
The Elusive Power of Local Training in Federated Learning: 10 Years of Research and a Resolution of the Mystery
The Elusive Power of Local Training in Federated Learning: 10 Years of Research and a Resolution of the Mystery
Organizer
Speaker
Time
Friday, November 22, 2024 2:00 PM - 3:00 PM
Venue
Shuangqing-C548
Online
Zoom 388 528 9728
(BIMSA)
Abstract
I will outline the history of the theoretical development of the local training “trick” employed in virtually all successful federated learning algorithms. In particular, I will identify five distinct generations of methods and results: 1) heuristic, 2) homogeneous, 3) sublinear, 4) linear, and 5) accelerated. The 5th generation, initiated by the ProxSkip algorithm by Mishchenko et al (ICML 2022), finally led to the proof that local training, if carefully executed, leads to provable acceleration of communication complexity, without requiring any data homogeneity assumptions. Because these latest advances are very new, there are many opportunities to develop the 5th generation of local training methods further. I will give a brief overview of what we know now, and what problems still remain open.
Speaker Intro
Peter Richtárik is a professor of Computer Science at the King Abdullah University of Science and Technology (KAUST), Saudi Arabia, where he leads the Optimization and Machine Learning Lab. His research interests lie at the intersection of mathematics, computer science, machine learning, optimization, numerical linear algebra, and high-performance computing. Through his work on randomized and distributed optimization algorithms, he has contributed to the foundations of machine learning, optimization, and randomized numerical linear algebra. He is one of the original developers of Federated Learning. Prof. Richtárik’s works attracted international awards, including the Charles Broyden Prize, SIAM SIGEST Best Paper Award, Distinguished Speaker Award at the 2019 International Conference on Continuous Optimization, the IMA Leslie Fox Prize (three times), and a Best Paper Award at the NeurIPS 2020 Workshop on Scalability, Privacy, and Security in Federated Learning. Several of his works are among the most read papers published by the SIAM Journal on Optimization and the SIAM Journal on Matrix Analysis and Applications. Prof. Richtárik serves as an Area Chair for leading machine learning conferences, including NeurIPS, ICML, and ICLR, and is an Action Editor of JMLR, and Associate Editor of Numerische Mathematik and Optimization Methods and Software. In the past, he served as an Action Editor of TMLR and an Area Editor of JOTA.