Deep Learning Dynamics: A Scientific Approach
Organizer
Speaker
Zeke Xie
Time
Friday, June 28, 2024 10:00 AM - 11:30 AM
Venue
A3-4-301
Online
Zoom 293 812 9202
(BIMSA)
Abstract
In this talk, I will introduce a series of my works on understanding and improving deep learning via scientific principles and methodology. The success of deep learning depends on both neural networks and optimization dynamcis. I will visit several very foundamental issues in deep learning dynamics: (1) SGD Dynamics and how it selects flat minima; (2) Adam dynamics and how it explains the power of Adam; (3) improving deep learning from a optimization dynamical perspetive; (4) the overlooked pitfalls of weight decay and how to mitigate them; (5) a bridge between protein dynamics and deep learning dynamics. Through this talk, we will also see that scientific principles and theories can provide useful insights and tools for understanding and improving deep learning.
Speaker Intro
Dr. Zeke Xie is an Assistant Professor at Information Hub, Hong Kong University of Science and Technology (Guangzhou). He is leading Xie Machine Learning Foundations Lab (xLeaF Lab) that generally interested in understanding and solving fundamental issues of modern AI, particularly large models, by scientific principles and methodology. He currently focuses on optimization and inference of Large Models and Generative AI. Previously, he was a researcher at Baidu Research responsible for large models and AIGC research. He obtained Ph.D. and M.E. both from The University of Tokyo. He received multiple faculty research awards from the industry, including ByteDance and Baidu.