Deep Learning Dynamics: A Scientific Approach
Organizer
Speaker
Zeke Xie
Time
Friday, June 28, 2024 10:00 AM - 11:30 AM
Venue
A3-4-301
Online
Zoom 293 812 9202
(BIMSA)
Abstract
In this talk, I will introduce a series of my works on understanding and improving deep learning via scientific principles and methodology. The success of deep learning depends on both neural networks and optimization dynamcis. I will visit several very foundamental issues in deep learning dynamics: (1) SGD Dynamics and how it selects flat minima; (2) Adam dynamics and how it explains the power of Adam; (3) improving deep learning from a optimization dynamical perspetive; (4) the overlooked pitfalls of weight decay and how to mitigate them; (5) a bridge between protein dynamics and deep learning dynamics. Through this talk, we will also see that scientific principles and theories can provide useful insights and tools for understanding and improving deep learning.
Speaker Intro
I am an Assistant Professor jointly appointed by DSA Thrust and AI Thrust, Hong Kong University of Science and Technology (Guangzhou). I will also be an affiliated Assistant Professor at Dept. of CSE, Hong Kong University of Science and Technology (Clear Water Bay).
Previously, I was a researcher at Baidu Research responsible for large models and AIGC research. I obtained Ph.D. and M.E. both from The University of Tokyo, respectively. I was fortunate to be advised by Prof. Issei Sato and Prof. Masashi Sugiyama. I was also affiliated with RIKEN AIP during the Ph.D. study. Before that, I obtained Bachelor of Science from University of Science and Technology of China.
My mission is to find a way towards the science of AI. I am generally interested in understanding and solving fundamental issues of modern AI by scientific principles and methodology. My current research projects focus on foundations of machine learning and generative AI.
Previously, I was a researcher at Baidu Research responsible for large models and AIGC research. I obtained Ph.D. and M.E. both from The University of Tokyo, respectively. I was fortunate to be advised by Prof. Issei Sato and Prof. Masashi Sugiyama. I was also affiliated with RIKEN AIP during the Ph.D. study. Before that, I obtained Bachelor of Science from University of Science and Technology of China.
My mission is to find a way towards the science of AI. I am generally interested in understanding and solving fundamental issues of modern AI by scientific principles and methodology. My current research projects focus on foundations of machine learning and generative AI.