Gaussian processes in machine learning
Machine learning considers many models. Some are interpretable, others are probabilistic, and others are used in practice. Gaussian process-based models have all these properties: they are interpretable, probabilistic, and lead to practical solutions. The history of applications of Gaussian process regression in machine learning lasts for more than 60 years, while some recent works revolutionized the scope of practical applications of it and theoretical tools to answer fundamental questions. Applications include helping train the models with superhuman powers in the Go game and reducing the cost of aircraft construction by 10%, saving millions of dollars in both cases.
I plan to achieve two main goals in this course: provide an interesting researcher with a powerful tool named Gaussian process-based machine learning models and equip him with a theoretical understanding of the success of Gaussian process regression. In particular, we'll find out how this approach solves the regression problem interpretably, how to estimate the uncertainty in a principled way, and how to select the optimal design of experiments. The lectures are based on my 10 years of experience in this field.
I plan to achieve two main goals in this course: provide an interesting researcher with a powerful tool named Gaussian process-based machine learning models and equip him with a theoretical understanding of the success of Gaussian process regression. In particular, we'll find out how this approach solves the regression problem interpretably, how to estimate the uncertainty in a principled way, and how to select the optimal design of experiments. The lectures are based on my 10 years of experience in this field.
Lecturer
Alexey Zaytsev
Date
21st February ~ 5th March, 2023
Location
| Weekday | Time | Venue | Online | ID | Password |
|---|---|---|---|---|---|
| Tuesday,Wednesday,Friday | 15:20 - 16:55 | 1120 | ZOOM 02 | 518 868 7656 | BIMSA |
Prerequisite
Probability theory, Mathematical statistics, Machine learning
Syllabus
Lecture 1. Gaussian process regression (GPR). Covariance functions. Prediction and uncertainty estimation for GPR
Lecture 2. Large-scale Gaussian process regression. How to deal with large amounts of training data.
Lecture 3. Interpolation error bounds. Minimax interpolation error bound. Optimal variable fidelity design of experiments. Sketch of the proof.
Lecture 4. Surrogate optimization and optimal adaptive design of experiments.
Lecture 5. Upper and lower bounds for regret in optimal adaptive design of experiments. Sketch of the proof.
Lecture 6. Deep Gaussian process regression. How to unite Gaussian process regression and deep learning
Lecture 2. Large-scale Gaussian process regression. How to deal with large amounts of training data.
Lecture 3. Interpolation error bounds. Minimax interpolation error bound. Optimal variable fidelity design of experiments. Sketch of the proof.
Lecture 4. Surrogate optimization and optimal adaptive design of experiments.
Lecture 5. Upper and lower bounds for regret in optimal adaptive design of experiments. Sketch of the proof.
Lecture 6. Deep Gaussian process regression. How to unite Gaussian process regression and deep learning
Reference
The main books for the initial part of the course are:
1A. C. Rasmussen “Gaussian processes for Machine learning”, 2005.
1B. Garnett, Roman, Bayesian Optimization, 2023, Cambridge University Press, to appear
However, the field significantly changed during the 17 years that passed since then. Below you’ll find relevant articles for each lecture, there will be more considered:
2. Liu, Haitao, et al. "When Gaussian process meets big data: A review of scalable GPs." IEEE transactions on neural networks and learning systems 31.11 (2020): 4405-4423.
3. Zaytsev, Alexey, and Evgeny Burnaev. "Minimax approach to variable fidelity data interpolation." Artificial Intelligence and Statistics. PMLR, 2017.
4. Koziel, Slawomir, and Leifur Leifsson. Surrogate-based modeling and optimization. New York: Springer, 2013.
5. Srinivas, Niranjan, et al. "Gaussian process optimization in the bandit setting: No regret and experimental design." arXiv preprint arXiv:0912.3995 (2009).
6. Wilson, Andrew Gordon, et al. "Deep kernel learning." Artificial intelligence and statistics. PMLR, 2016.
1A. C. Rasmussen “Gaussian processes for Machine learning”, 2005.
1B. Garnett, Roman, Bayesian Optimization, 2023, Cambridge University Press, to appear
However, the field significantly changed during the 17 years that passed since then. Below you’ll find relevant articles for each lecture, there will be more considered:
2. Liu, Haitao, et al. "When Gaussian process meets big data: A review of scalable GPs." IEEE transactions on neural networks and learning systems 31.11 (2020): 4405-4423.
3. Zaytsev, Alexey, and Evgeny Burnaev. "Minimax approach to variable fidelity data interpolation." Artificial Intelligence and Statistics. PMLR, 2017.
4. Koziel, Slawomir, and Leifur Leifsson. Surrogate-based modeling and optimization. New York: Springer, 2013.
5. Srinivas, Niranjan, et al. "Gaussian process optimization in the bandit setting: No regret and experimental design." arXiv preprint arXiv:0912.3995 (2009).
6. Wilson, Andrew Gordon, et al. "Deep kernel learning." Artificial intelligence and statistics. PMLR, 2016.
Audience
Graduate
Video Public
Yes
Notes Public
Yes
Language
English
Lecturer Intro
Alexey has deep expertise in machine learning and processing of sequential data. He publishes at top venues, including KDD, ACM Multimedia and AISTATS. Industrial applications of his results are now in service at companies Airbus, Porsche and Saudi Aramco among others.