BIMSA >
BIMSA Computational Math Seminar
Efficient, Accurate and Stable Gradients for Neural Differential Equations
Efficient, Accurate and Stable Gradients for Neural Differential Equations
演讲者
James Foster
时间
2025年10月09日 15:15 至 16:15
地点
Online
线上
Zoom 928 682 9093
(BIMSA)
摘要
Neural differential equations (NDEs) sit at the intersection of two dominant modelling paradigms – neural networks and differential equations. One of their features is that they can be trained with a small memory footprint through adjoint equations. This can be helpful in high-dimensional applications since the memory usage of standard backpropagation scales linearly with depth (or, in the NDE case, the number of steps taken by the solver).
However, adjoint equations have seen little use in practice as the resulting gradients are often inaccurate. Fortunately, there has emerged a class of numerical methods which allow NDEs to be trained using gradients that are both accurate and memory efficient. These solvers are known as "algebraically reversible" and produce numerical solutions which can be reconstructed backwards in time.
Whilst algebraically reversible solvers have seen some success in large-scale applications, they are known to have stability issues. In this talk, we propose a methodology for constructing reversible NDE solvers from non-reversible ones. We show that the resulting reversible solvers converge in the ODE setting, can achieve high order convergence, and even have stability regions. We then present a few examples demonstrating the memory efficiency of our approach.
If time allows, we will also discuss some related and ongoing work into Deep Equilibrium Models (DEQs). Here, we leverage similar ideas to construct an algebraically reversible solver for fixed point systems. Just as in the NDE case, using a reversible solver allows us to train DEQs with accurate and memory-efficient gradients.
Joint work with Sam McCallum and Kamran Arora.
However, adjoint equations have seen little use in practice as the resulting gradients are often inaccurate. Fortunately, there has emerged a class of numerical methods which allow NDEs to be trained using gradients that are both accurate and memory efficient. These solvers are known as "algebraically reversible" and produce numerical solutions which can be reconstructed backwards in time.
Whilst algebraically reversible solvers have seen some success in large-scale applications, they are known to have stability issues. In this talk, we propose a methodology for constructing reversible NDE solvers from non-reversible ones. We show that the resulting reversible solvers converge in the ODE setting, can achieve high order convergence, and even have stability regions. We then present a few examples demonstrating the memory efficiency of our approach.
If time allows, we will also discuss some related and ongoing work into Deep Equilibrium Models (DEQs). Here, we leverage similar ideas to construct an algebraically reversible solver for fixed point systems. Just as in the NDE case, using a reversible solver allows us to train DEQs with accurate and memory-efficient gradients.
Joint work with Sam McCallum and Kamran Arora.