In my opinion, model selection is the most appealing capability of Bayesian inference, which has the most to offer in deep learning. However, performing Bayesian model selection requires accurate approximate inference. In the first part of the talk, I will discuss accurate inference in the fundamental building block of deep neural networks: a single layer. Specifically, I will focus on the Gaussian process (GP) representation of neural network layers, and present some recent work on inducing point and conjugate gradient approximations, while paying close attention to the question of what we should expect from methods that we consider “good” or even “exact”. In the second part of the talk, I will discuss how these techniques can be of use in model selection in deep learning, with examples on learning invariances. I will close off with some thoughts on how these ideas may develop in the future.
This talk was given as part of a series where alumni of the group where I did my PhD were invited to return to discuss their research. This is one of those talks with a high-level overview of recent work, together with some opinions on how they relate to larger questions in the field.