Statistics and Data Science Seminar: Edward Milsom (University of Bath)

The successes of modern deep neural networks (DNNs) are founded on their ability to transform inputs across multiple layers to build good high-level representations. It is therefore critical to understand this process of representation learning. However, standard theoretical approaches involving infinite width limits give very limited insights into representation learning. For instance, the NNGP infinite-width limit entirely eliminates representation learning. Alternatively, mu-P just tells us whether or not representation learning is possible, without telling us anything about the representations that are actually learned. We therefore develop a new infinite width limit, the Bayesian representation learning limit, that exhibits representation learning mirroring that in finite-width networks, yet at the same time, remains extremely tractable. This limit gives rise to an elegant objective that describes how learning shapes representations at every layer. Using this objective, we develop a new, scalable family of "deep kernel methods", which are based on an infinite-width limit of deep Gaussian processes. In practice, deep kernel methods just use kernels without ever using any features or weights. We develop a convolutional variant, known as Convolutional Deep Kernel Machines, and push their performance to 94.5% on CIFAR-10, which is on-par with neural networks with a similar architecture (the previous SOTA for kernel methods was 91.2%, from Adlam et al. 2023).