Séminaire Borelli « Understanding Priors in Bayesian Neural Networks at the Unit Level » par Julyan Arbel  (INRIA Grenoble)

Date : 13 mars 2020
Lieu : Campus de Cachan, bâtiment d’Alembert, salle Condorcet

Nous nous retrouverons exceptionnellement avant le séminaire à partir de 12h30 au premier étage du bâtiment Laplace autour d’un buffet de sandwiches et salades. Merci de vous inscrire, ainsi que vos invité.e.s éventuel.le.s AVANT le mercredi 11 mars à midi.

We investigate deep Bayesian neural networks with Gaussian weight priors and a class of ReLU-like nonlinearities. Bayesian neural networks with Gaussian priors are well known to induce an L2 -« weight decay »- regularization.
Our results characterize a more intricate regularization effect at the level of the unit activations.
Our main result establishes that the induced prior distribution on the units before and after activation becomes increasingly heavy-tailed with the depth of the layer. We show that first layer units are Gaussian, second layer units are sub-exponential, and units in deeper layers are characterized by sub-Weibull distributions. Our results provide new theoretical insight on deep Bayesian neural networks, which we corroborate with simulation experiments.
Joint work with Mariia Vladimirova, Jakob Verbeek and Pablo Mesejo