Sampling-free variational inference of Bayesian neural networks by variance backpropagation
We propose a new Bayesian Neural Net formulation that affords variational inference for which the evidence lower bound is analytically tractable subject to a tight approximation. We achieve this tractability by (i) decomposing ReLU nonlinearities into the product of an identity and a Heaviside step...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article (Journal) Chapter/Article |
| Language: | English |
| Published: |
2018
|
| In: |
Arxiv
Year: 2018, Pages: 1-15 |
| DOI: | 10.48550/arXiv.1805.07654 |
| Online Access: | Verlag, lizenzpflichtig, Volltext: https://doi.org/10.48550/arXiv.1805.07654 Verlag, lizenzpflichtig, Volltext: http://arxiv.org/abs/1805.07654 
|
| Author Notes: | Manuel Haußmann, Fred A. Hamprecht, Melih Kandemir |
| Summary: | We propose a new Bayesian Neural Net formulation that affords variational inference for which the evidence lower bound is analytically tractable subject to a tight approximation. We achieve this tractability by (i) decomposing ReLU nonlinearities into the product of an identity and a Heaviside step function, (ii) introducing a separate path that decomposes the neural net expectation from its variance. We demonstrate formally that introducing separate latent binary variables to the activations allows representing the neural network likelihood as a chain of linear operations. Performing variational inference on this construction enables a sampling-free computation of the evidence lower bound which is a more effective approximation than the widely applied Monte Carlo sampling and CLT related techniques. We evaluate the model on a range of regression and classification tasks against BNN inference alternatives, showing competitive or improved performance over the current state-of-the-art. |
|---|---|
| Item Description: | Last revised 12 Jun 2019 Gesehen am 13.07.2022 |
| Physical Description: | Online Resource |
| DOI: | 10.48550/arXiv.1805.07654 |