We examine the problem of controlling divergences for latent space regularisation in variational autoencoders. Specifically, when aiming to reconstruct example x 2 Rm via latent space z 2 Rn (n = m), while balancing this against the need for generalisable latent representations. We present a regularisation mechanism based on the skew-geometric Jensen-Shannon divergence (JSGa). We find a variation in JSGa, motivated by limiting cases, which leads to an intuitive interpolation between forward and reverse KL in the space of both distributions and divergences. We motivate its potential benefits for VAEs through low-dimensional examples, before presenting quantitative and qualitative results. Our experiments demonstrate that skewing our variant of JSGa, in the context of JSGa-VAEs, leads to better reconstruction and generation when compared to several baseline VAEs. Our approach is entirely unsupervised and utilises only one hyperparameter which can be easily interpreted in latent space.

Constraining variational inference with geometric Jensen-Shannon divergence / Deasy, J.; Simidjievski, N.; Lio, P.. - 2020-:(2020). (Intervento presentato al convegno 34th Conference on Neural Information Processing Systems, NeurIPS 2020 tenutosi a Virtual, Online).

Constraining variational inference with geometric Jensen-Shannon divergence

Lio P.
2020

Abstract

We examine the problem of controlling divergences for latent space regularisation in variational autoencoders. Specifically, when aiming to reconstruct example x 2 Rm via latent space z 2 Rn (n = m), while balancing this against the need for generalisable latent representations. We present a regularisation mechanism based on the skew-geometric Jensen-Shannon divergence (JSGa). We find a variation in JSGa, motivated by limiting cases, which leads to an intuitive interpolation between forward and reverse KL in the space of both distributions and divergences. We motivate its potential benefits for VAEs through low-dimensional examples, before presenting quantitative and qualitative results. Our experiments demonstrate that skewing our variant of JSGa, in the context of JSGa-VAEs, leads to better reconstruction and generation when compared to several baseline VAEs. Our approach is entirely unsupervised and utilises only one hyperparameter which can be easily interpreted in latent space.
2020
34th Conference on Neural Information Processing Systems, NeurIPS 2020
Hyper-parameter; Jensen-Shannon divergence; Limiting case; Low dimensional; Mechanism-based; Potential benefits; Regularisation; Variational inference
04 Pubblicazione in atti di convegno::04b Atto di convegno in volume
Constraining variational inference with geometric Jensen-Shannon divergence / Deasy, J.; Simidjievski, N.; Lio, P.. - 2020-:(2020). (Intervento presentato al convegno 34th Conference on Neural Information Processing Systems, NeurIPS 2020 tenutosi a Virtual, Online).
File allegati a questo prodotto
File Dimensione Formato  
Deasy_constraining-variational_2020.pdf

accesso aperto

Tipologia: Versione editoriale (versione pubblicata con il layout dell'editore)
Licenza: Creative commons
Dimensione 2.74 MB
Formato Adobe PDF
2.74 MB Adobe PDF

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11573/1721102
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 7
  • ???jsp.display-item.citation.isi??? ND
social impact