Generative modelling in latent space – Sander Dieleman
Fantastic deep dive into the concept of latents and the tradeoffs around them by Sander Dieleman of DeepMind. It’s a long article, but there’s a conclusions section that pulls out some of the most interesting points, and each section is an expansion on those points.
- Latents add complexity, but the computational efficiency benefits are large enough for us to tolerate this complexity – at least for now.
- Three main aspects to consider when designing latent spaces are capacity (how many bits of information are encoded in the latents), curation (which bits from the input signals are retained) and shape (how this information is presented).
- Preserving structure (i.e. topology, statistics) in the latent space is important to make it easy to model, even if this is sometimes worse from an efficiency perspective.