VQ-VAE — the core idea.
hardAnswer
- Discrete latent codes: encoder output quantized to nearest code in a learned codebook (like k-means).
- Straight-through gradient estimator for backprop.
- Uses: (1) discrete-latent generative modeling (audio in WaveNet-style, video in Sora / VideoPoet), (2) tokenizer for text-image (DALL-E), (3) speech (Wav2Vec 2).
- Enables autoregressive priors on discrete latents.
- Foundation of many modern multi-modal generative models.
Check yourself — multiple choice
- Random
- Discrete latents via codebook quantization (like k-means); straight-through gradient; foundation of DALL-E / Sora / Wav2Vec / audio-video codebook models
- Same as VAE
- Not real
VQ-VAE: discrete codebook latents; straight-through grad; multi-modal generation.
#representation-learning#deep-learning
Practise Unsupervised Learning
214 interview questions in this topic.