Title: CMPUT 328 A7: Autoencoders and Generative Models

Question: What is an autoencoder and what are its two main components?
Answer: An autoencoder is a neural network that learns efficient representations of data in an unsupervised manner. Two components: 1. Encoder: Compresses input into lower-dimensional latent representation ($z = f_\phi(x)$) 2. Decoder: Reconstructs original input from latent code ($\hat{x} = g_\psi(z)$) Objective: Learn to reconstruct input such that $\hat{x} \approx x$ Loss: $L(x, \hat{x}) = \|x - \hat{x}\|^2$
Category: Introduction to Autoencoders

Question: What are the main applications of autoencoders? List at least 5.
Answer: Main applications: 1. Unsupervised learning: Learn from unlabeled data 2. Dimensionality reduction: Compress high-dimensional data (e.g., 784 → 32 for MNIST) 3. Feature learning: Extract meaningful representations for downstream tasks 4. Denoising: Remove noise from corrupted data 5. Generation: Create new data samples (with VAE/VQ-VAE) 6. Pre-training: Initialize networks for classification tasks 7. Anomaly detection: Detect outliers by reconstruction error
Category: Introduction to Autoencoders

Question: What is representation learning and why do autoencoders excel at it?
Answer: Representation learning: Automatic discovery of representations needed for feature detection or classification from raw data. Traditional approach: • Hand-craft features (edges, SIFT, HOG) • Train classifier on features Autoencoder approach: • Learn features automatically from unlabeled data • Use learned features for downstream tasks Benefits: • No manual feature engineering • Captures data structure automatically • Features are transferable to multiple tasks • Scales to large datasets
Category: Representation Learning

Question: Explain the architecture and loss function of a vanilla autoencoder for MNIST.
Answer: Architecture for MNIST (28×28 = 784 pixels): Encoder: $z = f_\phi(x)$ where $z \in \mathbb{R}^d$, $d < 784$ • Input: 784 • Hidden: 512 → 256 → 128 • Latent: 32 or 64 Decoder: $\hat{x} = g_\psi(z)$ • Mirror architecture • Output: 784 with Sigmoid activation Loss function: Mean Squared Error $$L = \frac{1}{n}\sum_{i=1}^n (x_i - \hat{x}_i)^2$$ Compression: 784 / 32 = 24.5× compression!
Category: Architecture
Code: # Vanilla Autoencoder
class VanillaAutoencoder(nn.Module):
def __init__(self, input_dim=784, latent_dim=32):
super().__init__()
self.encoder = nn.Sequential(
nn.Linear(input_dim, 256),
nn.ReLU(),
nn.Linear(256, 128),
nn.ReLU(),
nn.Linear(128, latent_dim)
)
self.decoder = nn.Sequential(
nn.Linear(latent_dim, 128),
nn.ReLU(),
nn.Linear(128, 256),
nn.ReLU(),
nn.Linear(256, input_dim),
nn.Sigmoid()
)
def forward(self, x):
z = self.encoder(x)
x_recon = self.decoder(z)
return x_recon, z

Question: How do autoencoders compare to PCA for dimensionality reduction? When does each excel?
Answer: PCA (Principal Component Analysis): • Linear transformation: $z = W^T x$ • Finds orthogonal directions of maximum variance • Fast and guaranteed optimal for linear case Autoencoder: • Non-linear transformation with activation functions • Can learn complex manifolds • More expressive than PCA When they match: If encoder/decoder are linear with no activation, autoencoder learns PCA subspace! When autoencoder excels: • With non-linear activations (ReLU, tanh) • Data lies on curved manifold • Complex relationships between features • Example: 3D data on curved 2D surface (PCA finds plane, AE finds curve)
Category: Dimensionality Reduction

Question: Why use convolutional autoencoders for images? Explain the architecture design.
Answer: Why convolutional for images? 1. Preserve spatial structure: Don't flatten early 2. Parameter efficiency: Weight sharing 3. Translation invariance: Features detected anywhere 4. Better visual features: Hierarchical patterns Architecture design: Encoder: Conv + Pooling (downsample) • 28×28 → 14×14 → 7×7 → 1×1 Decoder: Transposed Conv (upsample) • 1×1 → 7×7 → 14×14 → 28×28 Output size formulas: • Conv: $\lfloor(\text{in} + 2p - k)/s\rfloor + 1$ • TransConv: $(\text{in}-1) \times s - 2p + k + \text{out\_pad}$
Category: Convolutional Autoencoders
Code: # Convolutional Autoencoder
self.encoder = nn.Sequential(
nn.Conv2d(1, 16, 3, stride=2, padding=1), # 28->14
nn.ReLU(),
nn.Conv2d(16, 32, 3, stride=2, padding=1), # 14->7
nn.ReLU(),
nn.Conv2d(32, 64, 7) # 7->1
)
self.decoder = nn.Sequential(
nn.ConvTranspose2d(64, 32, 7), # 1->7
nn.ReLU(),
nn.ConvTranspose2d(32, 16, 3, stride=2,
padding=1, output_padding=1), # 7->14
nn.ReLU(),
nn.ConvTranspose2d(16, 1, 3, stride=2,
padding=1, output_padding=1), # 14->28
nn.Sigmoid()
)

Question: What is phased (greedy layer-wise) training for autoencoders? When is it useful?
Answer: Phased Training (Hinton & Salakhutdinov, 2006): Train layers one at a time Process: 1. Phase 1: Train first encoder-decoder pair Input → E1 → D1 → Reconstruction 2. Phase 2: Freeze E1, train second pair Input → E1 (frozen) → E2 → D2 → D1 (frozen) → Recon Benefits: • Easier optimization (simpler at each stage) • Better initialization for deep networks • Prevents vanishing gradients in very deep networks Modern usage: • Less common now (batch norm + good init like Xavier/He help) • Still useful for very deep networks • More of historical importance
Category: Training Strategies

Question: Compare the three main loss functions for autoencoders: MSE, Binary Cross-Entropy, and L1.
Answer: 1. Mean Squared Error (L2): $$L = \frac{1}{n} \sum_{i=1}^n (x_i - \hat{x}_i)^2$$ • Best for: Continuous values • Characteristic: Penalizes large errors heavily 2. Binary Cross-Entropy: $$L = -\frac{1}{n} \sum_{i=1}^n [x_i \log \hat{x}_i + (1-x_i) \log(1-\hat{x}_i)]$$ • Best for: Binary or normalized images [0,1] • Use with: Sigmoid output activation 3. L1 Loss: $$L = \frac{1}{n} \sum_{i=1}^n |x_i - \hat{x}_i|$$ • Best for: Robust to outliers • Characteristic: Encourages sparsity Choice: MSE for general, BCE for images with sigmoid
Category: Training Strategies
Code: # MSE Loss
criterion = nn.MSELoss()
loss = criterion(x_recon, x)
# Binary Cross-Entropy
criterion = nn.BCELoss()
loss = criterion(x_recon, x) # x_recon from Sigmoid
# L1 Loss
criterion = nn.L1Loss()
loss = criterion(x_recon, x)

Question: How can autoencoders be used for semi-supervised learning? Explain the two-stage approach.
Answer: Problem: Lots of unlabeled data, few labeled samples Two-Stage Approach: Stage 1: Unsupervised Pre-training • Train autoencoder on ALL data (labeled + unlabeled) • Learn general features from entire dataset • No labels needed Stage 2: Supervised Fine-tuning • Use pre-trained encoder + add classifier head • Train on labeled data only • Encoder weights can be frozen or fine-tuned Why it works: • Encoder learns data structure from all samples • Regularization: Prevents overfitting on small labeled set • Transfer learning: General features → specific task Results on MNIST (100 labeled samples): • No pre-training: ~70% accuracy • With autoencoder: ~85% accuracy
Category: Semi-Supervised Learning
Code: # Stage 1: Pre-train autoencoder
autoencoder = VanillaAutoencoder(784, 32)
train_autoencoder(autoencoder, all_data)
# Stage 2: Classification
class Classifier(nn.Module):
def __init__(self, encoder, num_classes=10):
super().__init__()
self.encoder = encoder # Pre-trained
self.classifier = nn.Linear(32, num_classes)
def forward(self, x):
z = self.encoder(x)
return self.classifier(z)
model = Classifier(autoencoder.encoder, 10)
train_classifier(model, labeled_data)

Question: What is a Denoising Autoencoder (DAE)? Explain the training process and benefits.
Answer: Denoising Autoencoder: Train to reconstruct clean input from corrupted version Process: 1. Start with clean input $x$ 2. Add corruption: $\tilde{x} = \text{Corrupt}(x)$ 3. Encode corrupted: $z = \text{Encoder}(\tilde{x})$ 4. Decode to clean: $\hat{x} = \text{Decoder}(z)$ 5. Loss: $L = \|x - \hat{x}\|^2$ (compare to original!) Types of corruption: • Gaussian noise: $\tilde{x} = x + \epsilon$, $\epsilon \sim \mathcal{N}(0, \sigma^2)$ • Masking (dropout): Set random pixels to 0 • Salt-and-pepper: Random pixels → 0 or 1 • Blur: Apply Gaussian blur Benefits: 1. Robustness: Features invariant to noise 2. Better representations: Must capture data structure 3. Prevents memorization: Can't just copy input 4. Implicit regularization: Acts as data augmentation
Category: Denoising Autoencoders
Code: def add_noise(x, noise_type='gaussian', level=0.3):
if noise_type == 'gaussian':
noise = torch.randn_like(x) * level
return torch.clamp(x + noise, 0, 1)
elif noise_type == 'masking':
mask = torch.rand_like(x) > level
return x * mask.float()
# Training
x_noisy = add_noise(x, 'gaussian', 0.3)
x_recon, z = model(x_noisy)
loss = F.mse_loss(x_recon, x) # Compare to clean x!

Question: What noise level should be used for denoising autoencoders? How does noise level affect training?
Answer: Noise level selection (for Gaussian noise σ): • Too low (σ < 0.1): Model might still memorize • Optimal (σ = 0.2-0.4): Good balance • Too high (σ > 0.5): Task too difficult, poor results Recommendation: Start with σ = 0.3, adjust based on reconstruction quality Effects of noise level: Low noise: • Easier task • Better reconstruction • Less robust features • May not force learning of structure High noise: • Harder task • Worse reconstruction • More robust features • May be too difficult to learn Tip: Visualize reconstructions at different noise levels to find sweet spot
Category: Denoising Autoencoders

Question: Explain the difference between Probability Mass Function (PMF) and Probability Density Function (PDF).
Answer: Probability Mass Function (PMF): For discrete random variables $X \in \{x_1, x_2, \ldots, x_n\}$ $$P(X = x_i) = p_i$$ Properties: • $0 \leq p_i \leq 1$ for all $i$ • $\sum_{i=1}^n p_i = 1$ • Example: Fair die, $P(X=k) = 1/6$ for $k \in \{1,2,3,4,5,6\}$ Probability Density Function (PDF): For continuous random variables $X \in \mathbb{R}$ $$p(x) = \frac{dP(X \leq x)}{dx}$$ Properties: • $p(x) \geq 0$ for all $x$ • $\int_{-\infty}^{\infty} p(x) dx = 1$ • Note: $p(x)$ can be > 1! (It's a density, not probability) • Probability: $P(a \leq X \leq b) = \int_a^b p(x) dx$
Category: Probability Primer

Question: Define expectation, variance, and explain the relationships between joint, marginal, and conditional distributions.
Answer: Expectation (mean): • Discrete: $\mathbb{E}[X] = \sum_i x_i P(X = x_i)$ • Continuous: $\mathbb{E}[X] = \int x \, p(x) dx$ Variance: $$\text{Var}(X) = \mathbb{E}[(X - \mu)^2] = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$$ Distributions: Joint: $p(x, y)$ - probability of both $x$ and $y$ Marginal: $p(x) = \int p(x, y) dy$ - probability of $x$ alone Conditional: $p(x | y) = \frac{p(x, y)}{p(y)}$ - probability of $x$ given $y$ Independence: $p(x, y) = p(x) p(y)$ if $X$ and $Y$ independent
Category: Probability Primer

Question: Write the formula for univariate and multivariate Gaussian distributions. Explain the parameters.
Answer: Univariate Gaussian (1D): $$p(x | \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$$ Parameters: • $\mu$: Mean (center) • $\sigma^2$: Variance (spread) • Notation: $X \sim \mathcal{N}(\mu, \sigma^2)$ Standard Normal: $\mu=0, \sigma^2=1$ Multivariate Gaussian (d-dimensional): $$p(\mathbf{x} | \boldsymbol{\mu}, \boldsymbol{\Sigma}) = \frac{1}{(2\pi)^{d/2} |\boldsymbol{\Sigma}|^{1/2}} \exp\left(-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1} (\mathbf{x}-\boldsymbol{\mu})\right)$$ Parameters: • $\boldsymbol{\mu} \in \mathbb{R}^d$: Mean vector • $\boldsymbol{\Sigma} \in \mathbb{R}^{d \times d}$: Covariance matrix Diagonal covariance: Variables independent, $\boldsymbol{\Sigma} = \text{diag}(\sigma_1^2, \ldots, \sigma_d^2)$
Category: Gaussian Distributions
Code: # Sampling
import numpy as np
# Univariate
samples = np.random.normal(mu, sigma, size=1000)
# Multivariate
mu = np.array([0, 0])
Sigma = np.array([[1, 0.5], [0.5, 1]])
samples = np.random.multivariate_normal(mu, Sigma, 1000)

Question: Why do we use Gaussian distributions for VAE? List at least 4 reasons.
Answer: Why Gaussian for VAE? 1. Mathematically tractable: • Easy to compute KL-divergence analytically • Closed-form solution exists 2. Central Limit Theorem: • Many distributions converge to Gaussian • Universal approximator property 3. Differentiable: • All operations differentiable • Easy to optimize with gradient descent 4. Simple parameterization: • Only need mean $\mu$ and variance $\sigma^2$ • Neural networks can output both 5. Reparameterization trick: • Can sample and still backpropagate • $z = \mu + \sigma \odot \epsilon$ where $\epsilon \sim \mathcal{N}(0, 1)$ 6. Smooth latent space: • Continuous distribution • Enables interpolation
Category: Gaussian Distributions

Question: What is a conditional density function? Explain its role in VAE with examples.
Answer: Conditional Density: $p(x | y)$ - probability of $x$ given $y$ Bayes' Rule: $$p(x | y) = \frac{p(y | x) p(x)}{p(y)}$$ • Posterior: $p(x | y)$ • Likelihood: $p(y | x)$ • Prior: $p(x)$ • Evidence: $p(y)$ In VAE: Decoder is conditional density: $$p_\theta(x | z)$$ "Probability of image $x$ given latent code $z$" Encoder is approximate posterior: $$q_\phi(z | x)$$ "Probability of latent $z$ given image $x$" Example - Image generation: • $p(\text{image} | y=\text{"cat"})$: Distribution of cat images • $p(\text{image} | y=\text{"dog"})$: Distribution of dog images
Category: Conditional Density

Question: What is the key difference between standard autoencoders and VAE? Why can VAE generate but standard AE cannot?
Answer: Standard Autoencoder: • Deterministic encoding: $z = f(x)$ • Latent space can have "holes" • No structure in latent space • Cannot generate - only reconstruct Variational Autoencoder (VAE): • Probabilistic encoding: $q(z|x) = \mathcal{N}(\mu(x), \sigma^2(x))$ • Latent distribution forced to match prior $\mathcal{N}(0, I)$ • Structured, continuous latent space • Can generate: Sample $z \sim \mathcal{N}(0, I)$, decode to $x$ Why VAE can generate: 1. KL term forces $q(z|x) \approx \mathcal{N}(0, I)$ for all $x$ 2. No holes: Every point in latent space maps to valid data 3. Smooth: Nearby $z$ → similar $x$ 4. Sampling: Just sample from $\mathcal{N}(0, I)$ and decode! Standard AE problem: Random $z$ likely in "hole" → garbage output
Category: Variational Autoencoders

Question: Derive and explain the VAE loss function (ELBO). What are the two terms and their roles?
Answer: Evidence Lower Bound (ELBO): Start from log-likelihood: $$\log p(x) \geq \mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)] - D_{KL}(q_\phi(z|x) \| p(z))$$ VAE Loss = -ELBO: $$L = -\mathbb{E}_{q(z|x)}[\log p(x|z)] + D_{KL}(q(z|x) \| p(z))$$ Two terms: 1. Reconstruction Loss: $-\mathbb{E}_{q(z|x)}[\log p(x|z)]$ • How well can we reconstruct $x$ from $z$? • In practice: MSE or BCE between $x$ and $\hat{x}$ • Role: Make reconstructions accurate 2. KL Divergence: $D_{KL}(q(z|x) \| p(z))$ • How different is $q(z|x)$ from prior $p(z) = \mathcal{N}(0, I)$? • Role: Regularize latent space, enable sampling Trade-off: Good reconstruction vs. structured latent space
Category: Variational Autoencoders
Code: def vae_loss(x, x_recon, mu, logvar):
# Reconstruction (BCE for normalized images)
recon = F.binary_cross_entropy(x_recon, x, reduction='sum')
# KL divergence
kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
return recon + kl

Question: Explain the reparameterization trick. Why is it necessary and how does it work?
Answer: Problem: Cannot backpropagate through sampling $z \sim q_\phi(z|x)$ • Sampling is not differentiable • Gradient cannot flow from decoder to encoder Reparameterization Trick: Original (non-differentiable): $$z \sim \mathcal{N}(\mu, \sigma^2)$$ Reparameterized (differentiable): $$\epsilon \sim \mathcal{N}(0, 1)$$ $$z = \mu + \sigma \odot \epsilon$$ where $\odot$ is element-wise multiplication. Key insight: • Randomness moved to $\epsilon$ (fixed distribution) • $z$ is now deterministic function of $\mu, \sigma, \epsilon$ • Gradient can flow through $\mu$ and $\sigma$! Backprop works: $$\frac{\partial L}{\partial \mu}, \quad \frac{\partial L}{\partial \sigma}$$
Category: Variational Autoencoders
Code: def reparameterize(mu, logvar):
"""
Reparameterization trick for VAE
"""
# Compute std from log variance
std = torch.exp(0.5 * logvar)
# Sample epsilon from standard normal
eps = torch.randn_like(std)
# Reparameterize: z = mu + sigma * epsilon
z = mu + eps * std
return z

Question: Define KL-divergence. What are its key properties and what does it measure?
Answer: KL-divergence (Kullback-Leibler): Measures how distribution $Q$ differs from $P$ Continuous: $$D_{KL}(P \| Q) = \int p(x) \log \frac{p(x)}{q(x)} dx$$ Discrete: $$D_{KL}(P \| Q) = \sum_i P(i) \log \frac{P(i)}{Q(i)}$$ Key Properties: 1. Non-negative: $D_{KL}(P \| Q) \geq 0$ 2. Zero iff equal: $D_{KL}(P \| Q) = 0$ iff $P = Q$ 3. Asymmetric: $D_{KL}(P \| Q) \neq D_{KL}(Q \| P)$ in general 4. Not a metric: Doesn't satisfy triangle inequality Intuition: "How much information is lost when $Q$ approximates $P$" • Large $D_{KL}$: Very different distributions • Small $D_{KL}$: Similar distributions
Category: KL-Divergence

Question: Derive the KL-divergence formula for two Gaussians. What is the special case for N(μ,σ²) || N(0,1)?
Answer: General case: $P = \mathcal{N}(\mu_1, \sigma_1^2)$, $Q = \mathcal{N}(\mu_2, \sigma_2^2)$ $$D_{KL}(P \| Q) = \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2\sigma_2^2} - \frac{1}{2}$$ Special case: $Q = \mathcal{N}(0, 1)$ (standard normal) $$D_{KL}(\mathcal{N}(\mu, \sigma^2) \| \mathcal{N}(0, 1)) = \frac{1}{2}\left(\mu^2 + \sigma^2 - \log \sigma^2 - 1\right)$$ Multivariate: $P = \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$, $Q = \mathcal{N}(0, I)$ With diagonal $\boldsymbol{\Sigma}$: $$D_{KL} = \frac{1}{2} \sum_{i=1}^k \left(\mu_i^2 + \sigma_i^2 - \log \sigma_i^2 - 1\right)$$
Category: KL-Divergence
Code: def kl_divergence_gaussian(mu, logvar):
"""
KL between N(mu, sigma^2) and N(0, 1)
Args:
mu: Mean, shape (batch, latent_dim)
logvar: Log variance, shape (batch, latent_dim)
"""
# Formula: -0.5 * sum(1 + log(sigma^2) - mu^2 - sigma^2)
kl = -0.5 * torch.sum(
1 + logvar - mu.pow(2) - logvar.exp(),
dim=1
)
return kl.mean() # Average over batch

Question: What is β-VAE? How does the β parameter affect the model?
Answer: β-VAE: Adjust weight of KL term in VAE loss $$L = \text{Recon Loss} + \beta \cdot D_{KL}(q(z|x) \| p(z))$$ Effect of β: β < 1: • Less KL regularization • Better reconstructions • Less structured latent space • Worse generation β = 1: • Standard VAE • Balance between reconstruction and generation β > 1: • More KL regularization • Worse reconstructions • More disentangled latent space • Better separation of factors Trade-offs: • High β: Blurry images but interpretable latents • Low β: Sharp images but entangled latents Disentanglement: Each latent dimension captures independent factor (e.g., one for rotation, one for scale)
Category: KL-Divergence
Code: # β-VAE loss
def beta_vae_loss(x, x_recon, mu, logvar, beta=1.0):
recon_loss = F.binary_cross_entropy(x_recon, x, reduction='sum')
kl_loss = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
return recon_loss + beta * kl_loss
# Try different β values
loss = beta_vae_loss(x, x_recon, mu, logvar, beta=2.0) # More disentangled

Question: What is VQ-VAE? How does it differ from standard VAE?
Answer: Vector Quantized VAE (VQ-VAE): Key difference: Uses discrete latent space instead of continuous Architecture: Input → Encoder → $z_e$ (continuous) → Quantization → $z_q$ (discrete) → Decoder → Output Codebook: $\mathcal{C} = \{e_1, e_2, \ldots, e_K\}$ where $e_k \in \mathbb{R}^D$ • $K$: Number of entries (e.g., 512) • $D$: Embedding dimension (e.g., 64) Quantization: Map $z_e$ to nearest codebook entry $$z_q = e_k \text{ where } k = \arg\min_j \|z_e - e_j\|_2$$ vs Standard VAE: • VAE: Continuous $z \sim \mathcal{N}(0, I)$ • VQ-VAE: Discrete $z \in \{e_1, \ldots, e_K\}$ Advantages: • No posterior collapse • Sharper reconstructions • Better for autoregressive models (e.g., PixelCNN)
Category: VQ-VAE

Question: Explain the three loss components of VQ-VAE. What is the straight-through estimator?
Answer: VQ-VAE Loss (three components): $$L = \underbrace{\|x - \hat{x}\|^2}_{\text{Reconstruction}} + \underbrace{\|\text{sg}[z_e] - e_k\|_2^2}_{\text{Codebook}} + \underbrace{\beta \|z_e - \text{sg}[e_k]\|_2^2}_{\text{Commitment}}$$ where $\text{sg}[\cdot]$ = stop gradient (no backprop) 1. Reconstruction: Standard autoencoder loss 2. Codebook Loss: • Moves codebook entries $e_k$ towards encoder outputs $z_e$ • Gradient only to $e_k$ • Updates codebook 3. Commitment Loss: • Forces encoder to commit to codebook entry • Prevents encoder from growing arbitrarily • Gradient only to encoder Straight-Through Estimator: • Problem: Quantization $z_q = \arg\min_k \|z_e - e_k\|$ is non-differentiable • Solution: In backward pass, copy gradient: $\frac{\partial L}{\partial z_e} = \frac{\partial L}{\partial z_q}$ • "Pretend" quantization is identity
Category: VQ-VAE
Code: # Straight-through estimator
z_q = self.quantize(z_e) # Non-differentiable
# Copy gradient in backward pass
z_q = z_e + (z_q - z_e).detach()
# Now gradient flows: dL/dz_e = dL/dz_q

Question: What is codebook collapse in VQ-VAE? How can it be prevented?
Answer: Codebook Collapse: Only a few codebook entries get used, others ignored Problem: • Wastes capacity • Poor representation diversity • Some $e_k$ never updated Prevention strategies: 1. EMA (Exponential Moving Average) updates: • Instead of gradient-based updates • $\mathbf{e}_k \leftarrow (1-\gamma) \mathbf{e}_k + \gamma \cdot \text{mean}(\text{encodings assigned to } k)$ 2. Restart unused entries: • Track usage count for each entry • If $\text{count}[k] < \text{threshold}$, reinitialize randomly 3. Increase commitment cost β: • Force encoder to use all entries • Typical: β = 0.25 4. Larger codebook: • More entries (e.g., 1024 instead of 512) • Reduces pressure on each entry Monitor: Track codebook usage during training
Category: VQ-VAE
Code: # Monitor codebook usage usage_count = torch.zeros(num_embeddings) # During training unique_indices, counts = torch.unique( encoding_indices, return_counts=True ) usage_count[unique_indices] += counts # Reset unused entries if epoch % 10 == 0: unused = (usage_count < threshold).nonzero() for idx in unused: model.codebook[idx] = torch.randn_like(model.codebook[idx]) usage_count.zero_()

Question: What is posterior collapse in VAE? What are the symptoms and how can it be fixed?
Answer: Posterior Collapse: KL divergence → 0, model ignores latent code Symptoms: • KL loss ≈ 0 • All generated samples look the same • Decoder becomes unconditional (ignores $z$) • Latent space not used Causes: • Decoder too powerful • KL weight too high early in training • Poor initialization Solutions: 1. KL Annealing: Start with β = 0, gradually increase to 1 2. Free Bits: Don't penalize KL below threshold $\text{KL} = \max(D_{KL}, \text{free\_bits})$ 3. Weaken Decoder: • Fewer layers • Smaller hidden dims • Add dropout 4. Cyclical Annealing: Cycle β between 0 and 1 during training
Category: Common Pitfalls
Code: # KL Annealing def kl_weight(epoch, total_epochs=100, start=10): if epoch < start: return 0.0 return min(1.0, (epoch - start) / (total_epochs - start)) loss = recon + kl_weight(epoch) * kl_loss # Free Bits def free_bits_kl(kl, free_bits=2.0): return torch.max(kl, torch.tensor(free_bits)) kl_loss = free_bits_kl(kl_loss, free_bits=2.0)

Question: Why are VAE reconstructions blurry? What solutions exist?
Answer: Why blurry? 1. MSE/BCE loss: Averages over all possible reconstructions 2. Sampling: Introduces noise in latent space 3. KL regularization: Constrains latent space, limits expressiveness Solutions: 1. Increase latent dimension: • Use 128 or 256 instead of 20 • More capacity to capture details 2. VQ-VAE: • Discrete latents eliminate sampling blur • Sharper reconstructions 3. Perceptual loss: • Use VGG features instead of pixel-wise MSE • Matches high-level structure 4. Adversarial loss: • Add discriminator (VAE-GAN) • Encourages realistic images 5. Reduce β: • β < 1 for better reconstruction • Trade-off with generation quality 6. Deeper/wider decoder: • More capacity to generate details
Category: Common Pitfalls
Code: # Perceptual Loss
class PerceptualLoss(nn.Module):
def __init__(self):
super().__init__()
vgg = models.vgg16(pretrained=True).features[:16]
self.features = vgg.eval()
for p in self.features.parameters():
p.requires_grad = False
def forward(self, x, x_recon):
f_x = self.features(x)
f_recon = self.features(x_recon)
return F.mse_loss(f_recon, f_x)
# Use in training
loss = recon + 0.1 * perceptual_loss(x, x_recon) + kl

Question: What common numerical stability issues arise in VAE training? How to fix them?
Answer: Common Issues: 1. NaN/Inf in KL loss: • Cause: $\log(0)$ or $\exp(\text{very large})$ • Fix: Clamp logvar 2. Gradient explosion: • Cause: Large gradients in early training • Fix: Gradient clipping 3. Underflow in BCE: • Cause: $\log(\text{very small})$ • Fix: Use `binary_cross_entropy_with_logits` 4. Extreme variance: • Cause: Unconstrained $\sigma^2$ • Fix: Clamp logvar to [-10, 10] 5. Learning rate too high: • Cause: Unstable training • Fix: Start with 1e-3, use scheduler Best practices: • Always clamp logvar • Use gradient clipping • Monitor loss values for NaN • Use stable loss functions
Category: Common Pitfalls
Code: # Clamp logvar
def encode(self, x):
h = self.encoder(x)
mu = self.fc_mu(h)
logvar = self.fc_logvar(h)
logvar = torch.clamp(logvar, min=-10, max=10) # Prevent extremes
return mu, logvar
# Gradient clipping
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
# Stable BCE
loss = F.binary_cross_entropy_with_logits(logits, target) # Not BCE on sigmoid output