Title: Diffusion Models - Complete Guide

Question: What are diffusion models and how do they generate data?
Answer: Definition: Generative models that learn to generate data by reversing a gradual noising process Two-phase process: 1. Forward diffusion: Gradually add noise to data until it becomes pure Gaussian noise 2. Reverse denoising: Learn to remove noise step-by-step to generate new samples Key insight: If we can reverse the process of adding noise, we can generate by starting with noise and denoising State-of-the-art: Powers DALL-E 2, Stable Diffusion, Imagen
Category: FUNDAMENTALS

Question: Explain the ink drop analogy for diffusion models.
Answer: Forward (ink dispersing): • t=0: Clear water with concentrated ink drop (clean image) • t=1...T-1: Ink gradually diffuses throughout water (adding noise) • t=T: Ink completely dispersed, uniform murky water (pure noise) Reverse (un-diffusing): • Learn to reverse this process • Start with murky water (noise) • Progressively un-diffuse back to clear drop (clean image) For generation: Sample random noise and apply learned un-diffusion process
Category: FUNDAMENTALS

Question: What are the key advantages of diffusion models over GANs?
Answer: Training stability: • No adversarial dynamics • No mode collapse • Single objective (MSE loss) • Robust to hyperparameters Sample quality: • State-of-the-art FID scores • Better diversity coverage • Consistent quality Likelihood-based: • Can compute log p(x) • Theoretical guarantees • Better evaluation Flexibility: • Easy conditioning (text, class, image) • Natural inpainting/editing • Scalable to high resolutions
Category: FUNDAMENTALS

Question: What is the main disadvantage of diffusion models compared to GANs?
Answer: Sampling speed: VERY slow inference Comparison: • GAN: 1 forward pass (~0.01 seconds per image) • DDPM: 1000 forward passes (~10 seconds per image) • 1000x slower than GANs! Why slow: • Must iteratively denoise T times (typically T=1000) • Sequential process, cannot parallelize • Each step requires full network forward pass Solutions: • DDIM: Reduce to 50-100 steps (~20x faster) • Latent diffusion: Run in compressed space • Distillation: Train faster sampling models Trade-off: Quality vs speed
Category: FUNDAMENTALS

Question: What is the Markov chain perspective on diffusion models?
Answer: Forward chain: x_0 to x_1 to x_2 to ... to x_T • Each step depends only on previous (Markov property) • Gradually destroys information • q(x_t | x_{t-1}) is Gaussian transition Reverse chain: x_T to x_{T-1} to x_{T-2} to ... to x_0 • Go backward through chain • Gradually creates information • p(x_{t-1} | x_t) learned by neural network Generation: Sample x_T ~ N(0,I), run reverse chain to get x_0
Category: FUNDAMENTALS

Question: Write the single-step forward diffusion equation.
Answer: Formula: q(x_t | x_{t-1}) = N(x_t; sqrt(1 - beta_t) * x_{t-1}, beta_t * I) Sampling form: x_t = sqrt(1 - beta_t) * x_{t-1} + sqrt(beta_t) * epsilon where epsilon ~ N(0, I) Components: • beta_t: Noise schedule parameter • sqrt(1 - beta_t): Scaling to preserve variance • sqrt(beta_t) * epsilon: Noise being added • I: Identity (isotropic Gaussian) Effect: Adds a small amount of Gaussian noise at each step
Category: FORWARD PROCESS

Question: What is the closed-form forward diffusion formula?
Answer: Key advantage: Jump directly from x_0 to x_t without intermediate steps Formula: q(x_t | x_0) = N(x_t; sqrt(alpha_bar_t) * x_0, (1 - alpha_bar_t) * I) Sampling: x_t = sqrt(alpha_bar_t) * x_0 + sqrt(1 - alpha_bar_t) * epsilon Where: • alpha_t = 1 - beta_t • alpha_bar_t = product of all alpha_s from s=1 to t Training benefit: Can sample any noise level directly for training
Category: FORWARD PROCESS
Code: def q_sample(x_0, t, noise=None):
if noise is None:
noise = torch.randn_like(x_0)
sqrt_alphas_cumprod_t = extract(
sqrt_alphas_cumprod, t, x_0.shape
)
sqrt_one_minus_alphas_cumprod_t = extract(
sqrt_one_minus_alphas_cumprod, t, x_0.shape
)
return (sqrt_alphas_cumprod_t * x_0 +
sqrt_one_minus_alphas_cumprod_t * noise)

Question: What is the noise schedule beta_t and why does it matter?
Answer: Definition: Sequence of values controlling how much noise to add at each timestep Common schedules: • Linear: beta_t increases linearly from 1e-4 to 0.02 (most common) • Cosine: Smoother increase, preserves structure better • Quadratic: Faster noise increase Impact: • Too fast: Information lost too quickly, hard to reverse • Too slow: Takes many steps to reach pure noise • Affects training stability and sample quality Standard DDPM: Linear from beta_1=1e-4 to beta_T=0.02
Category: FORWARD PROCESS

Question: What happens to the data distribution as t approaches T?
Answer: Convergence to pure noise: As t to T, alpha_bar_t to 0 Result: x_T approximately N(0, I) - standard Gaussian noise Independence from x_0: Regardless of original image, x_T is approximately same distribution Why this matters: • Reverse process can start from simple N(0,I) • Don't need to match complex data distribution • Just need to reverse the noising process Signal-to-noise ratio: SNR(t) = alpha_bar_t / (1 - alpha_bar_t) approaches 0
Category: FORWARD PROCESS

Question: Implement the linear beta schedule in PyTorch.
Answer: Standard linear schedule for DDPM:
Category: FORWARD PROCESS
Code: def linear_beta_schedule(timesteps,
beta_start=1e-4,
beta_end=0.02):
"""
Linear noise schedule from beta_start to beta_end
Args:
timesteps: Number of diffusion steps (T)
beta_start: Starting value (typically 1e-4)
beta_end: Ending value (typically 0.02)
Returns:
Tensor of beta values
"""
return torch.linspace(beta_start, beta_end, timesteps)
# Precompute alpha values
betas = linear_beta_schedule(1000)
alphas = 1.0 - betas
alphas_cumprod = torch.cumprod(alphas, dim=0)

Question: What does the neural network predict in DDPM?
Answer: Standard choice: Predict the noise epsilon that was added Network signature: epsilon_theta(x_t, t) approximately epsilon Three equivalent options: 1. Predict noise epsilon (DDPM standard - best empirically) 2. Predict mean mu directly 3. Predict clean image x_0 Why predict noise works best: • Better gradient signal • Simpler optimization • More stable training Relationship: x_t = sqrt(alpha_bar_t) * x_0 + sqrt(1 - alpha_bar_t) * epsilon Knowing epsilon lets us recover x_0
Category: REVERSE PROCESS

Question: Write the formula for the reverse denoising step.
Answer: Single denoising step x_t to x_{t-1}: 1. Predict noise: epsilon_hat = epsilon_theta(x_t, t) 2. Compute mean: mu = (1/sqrt(alpha_t)) * (x_t - (beta_t/sqrt(1-alpha_bar_t)) * epsilon_hat) 3. Sample with noise: x_{t-1} = mu + sigma_t * z where z ~ N(0,I) if t > 1, else z = 0 Variance: sigma_t^2 = beta_t or beta_tilde_t (posterior variance) Final step: No noise added at t=1 to get x_0
Category: REVERSE PROCESS

Question: Why is the reverse process approximately Gaussian?
Answer: Theorem: When beta_t is small, p(x_{t-1} | x_t) is approximately Gaussian Intuition: • Small steps mean small changes • Central limit theorem behavior • Reverse of adding small Gaussian noise is removing small Gaussian noise Implication: • Can parameterize as Gaussian: N(mu, sigma^2) • Only need to learn mean (variance can be fixed) • Simple neural network can approximate Practical: This is why diffusion models work - reversing is tractable
Category: REVERSE PROCESS

Question: What is the posterior variance beta_tilde_t?
Answer: Formula: beta_tilde_t = ((1 - alpha_bar_{t-1}) / (1 - alpha_bar_t)) * beta_t Purpose: Exact variance of q(x_{t-1} | x_t, x_0) Two choices for reverse variance: • sigma_t^2 = beta_t (upper bound, more random) • sigma_t^2 = beta_tilde_t (lower bound, less random) DDPM uses: beta_tilde_t for better sample quality Why it matters: • Affects sampling randomness • Too much: noisy samples • Too little: deterministic but may miss details
Category: REVERSE PROCESS

Question: Implement the reverse sampling step p_sample in PyTorch.
Answer: Single denoising step:
Category: REVERSE PROCESS
Code: @torch.no_grad()
def p_sample(model, x_t, t, t_index):
# Get precomputed values
betas_t = extract(betas, t, x_t.shape)
sqrt_one_minus_alphas_cumprod_t = extract(
sqrt_one_minus_alphas_cumprod, t, x_t.shape
)
sqrt_recip_alphas_t = extract(
sqrt_recip_alphas, t, x_t.shape
)
# Predict noise
predicted_noise = model(x_t, t)
# Compute mean
model_mean = sqrt_recip_alphas_t * (
x_t - betas_t * predicted_noise /
sqrt_one_minus_alphas_cumprod_t
)
if t_index == 0:
return model_mean # No noise at last step
else:
posterior_variance_t = extract(
posterior_variance, t, x_t.shape
)
noise = torch.randn_like(x_t)
return model_mean + torch.sqrt(posterior_variance_t) * noise

Question: What is the simplified DDPM training loss?
Answer: L_simple: E_{t, x_0, epsilon} [ ||epsilon - epsilon_theta(x_t, t)||^2 ] Components: • t: Random timestep ~ Uniform({1,...,T}) • x_0: Clean image from data • epsilon: Random noise ~ N(0,I) • x_t: Noisy image = sqrt(alpha_bar_t)*x_0 + sqrt(1-alpha_bar_t)*epsilon • epsilon_theta: Network prediction Simplicity: Just MSE between predicted and actual noise! Why it works: Direct supervision signal for denoising task
Category: TRAINING

Question: Write the DDPM training algorithm pseudocode.
Answer: Training loop: REPEAT until converged: 1. Sample clean image x_0 from training data 2. Sample timestep t ~ Uniform({1,...,T}) 3. Sample noise epsilon ~ N(0,I) 4. Create noisy image: x_t = sqrt(alpha_bar_t)*x_0 + sqrt(1-alpha_bar_t)*epsilon 5. Predict noise: epsilon_hat = epsilon_theta(x_t, t) 6. Compute loss: L = ||epsilon - epsilon_hat||^2 7. Update theta by gradient descent Key points: • Random t ensures learning at all noise levels • Closed form q_sample enables efficient training • No adversarial dynamics • Stable and straightforward
Category: TRAINING

Question: Implement the DDPM training loop in PyTorch.
Answer: Simplified training code:
Category: TRAINING
Code: def train_ddpm(model, dataloader, optimizer,
timesteps=1000, epochs=100):
for epoch in range(epochs):
for x_0, _ in dataloader:
x_0 = x_0.to(device)
batch_size = x_0.size(0)
# Random timesteps
t = torch.randint(
0, timesteps, (batch_size,),
device=device
).long()
# Sample noise
noise = torch.randn_like(x_0)
# Create noisy images (closed form)
x_t = q_sample(x_0, t, noise=noise)
# Predict noise
noise_pred = model(x_t, t)
# MSE loss
loss = F.mse_loss(noise_pred, noise)
# Update
optimizer.zero_grad()
loss.backward()
optimizer.step()

Question: Why is diffusion model training more stable than GAN training?
Answer: Reasons for stability: 1. Single objective: • Just minimize MSE loss • No competing objectives • No minimax game 2. Direct supervision: • Know ground truth (the noise) • Clear training signal • No adversarial dynamics 3. All timesteps contribute: • Random t sampling covers all noise levels • Comprehensive training • No mode collapse 4. Smooth optimization: • Loss decreases monotonically • No oscillations • Robust to hyperparameters Result: Reliable convergence, no special tricks needed
Category: TRAINING

Question: What typical hyperparameters are used for DDPM training?
Answer: Standard configuration: T (timesteps): 1000 Learning rate: 1e-4 to 2e-4 Optimizer: Adam Batch size: 32-128 (depends on image size) Noise schedule: beta_1: 1e-4 beta_T: 0.02 Schedule: Linear EMA decay: 0.9999 (very important!) Training time: • Small images (32x32): ~1-2 days • Large images (256x256): ~1 week • On modern GPU (V100/A100) Robustness: These work across many datasets with minimal tuning
Category: TRAINING

Question: What is EMA and why is it crucial for diffusion models?
Answer: Exponential Moving Average: Track slowly-updating copy of model weights Update rule: shadow_params = decay * shadow_params + (1-decay) * current_params Typical decay: 0.9999 (very slow update) Why crucial: • Significantly improves sample quality • Smooths out training noise • More stable for generation • Can improve FID by 10-30 points! Usage: • Train with regular weights • Generate samples with EMA weights Implementation: Maintain shadow copy, update after each training step
Category: TRAINING
Code: class EMA:
def __init__(self, model, decay=0.9999):
self.decay = decay
self.shadow = {}
for name, param in model.named_parameters():
self.shadow[name] = param.data.clone()
def update(self, model):
for name, param in model.named_parameters():
self.shadow[name].mul_(self.decay).add_(
param.data, alpha=1-self.decay
)

Question: Write the DDPM sampling (generation) algorithm.
Answer: Full generation process: 1. Sample x_T ~ N(0, I) (pure random noise) 2. FOR t = T, T-1, ..., 1: a. Predict noise: epsilon_hat = epsilon_theta(x_t, t) b. Compute mean mu using formula c. Sample z ~ N(0,I) if t > 1, else z = 0 d. x_{t-1} = mu + sigma_t * z 3. RETURN x_0 (generated image) Time: T forward passes through network (slow!) Quality: High-quality samples if well-trained
Category: SAMPLING

Question: How do you generate samples from a trained diffusion model?
Answer: Process:
Category: SAMPLING
Code: @torch.no_grad()
def generate_samples(model, timesteps, num_samples,
channels, img_size):
model.eval()
# Start from pure noise
shape = (num_samples, channels, img_size, img_size)
x = torch.randn(shape, device=device)
# Iteratively denoise
for i in reversed(range(timesteps)):
t = torch.full((num_samples,), i, device=device)
x = p_sample(model, x, t, i)
# Denormalize [-1,1] to [0,1]
x = (x + 1) / 2
return torch.clamp(x, 0, 1)
# Usage
samples = generate_samples(
model, timesteps=1000, num_samples=16,
channels=3, img_size=32
)

Question: What is DDIM and how does it accelerate sampling?
Answer: Denoising Diffusion Implicit Models Key innovation: Deterministic sampling with skipped timesteps Speedup: • DDPM: 1000 steps • DDIM: 50-100 steps (20x faster!) • Minimal quality loss How it works: • Remove stochastic noise term (z=0) • Use only subset of timesteps • Modify sampling equation to be deterministic Trade-off: • Faster sampling • Slightly less diversity • Nearly same quality Applications: When speed matters (real-time, interactive)
Category: SAMPLING

Question: What is classifier-free guidance?
Answer: Purpose: Strengthen conditioning without external classifier Formula: epsilon_hat = epsilon_uncond + w * (epsilon_cond - epsilon_uncond) Components: • epsilon_cond: Noise predicted with condition (e.g., text) • epsilon_uncond: Noise predicted without condition • w: Guidance weight (typically 7-15) Training: • Same model for conditional and unconditional • Randomly drop condition during training Effect: • Higher w = follow condition more strongly • Trade-off: accuracy vs diversity Used in: DALL-E 2, Imagen, Stable Diffusion
Category: SAMPLING

Question: Why is U-Net the standard architecture for diffusion models?
Answer: Key properties: 1. Spatial preservation: • Skip connections maintain fine details • Encoder and decoder at same resolution 2. Multi-scale processing: • Downsampling captures global structure • Upsampling reconstructs details • Process at multiple resolutions simultaneously 3. Time conditioning: • Easy to inject timestep embeddings • Network knows which noise level to handle 4. Proven effective: • Originally for segmentation • Perfect for dense prediction (pixel-wise noise) Result: Excellent for image denoising task
Category: ARCHITECTURE

Question: How are timesteps embedded in the U-Net?
Answer: Sinusoidal positional encoding: Similar to Transformer positional encodings Formula: emb[2i] = sin(t / 10000^(2i/d)) emb[2i+1] = cos(t / 10000^(2i/d)) Dimensionality: Typically 256 or 512 Injection: • Pass through MLP • Add to features at each resolution level • Enables time-conditional processing Why this works: • Different frequencies for different timescales • Network learns to condition on noise level • Generalizes across timesteps
Category: ARCHITECTURE
Code: class SinusoidalEmbedding(nn.Module):
def __init__(self, dim):
super().__init__()
self.dim = dim
def forward(self, time):
half_dim = self.dim // 2
embeddings = math.log(10000) / (half_dim - 1)
embeddings = torch.exp(
torch.arange(half_dim) * -embeddings
)
embeddings = time[:, None] * embeddings[None, :]
return torch.cat((embeddings.sin(),
embeddings.cos()), dim=-1)

Question: What are the key components of a diffusion U-Net?
Answer: 1. ResNet blocks: • Residual connections for gradient flow • Group normalization (not BatchNorm!) • SiLU/Swish activation 2. Self-attention: • Applied at lower resolutions (16x16, 8x8) • Captures long-range dependencies • Crucial for coherence 3. Skip connections: • Concatenate encoder to decoder features • Preserve spatial information • Enable accurate prediction 4. Adaptive Group Norm: • Condition on time embedding • Replace standard normalization • Time-aware feature scaling 5. Downsampling/Upsampling: • Stride convolutions (down) • Transpose convolutions (up) • Process multiple scales
Category: ARCHITECTURE

Question: What is the variational lower bound (ELBO) for diffusion models?
Answer: Goal: Maximize log p_theta(x_0) ELBO: log p_theta(x_0) >= E_q[log p_theta(x_{0:T}) / q(x_{1:T}|x_0)] Decomposition: ELBO = E_q[-log p(x_T) + sum of KL divergences] Practical: • L_simple works better than full ELBO • ELBO provides theoretical justification • L_simple is derived from ELBO with approximations Importance: Shows diffusion models are likelihood-based
Category: MATHEMATICS

Question: What is the connection between diffusion models and score matching?
Answer: Score function: s(x,t) = grad_x log p_t(x) Relationship: Predicting noise epsilon is equivalent to predicting score! Formula: epsilon_theta(x_t, t) = -sqrt(1 - alpha_bar_t) * s_theta(x_t, t) Score-based view: • Learn gradient of log density • Langevin dynamics for sampling • Same model, different perspective Unification: Score-based and diffusion models are equivalent frameworks
Category: MATHEMATICS

Question: Compare diffusion models vs GANs side-by-side.
Answer: Diffusion Models: Training: Stable, single MSE loss Quality: State-of-the-art (FID ~2-5) Diversity: Excellent, no mode collapse Speed: Slow (1000 steps, ~10s) Likelihood: Yes, can compute Conditioning: Easy and flexible GANs: Training: Unstable, adversarial Quality: Excellent (FID ~5-20) Diversity: Can have mode collapse Speed: Fast (1 step, ~0.01s) Likelihood: No Conditioning: More difficult When to use diffusion: Quality and diversity matter most When to use GANs: Speed is critical (real-time)
Category: COMPARISON

Question: What are the advantages of diffusion models over VAEs?
Answer: vs VAEs: Sample quality: • Diffusion: Much sharper, more detailed • VAE: Often blurry Training: • Diffusion: Simple MSE loss • VAE: KL divergence + reconstruction Latent space: • Diffusion: No explicit latent (noise is latent) • VAE: Learned latent space Flexibility: • Both are likelihood-based • Both support conditioning • Diffusion generally easier to train Result: Diffusion models dominate for image generation
Category: COMPARISON

Question: What are the most important practical tips for training diffusion models?
Answer: Critical: 1. Use EMA (decay=0.9999) - huge impact! 2. Normalize images to [-1,1] 3. Start with T=1000, linear schedule 4. Use Group Normalization, not Batch Norm Architecture: 5. Self-attention at low resolutions 6. Residual connections everywhere 7. Inject time at each resolution Training: 8. Adam optimizer, LR=1e-4 9. Batch size as large as GPU allows 10. Train for many epochs (patience!) Monitoring: 11. Generate samples regularly 12. Track FID on validation set Debugging: 13. Check loss decreases smoothly 14. Verify normalization correct
Category: PRACTICAL

Question: How should images be normalized for diffusion model training?
Answer: Standard: Normalize to [-1, 1] Reason: Matches Gaussian noise N(0,I) assumption
Category: PRACTICAL
Code: # Training: normalize to [-1, 1]
images = (images - 0.5) * 2
# OR
images = images * 2 - 1
# Generation: denormalize to [0, 1]
images = (images + 1) / 2
images = torch.clamp(images, 0, 1)
# Why [-1, 1]:
# - Matches noise distribution
# - Centered around 0
# - Symmetric
# - Better gradient flow

Question: What are common issues in diffusion model training and their solutions?
Answer: Blurry samples: • Train longer (patience!) • Use cosine schedule • Add self-attention layers • Verify EMA is being used NaN losses: • Reduce learning rate • Check image normalization • Clip gradients (max_norm=1.0) • Verify beta schedule values Slow convergence: • Increase learning rate • Increase batch size • Use EMA (critical!) • Check data preprocessing OOM errors: • Reduce batch size • Use mixed precision (FP16) • Gradient checkpointing • Train on patches Color shift: • Verify normalization [-1,1] • Check data loading
Category: PRACTICAL

Question: What is latent diffusion and how does it power Stable Diffusion?
Answer: Key idea: Run diffusion in compressed latent space, not pixel space Process: 1. VAE encoder: image to latent (8x compression) 2. Diffusion: Add/remove noise in latent space 3. VAE decoder: latent to image Advantages: • Much faster (8x fewer pixels) • Lower memory • Can handle high-res (512x512, 1024x1024) • Same quality as pixel-space Stable Diffusion architecture: • Text encoder (CLIP) • Latent diffusion U-Net • VAE decoder • Classifier-free guidance Result: State-of-the-art text-to-image
Category: ADVANCED

Question: What is cascaded diffusion?
Answer: Multi-stage generation: Progressively increase resolution Example (DALL-E 2, Imagen): 1. Base model: Generate 64x64 2. Super-res 1: Upsample to 256x256 3. Super-res 2: Upsample to 1024x1024 Each stage: • Separate diffusion model • Conditioned on lower-resolution image • Adds detail at its scale Advantages: • Can reach very high resolutions • Each model specialized for its scale • More efficient than single large model Used in: DALL-E 2, Imagen, Parti
Category: ADVANCED

Question: What are the main applications of diffusion models?
Answer: 1. Text-to-Image: • DALL-E 2, Stable Diffusion, Imagen, Midjourney • Condition on text embeddings • State-of-the-art quality 2. Super-Resolution: • Enhance image quality • Add plausible details • Better than traditional methods 3. Inpainting: • Fill missing regions • Natural completions • Easy with diffusion 4. Image Editing: • Style transfer • Object manipulation • Guided editing 5. Video Generation: • 3D U-Nets (space + time) • Coherent sequences 6. 3D Generation: • DreamFusion (text-to-3D) • NeRF generation 7. Audio/Speech: • WaveGrad, DiffWave
Category: ADVANCED