Title: GANs - Complete Guide

Question: What is a generative model?
Answer: A generative model is a machine learning model that learns to create new data samples resembling the training data. Key characteristics: • Learns P(X), the probability distribution of data • Can generate new samples from learned distribution • Contrasts with discriminative models that learn P(Y|X) Examples: GANs, VAEs, Diffusion Models, Autoregressive models
Category: FUNDAMENTALS

Question: What innovation did GANs introduce to generative modeling?
Answer: GANs (Goodfellow et al., 2014) framed generative modeling as an adversarial game between two neural networks. The innovation: • Generator network creates fake samples • Discriminator network judges real vs fake • Both improve through competition • No need for explicit density function Impact: Enabled sharp, realistic image generation that VAEs struggled with.
Category: FUNDAMENTALS

Question: Explain the GAN counterfeiter-detective analogy.
Answer: Generator = Counterfeiter • Creates fake money (data) to fool detective • Improves technique based on detective's feedback • Goal: Create indistinguishable fakes Discriminator = Detective • Examines money to determine if real or fake • Gets better at spotting subtle differences • Goal: Perfect detection accuracy The dynamic: As one improves, the other must adapt, driving both toward excellence.
Category: FUNDAMENTALS

Question: What is the role of the Generator in a GAN?
Answer: Input: Random noise vector z ~ N(0,I) Output: Synthetic data sample x_fake = G(z) Architecture: Upsampling network (low-dim → high-dim) Training objective: Fool the discriminator • Maximize D(G(z)) • Minimize discriminator's ability to detect fakes Learning signal: Gradient from discriminator
Category: FUNDAMENTALS

Question: What is the role of the Discriminator in a GAN?
Answer: Input: Data sample (real x_real OR fake x_fake) Output: Probability that input is real [0, 1] Architecture: Binary classifier (typically with sigmoid output) Training objective: Classify correctly • Maximize D(x_real) → output close to 1 • Minimize D(x_fake) → output close to 0 Feedback: Provides learning signal to generator
Category: FUNDAMENTALS

Question: Why doesn't the Generator see real data directly?
Answer: The generator never sees real data - it only learns from the discriminator's feedback. Information flow: 1. Generator creates samples from noise 2. Discriminator sees both real and fake 3. Discriminator provides gradient signal 4. Generator updates based on this signal Why this works: • Discriminator encodes information about real data in its gradients • Generator learns implicitly through adversarial feedback • This indirect learning enables flexible generation
Category: FUNDAMENTALS

Question: What does the minimax objective represent in GANs?
Answer: Formula: min_G max_D V(D,G) = E[log D(x)] + E[log(1 - D(G(z)))] Interpretation: • Discriminator (max): Wants to maximize objective by correctly classifying real/fake • Generator (min): Wants to minimize objective by fooling discriminator Equilibrium: Nash equilibrium where neither can improve In practice: Trained via alternating optimization, not simultaneous
Category: FUNDAMENTALS

Question: What is Binary Cross-Entropy (BCE) loss and why is it used in GANs?
Answer: Formula: BCE(y, ŷ) = -[y log(ŷ) + (1-y) log(1-ŷ)] Purpose: Measures difference between predicted and actual binary labels In GANs: • D loss on real: BCE(D(x_real), 1) • D loss on fake: BCE(D(x_fake), 0) • G loss: BCE(D(G(z)), 1) Connection to minimax: BCE implements the GAN objective function
Category: FUNDAMENTALS

Question: Why do we maximize log(D(G(z))) for the generator instead of minimizing log(1 - D(G(z)))?
Answer: Problem with original formulation: min log(1 - D(G(z))) → vanishing gradients early in training When D(G(z)) ≈ 0 (discriminator confident it's fake): • Gradient of log(1 - D(G(z))) ≈ 0 • Generator receives no learning signal Better alternative: max log(D(G(z))) provides strong gradients even when D(G(z)) is small Result: Same optimum, but better gradient flow during training
Category: FUNDAMENTALS

Question: Describe the alternating optimization process in GAN training.
Answer: Training loop: FOR each iteration: 1. Train Discriminator (1 or more steps) • Sample real data and noise • Compute D loss on real and fake • Update D parameters (gradient ascent) 2. Train Generator (1 step) • Sample noise • Generate fakes • Compute G loss • Update G parameters (gradient descent) Why alternating: Simultaneous optimization is unstable; alternating approximates Nash equilibrium
Category: FUNDAMENTALS

Question: What is the ideal discriminator output at GAN equilibrium?
Answer: At equilibrium: D(x) = 0.5 for all x Meaning: • Discriminator cannot distinguish real from fake • Generated distribution matches real distribution • D outputs uniform probability (50/50 guess) In practice: • Rarely achieved perfectly • D(real) and D(fake) should be close • Significant divergence indicates training imbalance Monitoring: Track D(real) and D(fake) to diagnose training issues
Category: FUNDAMENTALS

Question: What is the latent space in GANs?
Answer: Definition: The space of input noise vectors z that the generator maps to data samples Typical characteristics: • Low-dimensional (e.g., 64-512 dimensions) • Usually Gaussian: z ~ N(0, I) • Sometimes uniform: z ~ U(-1, 1) Properties: • Smooth interpolation between samples • Semantic meaning in directions • Can be structured or learned (e.g., disentangled) Usage: Sample z to generate diverse outputs
Category: FUNDAMENTALS

Question: Why use LeakyReLU instead of ReLU in GAN architectures?
Answer: Problem with ReLU: Can cause "dead neurons" with zero gradient LeakyReLU advantages: • Small negative slope (typically 0.2) for negative inputs • Allows gradient flow even for negative values • Prevents neurons from dying completely • Better gradient propagation in deep networks Formula: f(x) = x if x > 0, else α*x (α = 0.2) Result: More stable training and better performance in GANs
Category: FUNDAMENTALS

Question: What does the Tanh activation in the generator output accomplish?
Answer: Purpose: Maps generator output to [-1, 1] range Why Tanh: • Matches normalized image data range • Centers data around 0 (mean = 0) • Bounded output prevents extreme values • Symmetric around origin Preprocessing: Real images normalized to [-1, 1] to match: images = (images - 0.5) * 2 Alternative: Sigmoid [0,1] if data not mean-centered
Category: FUNDAMENTALS

Question: What is the purpose of BatchNorm in the generator?
Answer: Functions: • Normalizes activations within each batch • Reduces internal covariate shift • Stabilizes training dynamics • Allows higher learning rates In generator: • Helps gradient flow through deep network • Prevents mode collapse • Smooths optimization landscape Important: Usually NOT used in discriminator (can reduce its effectiveness) Note: Apply before activation function in generator
Category: FUNDAMENTALS

Question: What is the typical architecture of a vanilla GAN generator for images?
Answer: Example (FashionMNIST): Input: Noise z (64D) ↓ Linear → LeakyReLU → BatchNorm (256) ↓ Linear → LeakyReLU → BatchNorm (512) ↓ Linear → LeakyReLU → BatchNorm (1024) ↓ Linear → Tanh (784 = 28×28) ↓ Output: Image Key patterns: • Progressive upsampling (64 → 784) • BatchNorm after linear layers • Tanh final activation [-1, 1]
Category: VANILLA GAN
Code: class Generator(nn.Module):
def __init__(self):
super().__init__()
self.model = nn.Sequential(
nn.Linear(64, 256),
nn.LeakyReLU(0.2),
nn.BatchNorm1d(256),
nn.Linear(256, 512),
nn.LeakyReLU(0.2),
nn.BatchNorm1d(512),
nn.Linear(512, 1024),
nn.LeakyReLU(0.2),
nn.BatchNorm1d(1024),
nn.Linear(1024, 784),
nn.Tanh()
)

Question: What is the typical architecture of a vanilla GAN discriminator?
Answer: Example (FashionMNIST): Input: Image (784D) ↓ Linear → LeakyReLU (512) ↓ Linear → LeakyReLU (256) ↓ Linear → Sigmoid (1) ↓ Output: Probability [0,1] Key differences from generator: • No BatchNorm (can reduce effectiveness) • Downsampling (784 → 1) • Sigmoid for probability output • Simpler than generator
Category: VANILLA GAN
Code: class Discriminator(nn.Module):
def __init__(self):
super().__init__()
self.model = nn.Sequential(
nn.Linear(784, 512),
nn.LeakyReLU(0.2),
nn.Linear(512, 256),
nn.LeakyReLU(0.2),
nn.Linear(256, 1),
nn.Sigmoid() # Output probability
)

Question: Write the PyTorch code for the discriminator training step in vanilla GAN.
Answer: Process: 1. Sample real data and noise 2. Generate fakes 3. Compute losses for real and fake 4. Update discriminator
Category: VANILLA GAN
Code: # Discriminator training step
optimizer_D.zero_grad()
# Real images - label = 1
real_labels = torch.ones(batch_size, 1)
real_output = discriminator(real_images)
loss_D_real = criterion(real_output, real_labels)
# Fake images - label = 0
noise = torch.randn(batch_size, latent_dim)
fake_images = generator(noise)
fake_labels = torch.zeros(batch_size, 1)
fake_output = discriminator(fake_images.detach())
loss_D_fake = criterion(fake_output, fake_labels)
# Total loss and update
loss_D = loss_D_real + loss_D_fake
loss_D.backward()
optimizer_D.step()

Question: Write the PyTorch code for the generator training step in vanilla GAN.
Answer: Process: 1. Sample noise 2. Generate fakes 3. Compute loss (want D to output 1) 4. Update generator
Category: VANILLA GAN
Code: # Generator training step
optimizer_G.zero_grad()
# Generate fake images
noise = torch.randn(batch_size, latent_dim)
fake_images = generator(noise)
# Generator wants discriminator to output 1
real_labels = torch.ones(batch_size, 1)
fake_output = discriminator(fake_images)
loss_G = criterion(fake_output, real_labels)
loss_G.backward()
optimizer_G.step()

Question: Why do we use .detach() when training the discriminator on fake samples?
Answer: Purpose: Prevent gradients from flowing back to generator Without detach(): fake_output = D(G(z)) → Backprop would update both D and G With detach(): fake_output = D(G(z).detach()) → Backprop only updates D → G is frozen during D training Key principle: Train D and G separately in alternating steps Code: fake_images.detach() creates a new tensor without gradient connection
Category: VANILLA GAN

Question: What are typical hyperparameters for vanilla GAN training?
Answer: Learning rate: 2e-4 (0.0002) Optimizer: Adam • β₁ = 0.5 (lower than default 0.9) • β₂ = 0.999 Batch size: 64-128 Latent dim: 64-512 Update ratio: 1:1 (D:G) Epochs: Varies by dataset (5-200) Note: These are starting points; GANs are sensitive to hyperparameters
Category: VANILLA GAN

Question: Why is β₁ = 0.5 commonly used for Adam in GAN training instead of the default 0.9?
Answer: Reason: Reduces momentum to handle non-stationary optimization GAN dynamics: • Objective changes as both networks update • High momentum can overshoot • Lower β₁ makes optimizer more responsive Default Adam: β₁ = 0.9 (strong momentum) GAN Adam: β₁ = 0.5 (reduced momentum) Effect: More stable training, better adaptation to changing loss landscape
Category: VANILLA GAN

Question: What does it mean if D(real) ≈ 1 and D(fake) ≈ 0 during training?
Answer: Interpretation: Discriminator is too strong Implications: • D perfectly separates real from fake • Generator receives vanishing gradients • G cannot learn effectively • Training stalls Symptoms: • G loss increases or plateaus • Sample quality doesn't improve • D loss near zero Solutions: • Train G more times per D update • Reduce D learning rate • Add noise to D inputs
Category: VANILLA GAN

Question: What does it mean if D(real) ≈ 0.5 and D(fake) ≈ 0.5?
Answer: Two possibilities: 1. Good equilibrium: • G produces realistic samples • D cannot distinguish them • Ideal training state 2. Bad equilibrium (mode collapse): • G found a few samples that fool D • D adapted but G didn't diversify • Need to check sample diversity Diagnosis: Visually inspect generated samples for diversity
Category: VANILLA GAN

Question: How do you sample from a trained GAN generator?
Answer: Inference mode:
Category: VANILLA GAN
Code: # Set generator to evaluation mode
generator.eval()
# Sample random noise
num_samples = 16
noise = torch.randn(num_samples, latent_dim, device=device)
# Generate samples
with torch.no_grad():
fake_images = generator(noise)
# Post-process (Tanh output is [-1,1])
fake_images = (fake_images + 1) / 2 # Scale to [0,1]
# Reshape for visualization
fake_images = fake_images.view(-1, 1, 28, 28)

Question: What is label smoothing and why is it used in GAN training?
Answer: Concept: Use soft labels instead of hard 0/1 Hard labels: • Real: label = 1.0 • Fake: label = 0.0 Smoothed labels: • Real: label = 0.9 (or random in [0.8, 1.0]) • Fake: label = 0.1 (or random in [0.0, 0.2]) Benefits: • Prevents discriminator from becoming too confident • Reduces gradient saturation • Can improve training stability Trade-off: May slightly reduce final sample quality
Category: VANILLA GAN

Question: What is instance noise and how does it help GAN training?
Answer: Technique: Add random noise to discriminator inputs Implementation: • Add Gaussian noise to both real and fake images • Typically with decreasing variance over training • Applies before feeding to discriminator Purpose: • Makes D's job harder • Prevents D from becoming too strong too quickly • Smooths decision boundary • Can reduce mode collapse Annealing: Start with higher noise, decay to zero
Category: VANILLA GAN

Question: What is spectral normalization in discriminators?
Answer: Concept: Normalize weights by their spectral norm (largest singular value) Purpose: • Stabilizes discriminator training • Prevents extreme gradients • Enforces Lipschitz constraint (related to WGAN) • Improves sample quality Application: Applied to each layer's weight matrix Usage in PyTorch: nn.utils.spectral_norm(nn.Linear(...)) Result: More stable training without other tricks
Category: VANILLA GAN

Question: How do convolutional GANs (DCGANs) differ from fully-connected GANs?
Answer: Architecture differences: Fully-connected GAN: • Linear layers • Operates on flattened images • No spatial structure DCGAN (Deep Convolutional GAN): • Transposed convolutions in G (upsampling) • Strided convolutions in D (downsampling) • Preserves spatial structure • BatchNorm in G, not in D • No fully-connected hidden layers Benefits: Better for image generation, captures spatial features
Category: VANILLA GAN

Question: What is progressive growing in GANs?
Answer: Technique: Start training at low resolution, progressively increase Process: 1. Train at 4×4 resolution 2. Add layers for 8×8, train more 3. Continue to 16×16, 32×32, ... 1024×1024 Benefits: • More stable training • Faster initial convergence • Can reach very high resolutions • Used in StyleGAN, Progressive GAN Result: State-of-the-art high-resolution image synthesis
Category: VANILLA GAN

Question: What causes training collapse in vanilla GANs?
Answer: Common causes: 1. Discriminator dominance: D too strong, G gets no signal 2. Generator collapse: G produces limited modes 3. Gradient issues: Vanishing or exploding gradients 4. Hyperparameter issues: Learning rates mismatched 5. Architecture mismatch: D and G capacity imbalanced Symptoms: • Loss curves diverge or oscillate wildly • Generated samples degrade suddenly • Mode collapse (all samples look similar) Recovery: Often requires restarting training with adjustments
Category: VANILLA GAN

Question: How can you diagnose whether a GAN is training well without loss curves?
Answer: Visual inspection: Good signs: • Sample quality improves over epochs • Diverse outputs (not repetitive) • Sharp, recognizable features • Gradual refinement of details Bad signs: • All samples look similar (mode collapse) • No improvement over time • Blurry or noisy outputs • Sudden quality degradation Metrics: • Inception Score • FID (best quantitative measure) • Human evaluation Key insight: Loss values alone are unreliable for GANs
Category: VANILLA GAN

Question: What is the role of the sigmoid activation in the vanilla GAN discriminator?
Answer: Purpose: Convert raw scores to probabilities [0, 1] Function: σ(x) = 1 / (1 + e^(-x)) Properties: • Output = 1: Confident it's real • Output = 0: Confident it's fake • Output = 0.5: Uncertain Connection to BCE loss: Sigmoid output paired with BCE loss implements logistic regression Note: WGAN removes sigmoid to output raw scores instead
Category: VANILLA GAN

Question: What happens if you train the generator multiple times per discriminator update?
Answer: Effect: Generator becomes stronger relative to discriminator Potential benefits: • Prevents D from dominating • Can help if D is too strong • May improve G learning early on Potential problems: • G can exploit D weaknesses • May cause mode collapse • Unstable training dynamics Standard practice: Usually 1:1 ratio for vanilla GAN WGAN-GP: Uses 5:1 (D:G) ratio instead
Category: VANILLA GAN

Question: How do you implement gradient clipping in GAN training and why might you use it?
Answer: Implementation:
Category: VANILLA GAN
Code: # After loss.backward(), before optimizer.step()
torch.nn.utils.clip_grad_norm_(
discriminator.parameters(),
max_norm=1.0
)
optimizer_D.step()
# Or clip by value
torch.nn.utils.clip_grad_value_(
generator.parameters(),
clip_value=0.01
)

Question: What is mode collapse in GANs?
Answer: Definition: Generator learns to produce only a limited variety of samples, ignoring much of the data distribution Manifestation: • Generator produces similar/identical outputs • Fails to explore full diversity of training data • May produce high-quality samples, but lacks variety Example: On MNIST, might only generate 7s instead of all digits 0-9 Cause: Generator finds a few samples that reliably fool discriminator and gets stuck
Category: INSTABILITY

Question: Why does mode collapse occur in vanilla GANs?
Answer: Root causes: 1. No diversity incentive: Objective only rewards fooling D, not diversity 2. Local optima: G finds safe samples that work 3. Discriminator updates: When D adapts, G may jump to different mode 4. Missing feedback: G doesn't know it's missing modes Training dynamics: • G discovers mode A that fools D • D updates to detect mode A • G switches to mode B • Cycle repeats without covering full distribution Problem: No mechanism to encourage exploration
Category: INSTABILITY

Question: How can you detect mode collapse during training?
Answer: Visual inspection: • Generated samples look very similar • Limited variety across different noise inputs • Missing categories or styles from training data Quantitative signals: • Low variance in generated samples • Poor Inception Score (IS) • High FID score (far from real distribution) • D(fake) ≈ 0.5 but samples not diverse Training behavior: • Sudden drop in sample diversity • Generator loss stable but samples degrade • Discriminator easily adapts Best practice: Visualize samples every epoch
Category: INSTABILITY

Question: What are vanishing gradients in GAN training?
Answer: Problem: Generator receives near-zero gradients, stopping learning When it occurs: • Discriminator becomes too confident (D(fake) ≈ 0) • Sigmoid saturation at extreme values • Gradient of log(1 - D(G(z))) ≈ 0 Mathematical cause: When D perfectly separates real/fake: • BCE loss gradient vanishes • No learning signal to generator • Training stalls Symptoms: • G loss plateaus or increases • Sample quality stops improving • D loss approaches zero Solution: Use non-saturating loss: max log(D(G(z)))
Category: INSTABILITY

Question: What causes oscillating losses in vanilla GAN training?
Answer: Root cause: Competing objectives in minimax game Dynamics: 1. D becomes strong → G loss increases 2. G improves → D loss increases 3. D adapts → G loss increases again 4. Cycle continues without convergence Unlike standard ML: • Not optimizing same objective • No shared loss to minimize • Nash equilibrium rarely achieved Characteristics: • Losses go up and down unpredictably • No monotonic improvement • Can't determine convergence from loss alone Implication: Must monitor sample quality, not just loss
Category: INSTABILITY

Question: Why doesn't vanilla GAN loss correlate with sample quality?
Answer: The problem: BCE loss measures discriminator fooling, not sample realism Scenarios showing disconnect: 1. Low G loss, poor samples: • G found exploits in D • D is weak, easily fooled • Samples may be unrealistic 2. High G loss, good samples: • D improved its detection • G samples still high quality • Loss increased but quality didn't drop 3. Mode collapse with balanced loss: • D(fake) ≈ 0.5 (looks good) • But G only produces few modes Consequence: Cannot use loss for model selection or early stopping
Category: INSTABILITY

Question: What is discriminator dominance and how does it affect training?
Answer: Condition: Discriminator is too strong relative to generator Indicators: • D(real) ≈ 1, D(fake) ≈ 0 • D loss near zero • G loss very high or increasing Effects: • Generator receives vanishing gradients • No useful learning signal • Training stalls • Sample quality doesn't improve Causes: • D has more capacity than G • D learning rate too high • Not enough G updates per D update Solutions: • Train G multiple times per D update • Reduce D learning rate • Add noise to D inputs • Use different architecture balance
Category: INSTABILITY

Question: What is the meaning of 'non-convergence' in GANs?
Answer: Definition: GAN training never reaches a stable equilibrium Characteristics: • Losses oscillate indefinitely • No clear stopping point • Both networks keep changing • Nash equilibrium not achieved Contrast with supervised learning: • Supervised: Loss decreases to minimum • GAN: Competing objectives, no shared minimum • Supervised: Convergence is norm • GAN: Non-convergence is common Implications: • Must decide when to stop based on sample quality • May need to save checkpoints throughout training • Cannot rely on loss-based early stopping Reality: True equilibrium rarely achieved in practice
Category: INSTABILITY

Question: How does the Jensen-Shannon divergence contribute to GAN instability?
Answer: JS divergence in vanilla GAN: Implicitly optimized through BCE loss Problem with JS divergence: When real and fake distributions don't overlap: • JS divergence = log(2) (constant) • Gradient becomes zero • No learning signal Why distributions don't overlap: • Real data: Low-dimensional manifold in high-dim space • Generated data: Different manifold • Probability of overlap ≈ 0 Result: Frequent gradient saturation and instability WGAN solution: Use Wasserstein distance instead, which provides gradients even with no overlap
Category: INSTABILITY

Question: What is the relationship between discriminator capacity and training stability?
Answer: Too weak discriminator: • Cannot provide useful feedback • G doesn't learn properly • Poor sample quality Too strong discriminator: • Perfectly separates real/fake • Vanishing gradients to G • Training stalls Ideal balance: • D slightly ahead of G • Provides gradient signal • Not so strong that it saturates Capacity matching: • D and G should have similar capacity • If D has more layers/parameters, reduce learning rate • Adjust update ratios (k D updates per 1 G update) Rule of thumb: D should be just good enough to guide G
Category: INSTABILITY

Question: What techniques can mitigate mode collapse?
Answer: Architectural approaches: • Minibatch discrimination (compare samples in batch) • Unrolled GAN (look ahead in discriminator updates) Training techniques: • Experience replay (show D old G samples) • Multiple generators • Feature matching (match statistics instead of fooling D) Objective modifications: • Wasserstein loss (WGAN) • Various divergence measures Regularization: • Spectral normalization • Gradient penalty Most effective: WGAN-GP significantly reduces mode collapse through stable training
Category: INSTABILITY

Question: Why is it difficult to debug vanilla GAN training?
Answer: Challenges: 1. Unreliable metrics: Loss doesn't indicate quality 2. Visual inspection required: Tedious and subjective 3. Many failure modes: Mode collapse, divergence, poor quality 4. Coupled training: Hard to isolate D vs G issues 5. Hyperparameter sensitivity: Small changes cause big effects 6. Stochastic outcomes: Different results from different seeds Debugging process: • Generate samples frequently • Track D(real) and D(fake) • Monitor gradient norms • Compare to known working setups • Try multiple random seeds Reality: Often requires experience and trial-and-error
Category: INSTABILITY

Question: What motivated the development of Wasserstein GAN?
Answer: Problems with vanilla GAN: • Training instability • Mode collapse • Vanishing gradients • Non-meaningful loss metric Root cause identified: Jensen-Shannon divergence has issues when distributions don't overlap WGAN innovation (Arjovsky et al., 2017): • Use Wasserstein distance (Earth Mover's Distance) • Provides meaningful gradients everywhere • Continuous metric correlating with quality Theoretical foundation: Wasserstein distance is smoother and better-behaved for GAN training Result: More stable training with interpretable progress metric
Category: WGAN

Question: What is Earth Mover's Distance (Wasserstein distance)?
Answer: Intuition: Minimum cost to transform one distribution into another Analogy: Two piles of dirt with different shapes • How much work to reshape pile A into pile B? • Work = amount of dirt × distance moved • Wasserstein distance = minimum total work Mathematical: W(p_r, p_g) = inf E[||x - y||] over all joint distributions Key property: • Continuous and differentiable everywhere • Provides gradients even when distributions don't overlap • Lower distance = more similar distributions Advantage over JS divergence: Never saturates
Category: WGAN

Question: Why does Wasserstein distance provide better gradients than JS divergence?
Answer: JS divergence problem: When p_r and p_g have disjoint support: • JS(p_r || p_g) = log(2) (constant) • Gradient = 0 • No learning signal Wasserstein distance advantage: Even with disjoint support: • W(p_r, p_g) > 0 and varies smoothly • Always provides gradient • Gradient points toward improving distribution Geometric interpretation: • JS: "distributions match or they don't" • W: "how far apart are they?" Training impact: Generator can always receive useful feedback, enabling stable learning
Category: WGAN

Question: What is the Kantorovich-Rubinstein duality?
Answer: Purpose: Makes Wasserstein distance computationally tractable Dual formulation: W(p_r, p_g) = sup E[f(x)] - E[f(y)] where f is 1-Lipschitz Meaning: • Instead of computing inf over joint distributions (hard) • Compute sup over 1-Lipschitz functions (feasible) • Neural network approximates optimal f In WGAN: • Critic network = learned f • Training maximizes E[C(real)] - E[C(fake)] • Subject to Lipschitz constraint Key insight: This duality enables practical WGAN implementation
Category: WGAN

Question: What is a Lipschitz constraint and why is it needed in WGAN?
Answer: Definition: Function f is K-Lipschitz if: |f(x₁) - f(x₂)| ≤ K||x₁ - x₂|| for all x₁, x₂ For 1-Lipschitz (K=1): Function cannot change faster than input changes Why needed in WGAN: • Kantorovich-Rubinstein duality requires 1-Lipschitz • Ensures critic gradients are bounded • Prevents critic from growing arbitrarily large • Necessary for valid Wasserstein approximation Gradient interpretation: 1-Lipschitz means ||∇f||₂ ≤ 1 everywhere Enforcement: • Original WGAN: Weight clipping • WGAN-GP: Gradient penalty
Category: WGAN

Question: How does WGAN enforce the Lipschitz constraint through weight clipping?
Answer: Method: Clamp all critic weights to small range [-c, c]
Category: WGAN
Code: # After each critic update
for param in critic.parameters():
param.data.clamp_(-0.01, 0.01)
# Typical values: c = 0.01 or 0.001

Question: What are the problems with weight clipping in WGAN?
Answer: Issues identified: 1. Capacity limitation: • Constrains weights to tiny range • Reduces network expressiveness • Limits critic's ability to learn complex functions 2. Gradient pathologies: • Can lead to vanishing or exploding gradients • Creates pathological value surfaces • Layers tend toward extreme values 3. Slow convergence: • Critic needs many iterations to reach optimum • Training takes longer 4. Hyperparameter sensitivity: • Clipping threshold c is hard to tune • Wrong value causes problems Solution: WGAN-GP replaces clipping with gradient penalty
Category: WGAN

Question: What is the critic in WGAN and how does it differ from a discriminator?
Answer: Discriminator (vanilla GAN): • Outputs probability via sigmoid: D(x) ∈ [0,1] • Classifies real vs fake • Trained with BCE loss Critic (WGAN): • Outputs raw score: C(x) ∈ ℝ (unbounded) • Scores "realness" (higher = more real) • Trained to maximize W-distance approximation • NO sigmoid activation Why "critic": • Not doing binary classification • Scoring/critiquing samples • Different theoretical role Architecture difference: • Remove final sigmoid • Otherwise can be similar structure
Category: WGAN

Question: Write the WGAN critic loss function.
Answer: Objective: Maximize Wasserstein distance approximation
Category: WGAN
Code: # Critic wants to maximize:
# E[C(real)] - E[C(fake)]
# But we minimize negative:
critic_real = critic(real_images).mean()
critic_fake = critic(fake_images).mean()
loss_C = -(critic_real - critic_fake)
# Or equivalently:
loss_C = critic_fake - critic_real
# Then apply weight clipping
for p in critic.parameters():
p.data.clamp_(-0.01, 0.01)

Question: Write the WGAN generator loss function.
Answer: Objective: Maximize critic score on generated samples
Category: WGAN
Code: # Generator wants to maximize:
# E[C(G(z))]
# Minimize negative:
fake_images = generator(noise)
critic_fake = critic(fake_images).mean()
loss_G = -critic_fake
# No labels involved!
# Just want high critic scores

Question: Why does WGAN use more critic updates per generator update?
Answer: Reason: Critic must be near-optimal for valid W-distance approximation Theoretical requirement: • Wasserstein distance requires supremum over all 1-Lipschitz functions • Critic should approximate optimal f • Under-trained critic gives poor approximation Practical implementation: • n_critic = 5 (5 critic updates per G update) • Ensures critic stays ahead of generator • More stable training Contrast with vanilla GAN: • Vanilla: Often 1:1 ratio • WGAN: 5:1 ratio standard Trade-off: Slower training but much more stable
Category: WGAN

Question: What is the Wasserstein distance as a training metric?
Answer: Definition: Approximated by E[C(real)] - E[C(fake)] Properties: • Continuous during training • Correlates with sample quality • Lower W-distance = better generation • Monotonically improves (usually) Usage: • Track over epochs for convergence • Compare different models • Determine when to stop training • Debug training issues Advantage: Unlike BCE loss, actually meaningful! Typical behavior: • Starts high (large distance) • Decreases as G improves • Stabilizes when converged
Category: WGAN

Question: How do you implement the WGAN training loop?
Answer: Key differences from vanilla GAN:
Category: WGAN
Code: for epoch in range(num_epochs):
for real_images, _ in dataloader:
# Train Critic (n_critic times)
for _ in range(n_critic):
optimizer_C.zero_grad()
noise = torch.randn(batch_size, latent_dim)
fake = generator(noise)
critic_real = critic(real_images).mean()
critic_fake = critic(fake).mean()
loss_C = critic_fake - critic_real
loss_C.backward()
optimizer_C.step()
# Weight clipping (WGAN)
for p in critic.parameters():
p.data.clamp_(-0.01, 0.01)
# Train Generator (once)
optimizer_G.zero_grad()
noise = torch.randn(batch_size, latent_dim)
fake = generator(noise)
loss_G = -critic(fake).mean()
loss_G.backward()
optimizer_G.step()

Question: What are typical WGAN hyperparameters?
Answer: Learning rate: 1e-4 or 5e-5 (lower than vanilla GAN) Optimizer: RMSprop (original paper) or Adam • If Adam: β₁ = 0.5, β₂ = 0.9 Critic updates: n_critic = 5 Weight clipping: c = 0.01 (range [-0.01, 0.01]) Batch size: 64 Why lower learning rate: • More stable optimization • Critic needs careful training • Prevents overshooting Note: WGAN-GP uses same hyperparameters but replaces clipping with gradient penalty
Category: WGAN

Question: How does training stability improve in WGAN compared to vanilla GAN?
Answer: Stability improvements: 1. Smoother loss curves: • W-distance decreases monotonically • Less oscillation • Clear convergence trend 2. No mode collapse (mostly): • Better gradient flow • Explores distribution more fully 3. Meaningful feedback: • W-distance indicates progress • Can determine convergence • Easier to debug 4. No vanishing gradients: • Critic doesn't saturate • Generator always gets signal Empirical observation: • Works across wider range of architectures • Less hyperparameter tuning needed • Training succeeds more consistently
Category: WGAN

Question: Can you use ReLU or other activations in WGAN critic?
Answer: Yes, but carefully: Recommended: • LeakyReLU (negative slope 0.2) • ELU (Exponential Linear Unit) • SELU (Scaled ELU) Avoid: • ReLU (can cause dead neurons) • Sigmoid/Tanh in hidden layers (not necessary) NO final activation: • Critic outputs raw scores • No sigmoid, no tanh • Unbounded real numbers Why LeakyReLU preferred: • Allows negative values • Better gradient flow • Reduces dead neurons BatchNorm: Generally avoid in critic (can interfere with Lipschitz constraint)
Category: WGAN

Question: What is the theoretical optimal discriminator in vanilla GAN?
Answer: At equilibrium: D*(x) = p_data(x) / (p_data(x) + p_g(x)) When p_g = p_data (perfect generator): D*(x) = 1/2 for all x Meaning: • Optimal D outputs 0.5 (uniform probability) • Cannot distinguish real from fake • Generator has learned true distribution In practice: • Rarely achieved • Used for theoretical analysis • Guides understanding of GAN dynamics Implication: At equilibrium, discriminator is maximally uncertain
Category: WGAN

Question: What does it mean if WGAN Wasserstein distance stops decreasing?
Answer: Possible interpretations: 1. Convergence (good): • Generator learned distribution well • Reached optimal approximation • Check sample quality 2. Generator capacity limit: • Architecture insufficient for data • Need more complex model 3. Training issue: • Learning rate too low • Gradient clipping too aggressive • Critic not optimal Diagnosis: • Inspect generated samples • Check if samples are diverse and realistic • Compare to training data Action: If samples good, training successful. If not, investigate hyperparameters.
Category: WGAN

Question: Why might WGAN training be slower than vanilla GAN?
Answer: Reasons for slower training: 1. More critic updates: • 5 critic updates per 1 generator update • 5× more critic forward/backward passes 2. Lower learning rate: • 1e-4 vs 2e-4 for vanilla GAN • Slower parameter updates 3. Weight clipping overhead: • Extra operation after each update • Though computationally cheap 4. Conservative optimization: • Designed for stability over speed • Critic must be near-optimal Trade-off: • Slower per-epoch training • But more reliable convergence • Often reaches better results overall
Category: WGAN

Question: What is the gradient penalty in WGAN-GP?
Answer: Purpose: Enforce 1-Lipschitz constraint without weight clipping Formula: GP = λ · E[(||∇_x̂ C(x̂)||₂ - 1)²] Components: • λ = penalty coefficient (typically 10) • x̂ = interpolated samples between real and fake • ||∇_x̂ C(x̂)||₂ = L2 norm of critic gradient • Penalty = squared deviation from norm = 1 Intuition: Penalize critic when its gradient norm deviates from 1 Effect: Critic learns smooth function with bounded gradients Advantage: More principled than weight clipping
Category: WGAN-GP

Question: Why are interpolated samples used for gradient penalty?
Answer: Theoretical motivation: Optimal critic has gradient norm = 1 almost everywhere between p_r and p_g Interpolation formula: x̂ = ε·x_real + (1-ε)·x_fake, where ε ~ U(0,1) Why interpolations: • Samples the space between distributions • This is where Lipschitz constraint matters most • Computationally efficient • Empirically works better than uniform sampling Geometric insight: • Real and fake data lie on manifolds • Straight-line interpolations sample between them • Ensures smooth critic behavior on this path Result: Effective constraint enforcement
Category: WGAN-GP

Question: Implement the gradient penalty computation for WGAN-GP.
Answer: Complete implementation:
Category: WGAN-GP
Code: def compute_gradient_penalty(critic, real, fake, device):
batch_size = real.size(0)
# Random interpolation coefficient
epsilon = torch.rand(batch_size, 1, device=device)
epsilon = epsilon.expand_as(real)
# Interpolated samples
interpolated = epsilon * real + (1 - epsilon) * fake
interpolated.requires_grad_(True)
# Critic scores
critic_interpolated = critic(interpolated)
# Compute gradients
gradients = torch.autograd.grad(
outputs=critic_interpolated,
inputs=interpolated,
grad_outputs=torch.ones_like(critic_interpolated),
create_graph=True,
retain_graph=True,
)[0]
# Gradient norm
gradients = gradients.view(batch_size, -1)
gradient_norm = gradients.norm(2, dim=1)
# Penalty
penalty = ((gradient_norm - 1) ** 2).mean()
return penalty

Question: What is the role of create_graph=True in gradient penalty?
Answer: Purpose: Enable backpropagation through gradient computation How it works: • Normally: Gradients are not part of computation graph • create_graph=True: Gradients become graph nodes • Enables: Gradient of gradient (second-order derivative) Why needed for GP: 1. Compute ∇C(x̂) w.r.t. interpolated samples 2. Compute penalty based on gradient norm 3. Backprop through penalty to update critic 4. Requires differentiating through step 1 Without create_graph: • Cannot backprop through gradient computation • GP would not affect critic training • Constraint not enforced Cost: Higher memory usage, slower computation
Category: WGAN-GP

Question: Why is λ = 10 the standard gradient penalty coefficient?
Answer: Value: λ = 10 is empirically optimal across many datasets Reasoning: • Too low (λ < 1): Insufficient constraint enforcement • Too high (λ > 100): Dominates other loss terms • λ = 10: Good balance between constraint and optimization Effect on loss: Loss_C = W-distance + λ·GP • W-distance typically O(1-10) • GP typically O(0.1-1.0) • λ=10 makes them comparable magnitude Robustness: • Works well without tuning • Generalizes across architectures • Rarely needs adjustment From paper: Gulrajani et al. found λ=10 optimal experimentally
Category: WGAN-GP

Question: Write the complete WGAN-GP critic training step.
Answer: Full implementation with gradient penalty:
Category: WGAN-GP
Code: # Hyperparameters
lambda_gp = 10
n_critic = 5
for _ in range(n_critic):
optimizer_C.zero_grad()
# Real and fake samples
noise = torch.randn(batch_size, latent_dim, device=device)
fake = generator(noise)
# Critic scores
critic_real = critic(real_images).mean()
critic_fake = critic(fake.detach()).mean()
# Gradient penalty
gp = compute_gradient_penalty(
critic, real_images, fake.detach(), device
)
# Total critic loss
loss_C = critic_fake - critic_real + lambda_gp * gp
loss_C.backward()
optimizer_C.step()
# NO weight clipping needed!

Question: What are the advantages of WGAN-GP over vanilla WGAN?
Answer: Improvements over weight clipping: 1. No capacity reduction: • Weights not constrained to tiny range • Critic can learn complex functions • Better approximation of optimal f 2. Better gradient behavior: • No pathological value surfaces • Smoother optimization • More stable training 3. Faster convergence: • Critic trains more efficiently • Reaches optimum quicker • Overall faster training 4. Less hyperparameter tuning: • λ=10 works broadly • No clipping threshold to tune 5. Higher quality samples: • Better critic → better generator • State-of-the-art results Cost: Gradient computation overhead (acceptable trade-off)
Category: WGAN-GP

Question: Why doesn't WGAN-GP use BatchNorm in the critic?
Answer: Problem: BatchNorm interferes with gradient penalty Why BatchNorm is incompatible: • BatchNorm couples samples within a batch • Gradient penalty requires independent gradient for each sample • Coupling violates independence assumption • Can lead to incorrect gradient norms Technical issue: • ∇C(x̂) should depend only on x̂ • BatchNorm makes it depend on entire batch • Breaks gradient penalty computation Alternatives: • Layer Normalization (LayerNorm) • Instance Normalization • Group Normalization • Or no normalization in critic Generator: Can still use BatchNorm (GP only in critic)
Category: WGAN-GP

Question: What should the gradient penalty value be during training?
Answer: Typical range: 0.1 to 5.0 Ideal behavior: • Starts higher (2-10) • Decreases as training progresses • Stabilizes around 0.5-2.0 What values indicate: • GP < 0.1: Constraint already satisfied (good) • GP = 0.5-2.0: Healthy training (ideal) • GP > 10: Critic gradients far from norm=1 (concerning) • GP increasing: Possible training issue Monitoring: • Track GP over time • Should trend downward or stabilize • Stable low GP = successful constraint enforcement Note: Even small GP values contribute significantly (multiplied by λ=10)
Category: WGAN-GP

Question: How does WGAN-GP achieve state-of-the-art stability?
Answer: Stability factors: 1. Wasserstein distance: • Continuous gradients everywhere • Meaningful progress metric 2. Gradient penalty: • Smooth critic function • Bounded gradients • No saturation 3. No weight constraints: • Full model capacity • Better function approximation 4. 5:1 update ratio: • Well-optimized critic • Accurate W-distance approximation Combined effect: • Smooth, monotonic loss curves • Reliable convergence • Minimal mode collapse • Works across architectures Empirical results: 2-5× reduction in loss volatility vs vanilla GAN
Category: WGAN-GP

Question: What is the complete WGAN-GP generator training step?
Answer: Generator update (after n_critic critic updates):
Category: WGAN-GP
Code: optimizer_G.zero_grad()
# Generate fake samples
noise = torch.randn(batch_size, latent_dim, device=device)
fake_images = generator(noise)
# Generator loss: maximize critic score
loss_G = -critic(fake_images).mean()
loss_G.backward()
optimizer_G.step()
# Track Wasserstein distance estimate
with torch.no_grad():
critic_real = critic(real_images).mean()
critic_fake = critic(fake_images).mean()
w_distance = critic_real - critic_fake
print(f'W-distance: {w_distance:.4f}')

Question: How do you monitor WGAN-GP training progress?
Answer: Key metrics to track: 1. Wasserstein distance: • Should decrease over epochs • Lower = better • Correlates with sample quality 2. Gradient penalty: • Should stabilize at low value (0.5-2.0) • Indicates successful constraint enforcement 3. Critic loss: • Combined: W-dist + λ·GP • Should decrease smoothly 4. Generator loss: • Should decrease (become more negative) • Indicates improving samples 5. Visual samples: • Still the gold standard • Generate grid every epoch Good training: All metrics improve smoothly without oscillation
Category: WGAN-GP

Question: What are typical WGAN-GP hyperparameters?
Answer: Standard configuration: Learning rate: 1e-4 (both G and C) Optimizer: Adam • β₁ = 0.5 (or 0.0) • β₂ = 0.9 (or 0.999) Gradient penalty: λ = 10 Critic updates: n_critic = 5 Batch size: 64 Latent dim: 128 Robustness: • These work across most datasets • Minimal tuning needed • Stable across random seeds Compared to vanilla GAN: Much less hyperparameter sensitivity
Category: WGAN-GP

Question: Can WGAN-GP still experience mode collapse?
Answer: Answer: Rarely, but possible Why it's reduced: • Better gradient flow prevents collapse • Wasserstein distance encourages diversity • Smooth critic behavior • No gradient saturation When it might still occur: • Extremely limited generator capacity • Pathological data distributions • Implementation bugs Comparison: • Vanilla GAN: Mode collapse common • WGAN: Less common • WGAN-GP: Rare Diagnosis: • Wasserstein distance will show it (won't decrease) • Visual inspection still necessary Overall: WGAN-GP significantly mitigates but doesn't eliminate mode collapse
Category: WGAN-GP

Question: What is the computational cost of gradient penalty?
Answer: Additional operations: 1. Generate interpolated samples 2. Forward pass through critic 3. Compute gradients w.r.t. inputs (expensive) 4. Compute gradient norm 5. Compute penalty and backprop Time overhead: • ~20-40% slower per iteration than WGAN • Extra backward pass for gradient computation • create_graph=True increases memory Memory overhead: • Must store computation graph for gradients • ~1.5-2× memory usage vs vanilla WGAN Trade-off: • Slower per-iteration • But better convergence • Higher quality results • Worth it for stability Optimization: Can compute GP less frequently (every k iterations)
Category: WGAN-GP

Question: What problem does CycleGAN solve?
Answer: Challenge: Image-to-image translation without paired training data Traditional approach (pix2pix): • Requires paired examples: (input A, output B) • Expensive or impossible to obtain • Example: Can't get exact same photo as painting CycleGAN solution: • Learn mapping between domains without pairs • Only needs separate collections from each domain • Example: Collection of photos + collection of paintings Applications: • Style transfer (photos ↔ paintings) • Object transfiguration (horses ↔ zebras) • Season transfer (summer ↔ winter) • Domain adaptation (synthetic ↔ real)
Category: CYCLEGAN

Question: What is cycle consistency and why is it needed?
Answer: Concept: If you translate from X→Y→X, you should recover the original Forward cycle: x → G(x) → F(G(x)) ≈ x Backward cycle: y → F(y) → G(F(y)) ≈ y Why needed: • Without pairs, infinite valid mappings exist • Cycle consistency constrains the solution space • Ensures content preservation during translation • Prevents mode collapse to random outputs Intuition: Translation should preserve information - if you can reverse it and get the original back, the mapping is meaningful
Category: CYCLEGAN

Question: What is the cycle consistency loss formula?
Answer: Formula: L_cyc(G, F) = E[||F(G(x)) - x||₁] + E[||G(F(y)) - y||₁] Components: • G: X → Y generator • F: Y → X generator • ||·||₁: L1 (absolute difference) norm • First term: Forward cycle loss • Second term: Backward cycle loss Why L1 norm: • Encourages sharp reconstructions • Less blurry than L2 • Preserves details better Typical weight: λ = 10 (multiplier for cycle loss)
Category: CYCLEGAN

Question: Describe the CycleGAN architecture components.
Answer: Two generators: • G: X → Y (e.g., photo → painting) • F: Y → X (e.g., painting → photo) Two discriminators: • D_X: Distinguishes real X from F(y) • D_Y: Distinguishes real Y from G(x) Generator architecture: • Encoder-decoder with skip connections • Residual blocks in middle • Preserves spatial structure Discriminator architecture: • PatchGAN (classifies image patches) • More efficient than full-image classification • Encourages local realism Total parameters: 4 networks (2 G + 2 D)
Category: CYCLEGAN

Question: What is the complete CycleGAN loss function?
Answer: Total objective: L = L_GAN(G, D_Y) + L_GAN(F, D_X) + λ · L_cyc(G, F) Adversarial losses: • L_GAN(G, D_Y): G fools D_Y • L_GAN(F, D_X): F fools D_X • Standard GAN losses (can use LSGAN, vanilla, etc.) Cycle consistency: • λ · L_cyc(G, F): Reconstruction constraint • λ = 10 typically Balance: • Adversarial: Makes translations realistic • Cycle: Preserves content Training: Alternate between updating G+F and D_X+D_Y
Category: CYCLEGAN

Question: Write the CycleGAN generator training step.
Answer: Update both generators simultaneously:
Category: CYCLEGAN
Code: optimizer_G.zero_grad()
# Forward cycle: X -> Y -> X
fake_Y = G(real_X)
reconstructed_X = F(fake_Y)
loss_cycle_X = L1(reconstructed_X, real_X)
# Backward cycle: Y -> X -> Y
fake_X = F(real_Y)
reconstructed_Y = G(fake_X)
loss_cycle_Y = L1(reconstructed_Y, real_Y)
# Adversarial losses
loss_G_adv = adversarial_loss(D_Y(fake_Y), True)
loss_F_adv = adversarial_loss(D_X(fake_X), True)
# Total generator loss
loss_G = (loss_G_adv + loss_F_adv +
lambda_cyc * (loss_cycle_X + loss_cycle_Y))
loss_G.backward()
optimizer_G.step()

Question: What types of transformations can CycleGAN handle?
Answer: Works well for: • Texture/appearance changes • Color/style transfer • Seasonal changes (summer ↔ winter) • Artistic style (photo ↔ painting) • Simple shape variations (horses ↔ zebras) Struggles with: • Geometric transformations • Significant shape changes • Adding/removing objects • Structural modifications • Tasks requiring spatial correspondence Limitation: Preserves geometric structure by design Example success: Horse → Zebra (just texture change) Example failure: Dog → Cat (different face structure)
Category: CYCLEGAN

Question: What is identity loss in CycleGAN and when is it used?
Answer: Identity loss: Generators should preserve images from target domain Formula: L_identity = E[||G(y) - y||₁] + E[||F(x) - x||₁] Meaning: • If G gets a painting (Y domain), output unchanged • If F gets a photo (X domain), output unchanged • Generator learns to identify when no translation needed When to use: • Color/tone preservation important • Prevent color shift artifacts • Photo realism tasks Weight: λ_identity = 0.5 (half of cycle loss weight) Effect: Reduces color distortion, improves stability
Category: CYCLEGAN

Question: What is a PatchGAN discriminator?
Answer: Concept: Classify image patches instead of entire image How it works: • Discriminator outputs N×N grid of predictions • Each value classifies a patch as real/fake • Final loss is average over all patches Advantages: • Fewer parameters than full-image discriminator • Encourages local texture realism • Works well for high-resolution images • Faster training Typical patch size: 70×70 pixels (effective receptive field) Output shape: E.g., 30×30 for 256×256 input Usage: Standard in CycleGAN and pix2pix
Category: CYCLEGAN

Question: How does CycleGAN prevent mode collapse?
Answer: Mechanisms: 1. Cycle consistency: • Forces diverse outputs • Must preserve enough information to reconstruct • Can't collapse to single mode 2. Two discriminators: • Each evaluates different domain • Provides richer feedback • Harder to fool both simultaneously 3. Reconstruction constraint: • Penalizes information loss • Encourages exploration of input space Result: More stable than vanilla GAN Still possible: If both generators collapse to same limited mapping
Category: CYCLEGAN

Question: What is Inception Score (IS) and what does it measure?
Answer: Purpose: Evaluate GAN sample quality and diversity Formula: IS = exp(E[KL(p(y|x) || p(y))]) Uses pre-trained Inception network: • p(y|x): Conditional class distribution (quality) • p(y): Marginal class distribution (diversity) Interpretation: • Higher IS = better quality + diversity • Good samples: high confidence per image • Diverse samples: uniform overall distribution Typical values: • Random noise: IS ≈ 1 • Good GAN on CIFAR-10: IS ≈ 8-10 • Real CIFAR-10: IS ≈ 11.2
Category: EVALUATION

Question: What are the limitations of Inception Score?
Answer: Major problems: 1. No real data comparison: • Only evaluates generated samples • Doesn't check similarity to training data • Can't detect overfitting 2. ImageNet bias: • Requires images in Inception's domain • Doesn't work for non-ImageNet data • Biased toward certain object types 3. Mode collapse blind spot: • Can be fooled by one perfect sample per class • Doesn't measure intra-class diversity 4. Adversarial examples: • Can be gamed by fooling Inception • High IS doesn't guarantee visual quality Conclusion: Use with other metrics, not alone
Category: EVALUATION

Question: What is Fréchet Inception Distance (FID)?
Answer: Purpose: Measure distance between real and generated distributions Method: 1. Extract features from Inception network 2. Model both distributions as Gaussians 3. Compute Fréchet distance between them Formula: FID = ||μ_r - μ_g||² + Tr(Σ_r + Σ_g - 2√(Σ_r Σ_g)) Interpretation: • Lower FID = better (closer to real) • FID = 0: Identical distributions • Typical good: FID < 10-50 (dataset-dependent) Advantages: • Compares to real data directly • More sensitive to mode collapse • Correlates better with human judgment
Category: EVALUATION

Question: Why is FID better than Inception Score?
Answer: FID advantages: 1. Uses real data: • Compares generated to actual distribution • Can detect distribution mismatch • Meaningful baseline 2. Detects mode collapse: • Sensitive to missing modes • Captures diversity better • Penalizes incomplete coverage 3. Better human correlation: • Research shows FID matches human judgment • More reliable quality indicator 4. Consistent across datasets: • Works for various data types • Robust metric Why IS still used: • Faster to compute • Doesn't require real data storage • Useful complementary metric
Category: EVALUATION

Question: How do you compute FID in practice?
Answer: Steps:
Category: EVALUATION
Code: # 1. Load pre-trained Inception model
inception = torchvision.models.inception_v3(pretrained=True)
inception.fc = nn.Identity() # Remove classifier
inception.eval()
# 2. Extract features
def get_features(images):
with torch.no_grad():
features = inception(images)
return features.cpu().numpy()
real_features = get_features(real_images) # N x 2048
fake_features = get_features(fake_images) # M x 2048
# 3. Compute statistics
mu_real = np.mean(real_features, axis=0)
mu_fake = np.mean(fake_features, axis=0)
sigma_real = np.cov(real_features, rowvar=False)
sigma_fake = np.cov(fake_features, rowvar=False)
# 4. Compute FID
diff = mu_real - mu_fake
covmean = sqrtm(sigma_real @ sigma_fake)
fid = diff @ diff + np.trace(sigma_real + sigma_fake - 2*covmean)

Question: How many samples do you need for reliable FID calculation?
Answer: Recommended minimum: 10,000 samples (both real and generated) Why so many: • Computing covariance matrix requires many samples • Small sample size → noisy estimates • FID variance decreases with more samples Sample size effects: • 1,000 samples: High variance, unreliable • 5,000 samples: Moderate variance • 10,000+ samples: Stable, reliable estimates • 50,000 samples: Best practice for publications Practical consideration: • Use same number for real and fake • Use entire test set if < 10k • Generate 50k samples if possible Computation time: Increases with sample count
Category: EVALUATION

Question: What does it mean if FID score is increasing during training?
Answer: Interpretation: Generated distribution moving away from real Possible causes: 1. Mode collapse: • Generator focusing on fewer modes • Losing diversity • Distribution narrowing 2. Training instability: • Generator or discriminator diverging • Adversarial dynamics broken • Need to adjust hyperparameters 3. Overfitting: • Memorizing training set • Poor generalization Actions: • Stop training (use earlier checkpoint) • Reduce learning rate • Check for mode collapse visually • Review training dynamics
Category: EVALUATION

Question: Compare Inception Score and FID side-by-side.
Answer: Inception Score (IS): • Higher is better • Measures quality + diversity • No real data comparison • Faster to compute (~5k samples) • Prone to mode collapse blind spots • ImageNet-biased Fréchet Inception Distance (FID): • Lower is better • Measures distribution distance • Compares to real data • Slower to compute (~10k+ samples) • Detects mode collapse well • Still Inception-dependent Best practice: • Use FID as primary metric • Supplement with IS • Always include visual inspection • Report both for completeness
Category: EVALUATION

Question: What is precision and recall for GANs?
Answer: Precision: What fraction of generated samples are realistic? • High precision = high quality • Low precision = generating unrealistic samples Recall: What fraction of real distribution is covered? • High recall = high diversity • Low recall = mode collapse Trade-off: • High precision, low recall: Few perfect samples (mode collapse) • Low precision, high recall: Many poor samples • Both high: Ideal GAN Computation: • Define manifolds in feature space • Precision: % fake samples near real manifold • Recall: % real manifold covered by fakes Advantage: Separates quality from diversity
Category: EVALUATION

Question: Why can't we just use loss values to evaluate GANs?
Answer: Fundamental problems: 1. Competing objectives: • G and D have opposite goals • No single loss to minimize • Oscillation is normal 2. Loss doesn't correlate with quality: • Low G loss ≠ good samples • Balanced losses ≠ convergence • Can have mode collapse with good losses 3. Non-convergence: • GANs don't converge to fixed point • Losses oscillate indefinitely • No clear stopping criterion What works: • Visual inspection • FID/IS metrics • Downstream task performance • Human evaluation Reality: GAN evaluation remains challenging
Category: EVALUATION

Question: What are the key implementation differences between vanilla GAN and WGAN-GP?
Answer: Discriminator → Critic: • Remove sigmoid activation • Output unbounded real numbers Loss function: • No BCE loss • Use Wasserstein loss: -C(fake) for G, C(fake) - C(real) for C Gradient penalty: • Add compute_gradient_penalty() function • Apply to interpolated samples • Requires create_graph=True Training ratio: • n_critic = 5 (instead of 1:1) • Update C five times per G update No weight clipping in WGAN-GP: • Replaced by gradient penalty • No .clamp_() needed
Category: IMPLEMENTATION

Question: What are common debugging steps when GAN training fails?
Answer: 1. Check architecture: • Verify no sigmoid in WGAN critic • Ensure Tanh in generator output • Confirm input normalization [-1, 1] 2. Verify losses: • Check signs (G minimizes negative critic score) • Ensure losses computed correctly • Validate gradient flow 3. Monitor metrics: • Track D(real) and D(fake) • Watch for discriminator dominance • Check gradient norms 4. Visual inspection: • Generate samples every epoch • Look for mode collapse • Check for artifacts 5. Try baseline: • Use known working hyperparameters • Test on simple dataset (MNIST) • Isolate the issue
Category: IMPLEMENTATION

Question: How should you normalize images for GAN training?
Answer: Standard normalization for GANs:
Category: IMPLEMENTATION
Code: # Convert [0, 1] to [-1, 1]
# Matches generator Tanh output
images = (images - 0.5) * 2
# Or equivalently:
images = images * 2 - 1
# For denormalization (visualization):
images = (images + 1) / 2 # [-1,1] -> [0,1]
# Why [-1, 1]:
# - Tanh outputs in this range
# - Centered around 0 (mean = 0)
# - Symmetric
# - Better gradient flow

Question: What batch size should you use for GAN training?
Answer: Typical range: 64-128 Considerations: Larger batch (128-256): • More stable gradients • Better for BatchNorm • Faster training (if GPU allows) • May need to reduce learning rate Smaller batch (32-64): • Noisier gradients (can help exploration) • Less memory • May help escape local minima WGAN-GP specific: • Larger batches helpful for gradient penalty • More stable statistics Memory constraint: • Reduce batch size if OOM • Gradient accumulation as alternative Rule: Start with 64, increase if stable
Category: IMPLEMENTATION

Question: How do you save and load GAN checkpoints properly?
Answer: What to save:
Category: IMPLEMENTATION
Code: # Save checkpoint
checkpoint = {
'epoch': epoch,
'generator_state': generator.state_dict(),
'discriminator_state': discriminator.state_dict(),
'optimizer_G_state': optimizer_G.state_dict(),
'optimizer_D_state': optimizer_D.state_dict(),
'losses': {'G': loss_G_history, 'D': loss_D_history},
'fid_score': current_fid,
}
torch.save(checkpoint, f'gan_epoch_{epoch}.pth')
# Load checkpoint
checkpoint = torch.load('gan_epoch_50.pth')
generator.load_state_dict(checkpoint['generator_state'])
discriminator.load_state_dict(checkpoint['discriminator_state'])
optimizer_G.load_state_dict(checkpoint['optimizer_G_state'])
optimizer_D.load_state_dict(checkpoint['optimizer_D_state'])
start_epoch = checkpoint['epoch'] + 1

Question: What causes NaN losses in GAN training and how do you fix it?
Answer: Common causes: 1. Learning rate too high: • Gradients explode • Solution: Reduce lr to 1e-4 or 5e-5 2. Gradient penalty too large: • WGAN-GP: GP explodes • Solution: Check interpolation code, verify λ=10 3. Numerical instability: • Log of zero in BCE • Solution: Add epsilon: log(x + 1e-8) 4. Bad initialization: • Weights too large • Solution: Use proper initialization (Xavier/He) Prevention: • Gradient clipping (max_norm=1.0) • Monitor gradient norms • Use mixed precision carefully • Check for inf values early
Category: IMPLEMENTATION

Question: How do you initialize GAN networks?
Answer: Recommended initialization:
Category: IMPLEMENTATION
Code: def init_weights(m):
classname = m.__class__.__name__
if classname.find('Conv') != -1:
# Convolutional layers
nn.init.normal_(m.weight.data, 0.0, 0.02)
elif classname.find('BatchNorm') != -1:
# Batch normalization
nn.init.normal_(m.weight.data, 1.0, 0.02)
nn.init.constant_(m.bias.data, 0)
elif classname.find('Linear') != -1:
# Fully connected layers
nn.init.xavier_uniform_(m.weight)
if m.bias is not None:
nn.init.constant_(m.bias, 0)
# Apply initialization
generator.apply(init_weights)
discriminator.apply(init_weights)

Question: What is the recommended way to generate and save samples during training?
Answer: Best practices:
Category: IMPLEMENTATION
Code: # Fix random seed for consistent visualization
fixed_noise = torch.randn(64, latent_dim, device=device)
# Generate samples every N epochs
if epoch % save_interval == 0:
generator.eval()
with torch.no_grad():
fake_images = generator(fixed_noise)
fake_images = (fake_images + 1) / 2 # Denormalize
# Save grid
grid = torchvision.utils.make_grid(
fake_images, nrow=8, normalize=False
)
torchvision.utils.save_image(
grid, f'samples/epoch_{epoch}.png'
)
generator.train()
# Why fixed noise:
# - Track same samples over time
# - See consistent improvement
# - Easy comparison across epochs

Question: How do you handle different image sizes in GAN architecture?
Answer: Scaling architecture: For 32×32 images: • Generator: 100 → 256 → 512 → 1024 → 32×32×3 • 3-4 upsampling layers For 64×64 images: • Add one more upsampling layer • Or increase stride in existing layers For 128×128 images: • 5-6 upsampling layers • May need deeper discriminator General pattern: • Each transpose conv doubles spatial dims • For size S: need log₂(S) - log₂(initial_size) layers Discriminator: • Mirror generator structure • Downsample to same spatial dimensions Memory: Higher resolution = exponentially more memory
Category: IMPLEMENTATION

Question: What are the signs of a healthy WGAN-GP training run?
Answer: Good indicators: 1. Wasserstein distance: • Decreases smoothly over epochs • No wild oscillations • Eventually plateaus 2. Gradient penalty: • Starts 2-10, decreases to 0.5-2.0 • Stable, not increasing • Indicates constraint satisfaction 3. Critic loss: • Decreases (becomes more negative) • Smooth curve • W-dist + λ·GP combined 4. Generator loss: • Decreases (more negative) • Follows W-dist trend • Some noise acceptable 5. Visual samples: • Gradual quality improvement • Increasing sharpness • Growing diversity Red flags: GP exploding, W-dist increasing, sudden quality drops
Category: IMPLEMENTATION