Title: Assignment 3: CNNs for CIFAR-10 Image Classification

Question: What is a Convolutional Neural Network (CNN)?
Answer: A CNN is a specialized neural network designed for processing grid-like data, particularly images.
Key characteristics:
- Spatial hierarchy: Learns hierarchical patterns from low-level (edges) to high-level (objects)
- Parameter sharing: Same filters applied across entire image
- Translation invariance: Detects features regardless of position
- Sparse connectivity: Each neuron connects only to local region
Inspired by: Animal visual cortex
Category: CNN Fundamentals

Question: Why do CNNs NOT flatten the image at the input?
Answer: Flattening destroys spatial structure!
Problems with flattening:
- Loss of spatial relationships: Adjacent pixels treated same as distant pixels
- No translation invariance: Must learn same feature at every position
- Huge parameter count: 32×32×3 → 3,072 inputs × 1,024 hidden = 3.1M parameters!
- Poor generalization: Too many parameters cause overfitting
CNNs preserve structure: [batch, channels, height, width] throughout network
Category: CNN Fundamentals
Code: # WRONG - Flattening at input
x = x.view(x.size(0), -1) # Destroys spatial structure!
# CORRECT - Preserve spatial structure
x = self.conv1(x) # [batch, 3, 32, 32] → [batch, 64, 32, 32]

Question: What is convolution and how does it work?
Answer: Convolution slides a small filter (kernel) over the input to produce a feature map.
Mathematical operation: Output[i,j] = Σ Σ Input[i+m, j+n] × Kernel[m,n]
Process:
- Small filter (e.g., 3×3) slides across input
- Element-wise multiplication at each position
- Sum results to produce one output value
- Slide to next position (determined by stride)
Key benefit: Same filter detects patterns anywhere in image
Category: Convolutional Layers
Code: nn.Conv2d(
in_channels=3,
out_channels=64,
kernel_size=3,
stride=1,
padding=1
)

Question: What are the key parameters of a convolutional layer?
Answer: Four main parameters control convolution behavior:
1. Kernel size (k): Size of sliding window (e.g., 3×3, 5×5)
2. Stride (s): How many pixels to slide
- stride=1: overlapping windows
- stride=2: downsample by factor of 2
3. Padding (p): Add zeros around border
- padding=1 with 3×3 kernel maintains spatial size
4. Number of filters (out_channels): How many feature maps to produce
- Common: 32, 64, 128, 256
Category: Convolutional Layers
Code: # Example: maintain spatial size
nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1)
# 32×32 input → 32×32 output
# Example: downsample
nn.Conv2d(64, 128, kernel_size=3, stride=2, padding=1)
# 32×32 input → 16×16 output

Question: What is the formula for calculating output size after convolution?
Answer: Output size formula:
output_size = ⌊(input_size + 2×padding - kernel_size) / stride⌋ + 1
Example 1: Maintain size
- Input: 32×32
- Kernel: 3×3, Padding: 1, Stride: 1
- Output: ⌊(32 + 2×1 - 3) / 1⌋ + 1 = 32×32
Example 2: Downsample
- Input: 32×32
- Kernel: 3×3, Padding: 1, Stride: 2
- Output: ⌊(32 + 2×1 - 3) / 2⌋ + 1 = 16×16
Category: Convolutional Layers

Question: How do you calculate the number of parameters in a Conv2d layer?
Answer: Formula:
Parameters = (kernel_h × kernel_w × in_channels + 1) × out_channels
Components:
- Weights: kernel_h × kernel_w × in_channels per filter
- Bias: +1 per filter
- Total filters: × out_channels
Example: Conv2d(3, 64, kernel_size=3)
= (3 × 3 × 3 + 1) × 64 = 1,792 parameters
Compare to FC: 3,072 × 1,024 = 3,145,728 parameters!
CNN is 1,754× more efficient!
Category: Convolutional Layers
Code: def count_parameters(layer):
return sum(p.numel() for p in layer.parameters())
conv = nn.Conv2d(3, 64, 3)
print(count_parameters(conv)) # 1,792
fc = nn.Linear(3072, 1024)
print(count_parameters(fc)) # 3,146,752

Question: What is the purpose of pooling layers?
Answer: Pooling reduces spatial dimensions while preserving important information.
Four main purposes:
1. Reduce spatial dimensions: Decrease computational cost and memory
2. Increase receptive field: Each neuron "sees" more of the input
3. Add translation invariance: Small shifts don't change output
4. Reduce overfitting: Fewer parameters in subsequent layers
Typical progression:
- 32×32 → MaxPool(2) → 16×16
- 16×16 → MaxPool(2) → 8×8
- 8×8 → MaxPool(2) → 4×4
Category: Pooling
Code: nn.MaxPool2d(kernel_size=2, stride=2)

Question: What is the difference between MaxPool and AvgPool?
Answer: MaxPool vs AvgPool:
MaxPool2d:
- Operation: Takes maximum value in each window
- Preserves: Strongest activations (sharp features)
- Use case: Feature detection (edges, patterns)
- Most common in modern CNNs
AvgPool2d:
- Operation: Takes average value in each window
- Preserves: Overall intensity
- Use case: Smooth downsampling
- Less common nowadays
Global Average Pooling (GAP): Reduces each feature map to single value, replaces flatten + FC
Category: Pooling
Code: # MaxPool
nn.MaxPool2d(2) # Most common
# AvgPool
nn.AvgPool2d(2) # Less common
# Global Average Pooling
nn.AdaptiveAvgPool2d(1) # Before classifier

Question: Why do we need activation functions in neural networks?
Answer: Activation functions introduce non-linearity.
Without activations:
Linear → Linear → Linear ≡ Single Linear Layer
Problem: Stacking layers is useless without non-linearity!
With activations:
- Network can learn complex, non-linear patterns
- Each layer adds expressiveness
- Can approximate any function (universal approximation)
Most common in CNNs: ReLU (Rectified Linear Unit)
Category: Activation Functions
Code: # Bad - no activations (useless!)
x = self.fc1(x)
x = self.fc2(x)
x = self.fc3(x)
# Good - with activations
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = self.fc3(x)

Question: What is ReLU and why is it used in CNNs?
Answer: ReLU (Rectified Linear Unit): ReLU(x) = max(0, x)
Advantages:
- Fast to compute: Simple max operation
- Helps with vanishing gradients: Gradient is 0 or 1
- Sparse activations: Many zeros, efficient representation
- Works well in practice: Default choice for CNNs
Disadvantages:
- "Dying ReLU": Neurons can get stuck at 0 forever
- Not zero-centered: All outputs ≥ 0
Variants: Leaky ReLU, PReLU, ELU (prevent dying neurons)
Category: Activation Functions
Code: nn.ReLU(inplace=True)
# Typical usage in CNN
nn.Conv2d(64, 128, 3, padding=1),
nn.BatchNorm2d(128),
nn.ReLU(inplace=True), # After BatchNorm

Question: What is Batch Normalization and why is it important?
Answer: BatchNorm normalizes layer inputs to have mean=0, std=1.
Formula:
BatchNorm(x) = γ × (x - μ_batch) / √(σ²_batch + ε) + β
Where:
- μ_batch, σ²_batch: mean and variance of current batch
- γ, β: learned scale and shift parameters
Benefits:
- Faster convergence: Stabilized training dynamics
- Higher learning rates: Can train with larger LR
- Less sensitive to initialization
- Acts as regularization (reduces overfitting)
Category: Normalization
Code: # Typical ordering: Conv → BatchNorm → ReLU
nn.Conv2d(64, 128, 3, padding=1),
nn.BatchNorm2d(128),
nn.ReLU(inplace=True)

Question: How does BatchNorm behave differently during training vs inference?
Answer: Training mode vs Evaluation mode:
During Training (model.train()):
- Computes mean and variance from current batch
- Updates running statistics (exponential moving average)
- Normalizes using batch statistics
During Inference (model.eval()):
- Uses running statistics computed during training
- No batch statistics (might have batch_size=1!)
- Deterministic output
CRITICAL: Always call model.eval() before evaluation/testing!
Category: Normalization
Code: # Training
model.train()
for inputs, targets in train_loader:
outputs = model(inputs) # Uses batch stats
# Evaluation
model.eval() # IMPORTANT!
with torch.no_grad():
for inputs, targets in test_loader:
outputs = model(inputs) # Uses running stats

Question: What is Dropout and how does it prevent overfitting?
Answer: Dropout randomly drops (sets to 0) activations during training.
Mechanism:
- Training: Randomly set activations to 0 with probability p
- Inference: Use all activations, scaled by (1-p)
Why it works:
- Forces redundancy: Can't rely on single neuron
- Ensemble effect: Training many sub-networks
- Prevents co-adaptation: Neurons learn independent features
Typical values:
- 0.5 in fully connected layers (aggressive)
- 0.2-0.3 after convolutional layers (mild)
Category: Regularization
Code: # Higher dropout in FC layers
nn.Dropout(0.5),
nn.Linear(256, 128),
# Lower dropout after conv layers
nn.Dropout2d(0.25), # Drops entire feature maps
nn.Conv2d(64, 128, 3)

Question: What is weight decay (L2 regularization)?
Answer: Weight decay penalizes large weights to encourage simpler models.
Formula:
Loss_total = Loss_original + λ × Σ(w²)
Effect:
- Pushes weights toward zero
- Prevents any single weight from becoming too large
- Encourages distributed representations
- Reduces overfitting
Typical values:
- λ = 5e-4 (0.0005) - common default
- Range: 1e-4 to 1e-3
Category: Regularization
Code: optimizer = optim.AdamW(
model.parameters(),
lr=1e-3,
weight_decay=5e-4 # L2 regularization
)

Question: What is data augmentation and why is it important?
Answer: Data augmentation creates variations of training data to improve generalization.
Benefits:
- Increases effective dataset size: More training examples
- Improves generalization: Model learns invariances
- Reduces overfitting: More diverse examples
- Better calibration: More robust predictions
For CIFAR-10, typically gain:
- Basic augmentation (crop + flip): +2-5% accuracy
- With color jitter: +1-2% additional
- With random erasing: +0.5-1% additional
Total improvement: +5-10% accuracy possible!
Category: Data Augmentation

Question: What are the common data augmentations for CIFAR-10?
Answer: Four recommended augmentations for CIFAR-10:
1. RandomCrop(32, padding=4)
- Pads to 36×36, then random crops to 32×32
- Effect: Translation invariance
2. RandomHorizontalFlip()
- 50% chance to flip left-right
- Effect: Mirror symmetry (car facing left = car facing right)
3. ColorJitter(0.2, 0.2, 0.2, 0.1)
- Varies brightness, contrast, saturation, hue
- Effect: Robustness to lighting conditions
4. RandomErasing(p=0.15)
- Randomly masks rectangular regions
- Effect: Occlusion robustness, prevents single-feature reliance
Category: Data Augmentation
Code: train_transform = transforms.Compose([
transforms.RandomCrop(32, padding=4),
transforms.RandomHorizontalFlip(),
transforms.ColorJitter(0.2, 0.2, 0.2, 0.1),
transforms.ToTensor(),
transforms.Normalize(mean, std),
transforms.RandomErasing(p=0.15),
])

Question: Should you augment the test set? Why or why not?
Answer: NO! NEVER augment the test set.
Reasons:
- Fair evaluation: Test should reflect real-world clean images
- Consistency: Everyone evaluates on same data
- Reproducibility: Random augmentations give different results each run
- Meaningful metrics: Want true performance, not boosted numbers
Test transform should only include:
- ToTensor() - convert to tensor
- Normalize() - same normalization as training
Training = augment heavily
Testing = clean data only
Category: Data Augmentation
Code: # WRONG - augmenting test set
test_transform = transforms.Compose([
transforms.RandomCrop(32, padding=4), # NO!
transforms.ToTensor(),
])
# CORRECT - clean test data
test_transform = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize(mean, std),
])

Question: What loss function should you use for multi-class classification?
Answer: CrossEntropyLoss is the standard for multi-class classification.
What it does:
- Combines LogSoftmax + Negative Log Likelihood
- Expects raw logits (before softmax)
- Outputs probability distribution via softmax
- Penalizes wrong predictions more heavily
Formula:
Loss = -log(softmax(logits)[target_class])
With label smoothing (recommended):
- Prevents overconfident predictions
- Better calibration
- Acts as regularization
Category: Training
Code: # Basic
criterion = nn.CrossEntropyLoss()
# With label smoothing (better)
criterion = nn.CrossEntropyLoss(label_smoothing=0.05)
# Usage
loss = criterion(outputs, targets)
# outputs: raw logits [batch, 10]
# targets: class indices [batch]

Question: Adam vs SGD: which optimizer should you use for CNNs?
Answer: Both have trade-offs. Choose based on your goals:
Adam / AdamW:
- Pros: Fast convergence, adaptive LR per parameter, works out-of-the-box
- Cons: Can generalize slightly worse than SGD
- Use when: Quick experiments, prototyping, limited tuning time
- LR: 1e-4 to 1e-3
SGD + Momentum:
- Pros: Better generalization, proven track record
- Cons: Requires careful LR tuning, slower convergence
- Use when: Final production model, have time to tune
- LR: 0.01 to 0.1
For Assignment 3: AdamW is recommended
Category: Training
Code: # AdamW (recommended for A3)
optimizer = optim.AdamW(
model.parameters(),
lr=2e-3,
weight_decay=5e-4
)
# SGD + Momentum (for production)
optimizer = optim.SGD(
model.parameters(),
lr=0.1,
momentum=0.9,
weight_decay=5e-4,
nesterov=True
)

Question: What are learning rate schedules and why use them?
Answer: LR schedules decrease learning rate during training for better convergence.
Why use schedules:
- Early training: High LR for fast exploration
- Late training: Low LR for fine-tuning
- Better final accuracy: Typically +1-2%
Three common schedules:
1. CosineAnnealingLR: Smooth cosine decrease
2. StepLR: Decrease by factor every N epochs
3. ReduceLROnPlateau: Decrease when validation stops improving
For Assignment 3: CosineAnnealing recommended
Category: Training
Code: # Cosine (recommended)
scheduler = optim.lr_scheduler.CosineAnnealingLR(
optimizer, T_max=50, eta_min=1e-6
)
# Step
scheduler = optim.lr_scheduler.StepLR(
optimizer, step_size=30, gamma=0.1
)
# Adaptive
scheduler = optim.lr_scheduler.ReduceLROnPlateau(
optimizer, mode='min', factor=0.5, patience=5
)
# Update after each epoch
scheduler.step()

Question: What is gradient clipping and when should you use it?
Answer: Gradient clipping limits the magnitude of gradients to prevent instability.
How it works:
- Compute gradient norm: ||g|| = √(Σ g_i²)
- If ||g|| > threshold, scale down: g ← g × (threshold / ||g||)
When to use:
- Training with high learning rates
- When loss occasionally spikes to NaN
- Recurrent networks (RNNs/LSTMs)
- Very deep networks
Typical threshold: 0.5 to 2.0
For Assignment 3: Use max_norm=1.0
Category: Training
Code: loss.backward()
# Clip gradients
torch.nn.utils.clip_grad_norm_(
model.parameters(),
max_norm=1.0
)
optimizer.step()

Question: What are the key statistics of the CIFAR-10 dataset?
Answer: CIFAR-10 dataset statistics:
Size:
- 60,000 total images
- 50,000 training images
- 10,000 test images
Format:
- 32×32 pixels
- 3 channels (RGB)
- 10 balanced classes (6,000 each)
Classes:
airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck
Normalization values:
mean = (0.4914, 0.4822, 0.4465)
std = (0.2470, 0.2435, 0.2616)
Category: CIFAR-10
Code: from torchvision import datasets
train_dataset = datasets.CIFAR10(
root='./data',
train=True,
download=True,
transform=train_transform
)
test_dataset = datasets.CIFAR10(
root='./data',
train=False,
download=True,
transform=test_transform
)

Question: How should you split CIFAR-10 for training?
Answer: Standard split for Assignment 3:
Recommended:
- 45,000 training images (90% of train set)
- 5,000 validation images (10% of train set)
- 10,000 test images (official test set)
Why use validation set:
- Monitor overfitting during training
- Early stopping based on val accuracy
- Save best model based on val performance
- Tune hyperparameters
NEVER tune on test set! Use only for final evaluation.
Category: CIFAR-10
Code: # Create validation split
val_size = 5000
indices = torch.randperm(len(train_dataset))
train_indices = indices[val_size:].tolist() # 45,000
val_indices = indices[:val_size].tolist() # 5,000
train_subset = Subset(train_dataset, train_indices)
val_subset = Subset(train_dataset, val_indices)

Question: Why do we normalize CIFAR-10 images?
Answer: Normalization centers pixel values and standardizes scale.
Formula:
normalized = (x - mean) / std
Benefits:
- Zero-centered inputs: Speeds up convergence
- Prevents saturation: Activations stay in good range
- Consistent scale: All channels have similar importance
- Better gradients: More stable training
CIFAR-10 statistics:
Computed from all 50,000 training images
CRITICAL: Use same normalization for train AND test!
Category: CIFAR-10
Code: # CIFAR-10 statistics
mean = (0.4914, 0.4822, 0.4465)
std = (0.2470, 0.2435, 0.2616)
normalize = transforms.Normalize(mean=mean, std=std)
# Apply to both train and test
train_transform = transforms.Compose([
transforms.RandomCrop(32, padding=4),
transforms.ToTensor(),
normalize, # Same normalization
])
test_transform = transforms.Compose([
transforms.ToTensor(),
normalize, # Same normalization
])

Question: What metrics should you track during CNN training?
Answer: Track both training and validation metrics:
Essential metrics:
1. Training loss: Are we learning?
2. Training accuracy: Performance on train set
3. Validation loss: Generalization indicator
4. Validation accuracy: True performance metric
Watch for patterns:
- Train loss ↓, Val loss ↓: Good! Keep training
- Train loss ↓, Val loss ↑: Overfitting! Add regularization
- Both high: Underfitting! Increase capacity or train longer
Additional metrics:
- Learning rate (if using scheduler)
- Time per epoch
- Best epoch / best validation accuracy
Category: Evaluation
Code: history = []
for epoch in range(num_epochs):
train_loss, train_acc = train_epoch(...)
val_loss, val_acc = evaluate(...)
history.append({
'epoch': epoch,
'train_loss': train_loss,
'train_acc': train_acc,
'val_loss': val_loss,
'val_acc': val_acc,
'lr': optimizer.param_groups[0]['lr']
})

Question: How do you properly evaluate a model on the test set?
Answer: Proper evaluation procedure:
Steps:
1. Set to eval mode: model.eval()
2. Disable gradients: with torch.no_grad()
3. Use test transform: No augmentation!
4. Track predictions: Compute metrics
Why model.eval() is critical:
- Switches BatchNorm to use running statistics
- Disables Dropout (use all neurons)
- Deterministic behavior
Why torch.no_grad() is important:
- Saves memory (no gradient computation)
- Faster evaluation
- Prevents accidental gradient updates
Category: Evaluation
Code: def evaluate(model, dataloader, criterion, device):
model.eval() # CRITICAL!
running_loss = 0.0
correct = 0
total = 0
with torch.no_grad(): # CRITICAL!
for inputs, targets in dataloader:
inputs = inputs.to(device)
targets = targets.to(device)
outputs = model(inputs)
loss = criterion(outputs, targets)
running_loss += loss.item() * inputs.size(0)
predictions = outputs.argmax(dim=1)
correct += (predictions == targets).sum().item()
total += targets.size(0)
return running_loss / total, correct / total

Question: Why do CNNs outperform fully connected networks on images?
Answer: CNNs have inductive biases that match image structure.
1. Spatial structure preservation:
- FC: Flattens to [batch, 3072] - destroys spatial relationships
- CNN: Keeps [batch, C, H, W] - preserves adjacency
2. Parameter efficiency:
- FC first layer: 3,145,728 parameters
- CNN first layer: 1,792 parameters
- 1,754× fewer parameters!
3. Translation invariance:
- FC: Must learn cat at every position separately
- CNN: Same filter detects cat anywhere
4. Hierarchical learning:
- Layer 1: Edges → Layer 2: Shapes → Layer 3: Objects
Category: CNN vs FC
Code: # FC Network
class FC_Net(nn.Module):
def forward(self, x):
x = x.view(x.size(0), -1) # Flatten!
x = F.relu(self.fc1(x)) # 3M parameters
...
# CNN
class CNN(nn.Module):
def forward(self, x):
# x: [batch, 3, 32, 32] - preserved!
x = self.conv1(x) # Only 1.7k parameters
...

Question: What are typical accuracy results on CIFAR-10?
Answer: Expected test accuracy by model type:
Model | Test Accuracy
Random guessing | 10%
3-layer FC network | 55-60%
Simple CNN (no augmentation) | 70-75%
CNN + basic augmentation | 80-85%
CNN + full augmentation | 85-90%
ResNet-18 | 92-95%
State-of-the-art | 99%+
For Assignment 3: Target 80-85%+ for strong grade
Category: CNN vs FC

Question: What is a typical CNN architecture for CIFAR-10?
Answer: Standard pattern: [Conv-BN-ReLU-Pool] blocks → GAP → FC
Example architecture:
- Block 1: 3→64 Conv, BN, ReLU, MaxPool → 32×32 to 16×16
- Block 2: 64→128 Conv, BN, ReLU, MaxPool → 16×16 to 8×8
- Block 3: 128→256 Conv, BN, ReLU, AdaptiveAvgPool → 8×8 to 1×1
- Classifier: Flatten, Dropout, FC(256→128), ReLU, FC(128→10)
Key principles:
- Increasing channels: 3 → 64 → 128 → 256
- Decreasing spatial size: 32 → 16 → 8 → 1
- BN after Conv, before ReLU
- Dropout for regularization
Category: Architecture
Code: self.features = nn.Sequential(
# Block 1
nn.Conv2d(3, 64, 3, padding=1),
nn.BatchNorm2d(64),
nn.ReLU(inplace=True),
nn.MaxPool2d(2),
# Block 2
nn.Conv2d(64, 128, 3, padding=1),
nn.BatchNorm2d(128),
nn.ReLU(inplace=True),
nn.MaxPool2d(2),
# Block 3
nn.Conv2d(128, 256, 3, padding=1),
nn.BatchNorm2d(256),
nn.ReLU(inplace=True),
nn.AdaptiveAvgPool2d(1),
)

Question: What is the correct order of Conv-BatchNorm-ReLU?
Answer: Correct order: Conv → BatchNorm → ReLU
Why this order:
- Conv: Linear transformation
- BatchNorm: Normalize before activation (stabilizes)
- ReLU: Non-linearity after normalization
Wrong orders and why:
- BN → Conv → ReLU: BN expects output of Conv, not input
- Conv → ReLU → BN: ReLU zeroes negative values, reducing BN effectiveness
- ReLU → Conv → BN: Completely wrong, doesn't make sense
Remember: Process data, normalize, activate!
Category: Architecture
Code: # CORRECT
nn.Conv2d(64, 128, 3, padding=1),
nn.BatchNorm2d(128),
nn.ReLU(inplace=True),
# WRONG
nn.BatchNorm2d(64),
nn.Conv2d(64, 128, 3, padding=1), # NO!
nn.ReLU(inplace=True),

Question: What are the most common mistakes when implementing CNNs?
Answer: Top 10 pitfalls to avoid:
1. Flattening at input - Destroys spatial structure
2. Augmenting test set - Unfair evaluation
3. Not calling model.eval() - BatchNorm/Dropout wrong mode
4. Wrong BatchNorm placement - Should be Conv→BN→ReLU
5. LR too high - Loss explodes to NaN
6. Not saving best model - Last epoch might not be best
7. Forgetting to normalize inputs - Slow/unstable training
8. Not setting seeds - Results not reproducible
9. Using same normalization for train/test - Must match!
10. Training too long without validation - Overfitting
Category: Common Pitfalls

Question: How do you fix training when loss becomes NaN?
Answer: Loss = NaN usually means gradients exploded.
Solutions to try (in order):
1. Lower learning rate:
- Try 10× smaller (e.g., 1e-3 → 1e-4)
2. Add gradient clipping:
- torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
3. Check data normalization:
- Ensure inputs are normalized to [-1, 1] or [0, 1]
4. Verify no bugs:
- Check for division by zero
- Ensure targets are in correct range [0, num_classes)
- Check model initialization
Category: Common Pitfalls
Code: # Solution 1: Lower LR
optimizer = optim.AdamW(
model.parameters(),
lr=1e-4, # Instead of 1e-3
weight_decay=5e-4
)
# Solution 2: Gradient clipping
loss.backward()
torch.nn.utils.clip_grad_norm_(
model.parameters(),
max_norm=1.0
)
optimizer.step()

Question: How do you ensure reproducible results in PyTorch?
Answer: Set ALL random seeds at start of training.
Must set seeds for:
1. Python random: random.seed()
2. NumPy: np.random.seed()
3. PyTorch CPU: torch.manual_seed()
4. PyTorch CUDA: torch.cuda.manual_seed_all()
5. cuDNN: deterministic=True, benchmark=False
For Assignment 3: Use seed=328
Additional tips:
- Document PyTorch version
- Save model architecture code
- Record hyperparameters
- Use same hardware when possible
Category: Reproducibility
Code: def set_seed(seed=328):
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
if torch.cuda.is_available():
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
set_seed(328)