Title: CMPUT 328 A2: FFNN CIFAR-10 Classification

Question: What is a Feedforward Neural Network (FFNN)?
Answer: A neural network where information flows in ONE direction only: from input to output.
Key Properties:
- No cycles or loops
- Fully connected layers
- Directed acyclic graph structure
- Universal function approximators
Also called "Fully Connected Neural Network" or "Multi-Layer Perceptron (MLP)".
Category: FFNN Fundamentals

Question: Why do neural networks need non-linear activation functions?
Answer: Without non-linearity, multiple layers collapse into a single linear transformation.
Proof:
Layer 1: h₁ = W₁x + b₁
Layer 2: y = W₂h₁ + b₂
= W₂(W₁x + b₁) + b₂
= (W₂W₁)x + (W₂b₁ + b₂)
= W'x + b' ← Still linear!
Non-linear activations allow networks to learn complex, non-linear patterns.
Category: FFNN Fundamentals

Question: What is the mathematical formula for a simple FFNN with one hidden layer?
Answer: h = f(W₁x + b₁)
y = g(W₂h + b₂)
Where:
- x: input vector
- W₁, b₁: weights and biases for hidden layer
- f: non-linear activation function (e.g., ReLU)
- h: hidden layer output
- W₂, b₂: weights and biases for output layer
- g: output activation (e.g., softmax)
- y: final prediction
Category: FFNN Fundamentals

Question: What are the key statistics of the CIFAR-10 dataset?
Answer: Dataset Statistics:
- Total images: 60,000
- Training images: 50,000
- Test images: 10,000
- Image size: 32×32×3 (RGB)
- Number of classes: 10
- Images per class: 6,000
Classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck
Category: CIFAR-10 Dataset

Question: What are the normalization values for CIFAR-10 and why normalize?
Answer: Mean (RGB): (0.4914, 0.4822, 0.4465)
Std (RGB): (0.2470, 0.2435, 0.2616)
Normalization formula:
normalized = (pixel - mean) / std
Benefits:
- Faster convergence
- More stable gradients
- Better generalization
- Standardizes inputs to mean ≈ 0, std ≈ 1
Category: CIFAR-10 Dataset

Question: How should CIFAR-10 be split for training, validation, and testing?
Answer: Recommended Split:
- Training: 45,000 images (90% of 50,000)
- Validation: 5,000 images (10% of 50,000)
- Test: 10,000 images (held out)
CRITICAL: Use validation set for hyperparameter tuning, NOT test set! Test set should only be used for final performance reporting.
Category: CIFAR-10 Dataset

Question: Why must images be flattened for FFNNs and what is the flattened size for CIFAR-10?
Answer: FFNNs require 1D input vectors, but images are 3D (height × width × channels).
Original: (32, 32, 3)
Flattened: (3072,)
Calculation: 32 × 32 × 3 = 3,072
Problem: Flattening destroys spatial structure!
Solution: Use CNNs (Assignment 3) which preserve spatial relationships.
Category: Architecture

Question: How many parameters does a FFNN with architecture (3072→512→256→10) have?
Answer: Layer-by-layer calculation:
- Layer 1: (3072 × 512) + 512 = 1,573,376
- Layer 2: (512 × 256) + 256 = 131,328
- Layer 3: (256 × 10) + 10 = 2,570
Total: 1,707,274 parameters
Formula:
For layer with n_in inputs, n_out outputs:
parameters = n_out × (n_in + 1) The "+1" accounts for bias terms.
Category: Architecture

Question: What is ReLU and what are its key properties?
Answer: ReLU(x) = max(0, x)
Properties:
- Simple & fast to compute
- Non-saturating (no vanishing gradient for x > 0)
- Sparse activations (~50% neurons are 0)
- Can have "dead neurons" (always output 0)
Derivative:
ReLU'(x) = 1 if x > 0, else 0 Most common activation for hidden layers!
Category: Activation Functions

Question: What is Softmax and when is it used?
Answer: Softmax(zᵢ) = e^zᵢ / Σⱼ e^zⱼ
Used for: Multi-class classification output layer
Properties:
- Outputs sum to 1 (valid probabilities)
- Differentiable with smooth gradients
- Emphasizes largest values
IMPORTANT: Don't apply softmax manually before nn.CrossEntropyLoss in PyTorch - it's applied internally!
Category: Activation Functions

Question: What is the Cross-Entropy Loss formula and why use it for classification?
Answer: L = -Σᵢ yᵢ log(ŷᵢ)
For single correct class c:
L = -log(ŷ_c)
Advantages:
- Penalizes confident wrong predictions heavily
- Works well with softmax (smooth gradients)
- Probabilistic interpretation (maximizes likelihood)
- Better than MSE for classification
Where y = true label (one-hot), ŷ = predicted probability
Category: Loss Functions

Question: What is the beautiful gradient property of Softmax + Cross-Entropy?
Answer: The gradient with respect to logits (pre-softmax scores) simplifies to:
∂L/∂z = ŷ - y
Where:
- z: logits (raw scores before softmax)
- ŷ: predicted probabilities (after softmax)
- y: true labels (one-hot encoded)
This elegant property is why PyTorch combines them in nn.CrossEntropyLoss!
Category: Loss Functions

Question: What is backpropagation and what does it compute?
Answer: Backpropagation is an algorithm to compute gradients of the loss with respect to all parameters using the chain rule. Goal: Compute ∂L/∂W and ∂L/∂b for all weights and biases
Process:
1. Forward pass: compute outputs left-to-right
2. Backward pass: compute gradients right-to-left
3. Update: adjust parameters using gradients
Uses the chain rule to efficiently propagate errors backward through the network.
Category: Backpropagation

Question: What are the backprop formulas for a linear layer Z = XW + b?
Answer: Given: ∂L/∂Z (gradient from next layer)
∂L/∂X = (∂L/∂Z) Wᵀ
∂L/∂W = Xᵀ (∂L/∂Z)
∂L/∂b = Σᵢ (∂L/∂Z)ᵢ
Key points:
- Wᵀ = transpose of weights
- Matrix multiplication order matters!
- Bias gradient sums over batch dimension
PyTorch does this automatically with loss.backward()
Category: Backpropagation

Question: What are the backprop formulas for a layer with activation: Z = f(XW + b)?
Answer: Given: ∂L/∂Z (gradient from next layer)
∂L/∂X = [f'(XW + b) ⊙ ∂L/∂Z] Wᵀ
∂L/∂W = Xᵀ [f'(XW + b) ⊙ ∂L/∂Z]
∂L/∂b = Σᵢ [f'(XW + b) ⊙ ∂L/∂Z]ᵢ
Where:
- ⊙ = element-wise multiplication
- f' = derivative of activation function
For ReLU: f'(x) = 1 if x > 0, else 0
Category: Backpropagation

Question: What is Gradient Descent and what is the update rule?
Answer: Goal: Minimize loss L(θ) by moving opposite to the gradient
θ_new = θ_old - α ∇L(θ_old)
Where:
- θ = parameters (weights and biases)
- α = learning rate
- ∇L(θ) = gradient (direction of steepest increase)
The gradient points "uphill" so we move in the opposite direction to descend.
Category: Optimization

Question: What are the differences between Batch GD, Stochastic GD, and Mini-batch GD?
Answer: Batch GD: Use all data
- Accurate but slow
- ∇L = (1/N) Σ ∇Lᵢ
Stochastic GD: Use 1 example
- Fast but noisy
- ∇L ≈ ∇Lᵢ
Mini-batch GD: Use small batch
- Best of both worlds!
- ∇L ≈ (1/B) Σ ∇Lᵢ
- Typical: B = 32, 64, 128, 256
Mini-batch is most common in practice.
Category: Optimization

Question: What is Adam optimizer and why is it popular?
Answer: Adam = Adaptive Moment Estimation
Key ideas:
- Uses exponential moving average of gradients (momentum)
- Adapts learning rate for each parameter
- Works well with default hyperparameters
Default settings:
- lr = 0.001
- betas = (0.9, 0.999)
Advantages: Fast convergence, automatic adaptation, more memory for momentum terms
Category: Optimization

Question: How do you recognize if learning rate is too high or too low?
Answer: Too HIGH:
- Loss oscillates wildly
- Loss increases or explodes
- NaN values appear
Too LOW:
- Very slow decrease in loss
- Takes many epochs to converge
- Training seems stuck
Just right:
- Steady, smooth decrease in loss
- Reasonable convergence speed
Try: [0.1, 0.01, 0.001, 0.0001] and pick largest that doesn't diverge.
Category: Optimization

Question: What is the correct order of operations in a PyTorch training loop?
Answer: for images, labels in train_loader:
    # 1. Forward pass
    outputs = model(images)
    loss = criterion(outputs, labels)
    # 2. Zero gradients (CRITICAL!)
    optimizer.zero_grad()
    # 3. Backward pass
    loss.backward()
    # 4. Update weights
    optimizer.step()
CRITICAL: Must call optimizer.zero_grad() because PyTorch accumulates gradients!
Category: Training Process

Question: What is the difference between model.train() and model.eval()?
Answer: model.train():
- Enables dropout
- Batch normalization uses training mode
- Used during training phase
model.eval():
- Disables dropout (all neurons active)
- Batch normalization uses inference mode
- Used during validation/testing
Also use torch.no_grad() during validation to save memory by not tracking gradients.
Category: Training Process

Question: What is early stopping and how is it implemented?
Answer: Early stopping prevents overfitting by stopping training when validation performance stops improving.
Algorithm:
1. Track best validation accuracy
2. If current > best: save model, reset counter
3. Else: increment counter
4. If counter >= patience: stop training
5. Restore best saved model
Typical patience: 5-10 epochs Prevents training too long and overfitting to training data.
Category: Training Process

Question: What is a confusion matrix and what does it reveal?
Answer: A confusion matrix compares predicted vs actual labels for all classes.
Structure: Rows = True labels, Columns = Predicted labels Diagonal: Correct predictions Off-diagonal: Misclassifications Reveals:
- Overall accuracy (diagonal sum / total)
- Per-class accuracy (diagonal / row sum)
- Specific confusion patterns (e.g., cats → dogs)
- Which classes are hardest to classify
Essential for understanding model behavior beyond overall accuracy!
Category: Evaluation

Question: What key metrics should be tracked during training?
Answer: Essential metrics:
- Training loss (per epoch)
- Training accuracy (per epoch)
- Validation loss (per epoch)
- Validation accuracy (per epoch)
Plot these curves to detect:
- Overfitting (train/val diverge)
- Underfitting (both plateau low)
- When to stop training
- Learning rate issues
Always monitor both training AND validation!
Category: Evaluation

Question: What are the key hyperparameters for FFNN and typical starting values?
Answer: Architecture:
- Hidden dims: (512, 256)
- Dropout: 0.1-0.3
- Activation: ReLU
Training:
- Learning rate: 0.001 (Adam)
- Batch size: 256
- Epochs: 20-50
- Optimizer: Adam
Regularization:
- Weight decay: 1e-4
- Early stopping patience: 5-7
Category: Hyperparameters

Question: How should hyperparameters be tuned systematically?
Answer: Systematic Approach:
1. Start with baseline configuration
2. Change ONE variable at a time
3. Train and evaluate each variant
4. Compare validation performance
5. Keep improvements, revert failures
6. Iterate
CRITICAL: Use validation set for tuning, NOT test set! Test set should only be used ONCE for final reporting.
Category: Hyperparameters

Question: What are guidelines for choosing hidden layer sizes?
Answer: Rule of thumb:
- Input: 3072 (32×32×3 flattened)
- Hidden 1: 512-2048 (smaller than input)
- Hidden 2: 256-512 (smaller than H1)
- Output: 10 (number of classes)
More neurons: ✓ More capacity, ✗ Slower, ✗ Overfitting Fewer neurons: ✓ Faster, ✓ Less overfitting, ✗ May underfit
Balance between capacity and regularization!
Category: Hyperparameters

Question: What are 5 critical mistakes to avoid when training FFNNs?
Answer: 1. Not shuffling training data → Add shuffle=True to DataLoader
2. Forgetting to normalize images → Use transforms.Normalize with CIFAR-10 values
3. Using test set for hyperparameter tuning → Use validation set instead!
4. Not calling model.eval() during validation → Dropout stays active, inconsistent results
5. Forgetting optimizer.zero_grad() → Gradients accumulate, wrong updates!
Category: Common Pitfalls

Question: How do you recognize and fix overfitting?
Answer: Symptoms:
- High training accuracy, low validation accuracy
- Large gap between train/val loss curves
- Validation accuracy plateaus or decreases
Solutions:
- Increase dropout (0.2 → 0.3 → 0.5)
- Add weight decay (1e-4, 1e-3)
- Use early stopping
- Get more training data (if possible)
- Reduce model size
Category: Common Pitfalls

Question: How do you recognize and fix underfitting?
Answer: Symptoms:
- Both training and validation accuracy low
- Loss decreases very slowly
- Model seems "stuck"
Solutions:
- Increase model capacity (larger/more layers)
- Train for more epochs
- Lower learning rate (sometimes helps)
- Remove/reduce regularization
- Check data preprocessing is correct
Category: Common Pitfalls

Question: Why should you NOT apply softmax before nn.CrossEntropyLoss in PyTorch?
Answer: nn.CrossEntropyLoss already applies softmax internally! WRONG:
outputs = F.softmax(model(images), dim=1)
loss = nn.CrossEntropyLoss()(outputs, labels)
CORRECT:
logits = model(images)  # Raw scores
loss = nn.CrossEntropyLoss()(logits, labels)
Applying softmax twice leads to incorrect gradients and poor training!
Category: Common Pitfalls

Question: What are the requirements for Assignment 2 Part A (In-Lab)?
Answer: Part A - In-Lab (2%):
1. Load CIFAR-10 with train/val/test splits (0.25%)
2. Implement FFNN, train to ≥50% accuracy (1.0%)
3. Plot train vs val loss and accuracy (0.25%)
4. Compute confusion matrix on validation (0.25%)
5. Display ≥16 misclassified examples (0.25%)
Must be completed during 3-hour lab session.
Category: Assignment Requirements

Question: What are the requirements for Assignment 2 Part B (Take-Home)?
Answer: Part B - Take-Home (3%):
1. Modify ≥1 hyperparameter + justify (1.0%)
2. Re-train and update all plots/metrics (1.0%)
3. Write 1-2 page report discussing:
  - Training dynamics
  - Confusion patterns
  - Impact of your changes
(1.0%)
Due 1 week after lab. Submit notebook + PDF report.
Category: Assignment Requirements

Question: What are realistic performance expectations for FFNN on CIFAR-10?
Answer: Expected Performance:
- Baseline (512, 256): 50-55% validation accuracy
- Improved (1024, 512, 256): 55-58% validation accuracy
- Training time: ~12-18s/epoch (GPU)
- Parameters: 1.7M - 3.8M
Note: FFNN performance is LIMITED because spatial structure is lost when flattening images. CNNs (Assignment 3) achieve 80-90% on CIFAR-10!
Category: Assignment Requirements

Question: What is the correct way to flatten images in PyTorch FFNN forward pass?
Answer: def forward(self, x):
    # Input x shape: (batch, 3, 32, 32)
    x = x.view(x.size(0), -1)
    # Output shape: (batch, 3072)
    x = self.fc1(x)
    x = F.relu(x)
    # ... rest of network
    return x
Key points:
- x.size(0) = batch size
- -1 = infer remaining dimensions
- Result: (batch, 3072)
Category: PyTorch Implementation

Question: How do you properly compute accuracy in PyTorch?
Answer: def compute_accuracy(model, dataloader, device):
    model.eval()
    correct = 0
    total = 0
    with torch.no_grad():
        for images, labels in dataloader:
            images = images.to(device)
            labels = labels.to(device)
            outputs = model(images)
            _, predicted = torch.max(outputs, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
    return correct / total
Category: PyTorch Implementation