Title: CMPUT 328 Assignment 1: Logistic Regression on MNIST

Question: What is linear regression?
Answer: Linear regression is a supervised learning algorithm that models the relationship between input features and a continuous output using a linear function.
Mathematical form:
ŷ = w₁x₁ + w₂x₂ + ... + wₙxₙ + b
Key properties:
- Linear decision boundary: Separates data with a straight line/hyperplane
- Differentiable: Can optimize using gradient descent
- Interpretable: Weights show feature importance
Category: Linear Regression

Question: What is the loss function for linear regression?
Answer: Mean Squared Error (MSE) is the standard loss function for linear regression.
Formula:
MSE = (1/N) Σ(yᵢ - ŷᵢ)²
Where:
- N: number of samples
- yᵢ: true value
- ŷᵢ: predicted value
Why MSE?
- Differentiable everywhere
- Penalizes large errors more (squared term)
- Convex for linear models (single global minimum)
Category: Linear Regression

Question: What is logistic regression?
Answer: Logistic regression extends linear regression for classification by applying a sigmoid (or softmax) function to convert linear outputs into probabilities.
For binary classification:
P(y=1|x) = σ(wᵀx + b) = 1/(1 + e⁻⁽ʷᵀˣ⁺ᵇ⁾)
For multi-class (MNIST):
P(y=k|x) = softmax(wₖᵀx + bₖ)
Key difference from linear regression:
- Linear regression: outputs continuous values
- Logistic regression: outputs probabilities (0-1)
Category: Logistic Regression

Question: What is the sigmoid function and what are its properties?
Answer: Sigmoid function: σ(z) = 1 / (1 + e⁻ᶻ)
Properties:
- Output range: (0, 1) - perfect for probabilities
- σ(0) = 0.5: Decision boundary
- σ(+∞) = 1: Approaches 1 for large positive values
- σ(-∞) = 0: Approaches 0 for large negative values
- Smooth and differentiable: Enables gradient descent
- S-shaped curve: Gradual transition from 0 to 1
Used for: Binary classification (2 classes)
Category: Logistic Regression

Question: What is the softmax function and how is it used?
Answer: Softmax function: Generalizes sigmoid to multiple classes
Formula:
softmax(zᵢ) = e^zᵢ / Σⱼ e^zⱼ
Properties:
- Output sums to 1: Σᵢ softmax(zᵢ) = 1
- Each output is a probability: 0 < softmax(zᵢ) < 1
- Preserves ordering: Larger inputs → larger probabilities
For MNIST (10 classes):
P(y=k|x) = e^(wₖᵀx + bₖ) / Σⱼ₌₀⁹ e^(wⱼᵀx + bⱼ)
Usage: Multi-class classification (3+ classes)
Category: Logistic Regression

Question: What is cross-entropy loss?
Answer: Cross-entropy loss measures the difference between predicted probabilities and true labels.
Binary cross-entropy:
L = -[y log(ŷ) + (1-y) log(1-ŷ)]
Multi-class cross-entropy (categorical):
L = -Σₖ yₖ log(ŷₖ)
Why use cross-entropy?
- Probabilistic interpretation (maximum likelihood)
- Heavily penalizes confident wrong predictions
- Convex for logistic regression
- Gradient descent converges well
Category: Logistic Regression

Question: How does PyTorch CrossEntropyLoss work?
Answer: PyTorch CrossEntropyLoss combines LogSoftmax + NegativeLo gLikelihood Loss.
Important: Expects raw logits (unnormalized scores), NOT probabilities!
Input format:
- Predictions: Raw logits [batch_size, num_classes]
- Targets: Class indices (NOT one-hot) [batch_size]
Internal operations:
1. Apply LogSoftmax to logits
2. Compute negative log-likelihood
3. Return average loss
Common mistake: Applying softmax before CrossEntropyLoss (double softmax!)
Category: Logistic Regression
Code: # CORRECT
logits = model(x) # Raw scores
loss = nn.CrossEntropyLoss()(logits, y)
# WRONG - double softmax!
probs = F.softmax(model(x), dim=1)
loss = nn.CrossEntropyLoss()(probs, y)

Question: What is the MNIST dataset?
Answer: MNIST (Modified National Institute of Standards and Technology) is a benchmark dataset of handwritten digits.
Dataset statistics:
- Training samples: 60,000 images
- Test samples: 10,000 images
- Image size: 28×28 pixels (grayscale)
- Classes: 10 (digits 0-9)
- Input features: 784 (28×28 flattened)
- Pixel range: 0-255 (grayscale intensity)
Common benchmark: Simple models achieve ~92-98% accuracy
Category: MNIST

Question: How do you normalize MNIST data in PyTorch?
Answer: Normalization scales pixel values to have zero mean and unit variance.
MNIST normalization constants:
- Mean: 0.1307 (computed from training set)
- Std: 0.3081 (computed from training set)
Why normalize?
- Faster convergence
- Numerical stability
- Better gradient flow
- Prevents one feature from dominating
Category: MNIST
Code: transform = transforms.Compose([
transforms.ToTensor(), # [0,255] → [0,1]
transforms.Normalize((0.1307,), (0.3081,)) # Standardize
])
train_ds = datasets.MNIST(
root="./data",
train=True,
download=True,
transform=transform
)

Question: How do you split MNIST into train/val/test sets?
Answer: Standard split: 50k train, 10k validation, 10k test
Why use validation set?
- Tune hyperparameters without touching test set
- Early stopping based on val performance
- Unbiased test evaluation at the end
Rule: NEVER use test set during training or hyperparameter tuning!
Category: MNIST
Code: # Original: 60k train, 10k test
train_full = datasets.MNIST(train=True, ...)
test_ds = datasets.MNIST(train=False, ...)
# Split train into 50k train + 10k val
train_ds, val_ds = random_split(
train_full,
[50_000, 10_000]
)
# Create data loaders
train_loader = DataLoader(train_ds, batch_size=128, shuffle=True)
val_loader = DataLoader(val_ds, batch_size=128, shuffle=False)
test_loader = DataLoader(test_ds, batch_size=128, shuffle=False)

Question: How do you implement logistic regression for MNIST in PyTorch?
Answer: Architecture: Single linear layer (784 → 10)
Key steps:
1. Flatten images: 28×28 → 784
2. Linear transformation: 784 inputs → 10 outputs (one per class)
3. Return raw logits (no softmax - handled by CrossEntropyLoss)
Number of parameters: 784×10 + 10 = 7,850
Category: Implementation
Code: class LogisticRegression(nn.Module):
def __init__(self, in_dim=28*28, out_dim=10):
super().__init__()
self.fc = nn.Linear(in_dim, out_dim)
def forward(self, x):
# Flatten: [B, 1, 28, 28] → [B, 784]
x = x.view(x.size(0), -1)
# Linear: [B, 784] → [B, 10]
logits = self.fc(x)
return logits # Raw scores (no softmax)

Question: What is the complete training loop for logistic regression?
Answer: Training loop steps:
1. Set model to train mode: model.train()
2. Iterate over batches
3. Zero gradients: optimizer.zero_grad()
4. Forward pass: logits = model(x)
5. Compute loss: loss = criterion(logits, y)
6. Backward pass: loss.backward()
7. Update weights: optimizer.step()
8. Track metrics: loss and accuracy
Important: Always validate after each epoch!
Category: Implementation
Code: model.train()
for epoch in range(num_epochs):
for x, y in train_loader:
x, y = x.to(device), y.to(device)
# Zero gradients
optimizer.zero_grad()
# Forward
logits = model(x)
loss = criterion(logits, y)
# Backward
loss.backward()
optimizer.step()
# Validate
val_acc = validate(model, val_loader)

Question: How do you implement validation in PyTorch?
Answer: Validation best practices:
Key steps:
1. Set evaluation mode: model.eval()
2. Disable gradients: with torch.no_grad():
3. Compute metrics: loss and accuracy
4. NO weight updates!
Why model.eval()?
- Disables dropout (if present)
- Uses running stats for batch norm
- Ensures consistent evaluation
Category: Implementation
Code: def validate(model, loader, criterion, device):
model.eval() # Evaluation mode
total_loss = 0.0
correct = 0
total = 0
with torch.no_grad(): # No gradients
for x, y in loader:
x, y = x.to(device), y.to(device)
logits = model(x)
loss = criterion(logits, y)
total_loss += loss.item() * x.size(0)
preds = logits.argmax(dim=1)
correct += (preds == y).sum().item()
total += y.size(0)
return total_loss/total, correct/total

Question: How do you compute accuracy from logits?
Answer: Accuracy computation: Percentage of correct predictions
Steps:
1. Get predictions: argmax of logits (highest score class)
2. Compare with true labels
3. Compute percentage: correct / total
Important: Use argmax on logits, not on labels!
Category: Implementation
Code: # Get class predictions from logits
preds = logits.argmax(dim=1) # [batch_size]
# Compare with true labels
correct = (preds == y).sum().item()
total = y.size(0)
# Compute accuracy
accuracy = correct / total
# Or in one line:
acc = (logits.argmax(dim=1) == y).float().mean().item()

Question: What is L2 regularization (Ridge)?
Answer: L2 regularization penalizes large weights by adding the sum of squared weights to the loss.
Loss with L2:
L = CrossEntropy + λ × Σ wᵢ²
Effect:
- Encourages small weights (but non-zero)
- Prevents overfitting
- Smoother decision boundaries
- All features contribute (no feature selection)
Typical values: λ = 1e-5 to 1e-3
Category: Regularization
Code: # L2 in PyTorch: weight_decay parameter
optimizer = torch.optim.SGD(
model.parameters(),
lr=0.1,
weight_decay=1e-4 # L2 regularization
)

Question: What is L1 regularization (Lasso)?
Answer: L1 regularization penalizes the absolute value of weights.
Loss with L1:
L = CrossEntropy + λ × Σ |wᵢ|
Effect:
- Encourages sparse weights (many weights → 0)
- Performs feature selection
- Removes irrelevant features
- Less smooth than L2
Use when: High-dimensional data with many irrelevant features
Typical values: λ = 1e-6 to 1e-4
Category: Regularization
Code: # L1 requires manual implementation
def train_with_l1(model, x, y, l1_lambda=1e-5):
optimizer.zero_grad()
logits = model(x)
loss = criterion(logits, y)
# Add L1 penalty
l1_penalty = 0.0
for param in model.parameters():
l1_penalty += param.abs().sum()
total_loss = loss + l1_lambda * l1_penalty
total_loss.backward()
optimizer.step()

Question: What are the differences between L1 and L2 regularization?
Answer: L1 vs L2 comparison:
L2 (Ridge):
- Penalty: Σ wᵢ²
- Weights: Small, non-zero
- Feature selection: No
- PyTorch: weight_decay parameter
- Differentiable: Smooth everywhere
L1 (Lasso):
- Penalty: Σ |wᵢ|
- Weights: Sparse (many zeros)
- Feature selection: Yes
- PyTorch: Manual implementation
- Differentiable: Not at 0 (subgradient)
Category: Regularization

Question: What is Stochastic Gradient Descent (SGD)?
Answer: SGD updates weights in the opposite direction of the gradient.
Update rule:
w ← w - η × ∇L(w)
Where:
- w: weights
- η: learning rate
- ∇L(w): gradient of loss
Characteristics:
- Simple and interpretable
- Requires careful LR tuning
- Often better generalization
- Slower convergence than adaptive methods
Typical LR for MNIST: 0.01 - 0.1
Category: Optimizers
Code: # Basic SGD
optimizer = torch.optim.SGD(
model.parameters(),
lr=0.1
)
# SGD with momentum (faster convergence)
optimizer = torch.optim.SGD(
model.parameters(),
lr=0.1,
momentum=0.9
)

Question: What is the Adam optimizer?
Answer: Adam (Adaptive Moment Estimation) adapts the learning rate for each parameter.
Key features:
- Maintains running averages of gradients and squared gradients
- Adjusts LR per parameter
- Combines momentum + adaptive learning rates
Advantages:
- Faster convergence than SGD
- Less sensitive to initial LR
- Works well with default hyperparameters
- Good for rapid prototyping
Typical LR for MNIST: 1e-4 to 1e-3
Category: Optimizers
Code: optimizer = torch.optim.Adam(
model.parameters(),
lr=1e-3, # Typical default
betas=(0.9, 0.999), # Defaults (momentum terms)
eps=1e-8 # Numerical stability
)

Question: When should you use SGD vs Adam?
Answer: Choosing an optimizer:
Use Adam when:
- Rapid prototyping / experimentation
- Don't want to tune learning rate extensively
- Need fast convergence
- Noisy gradients or sparse data
Use SGD when:
- Final model after hyperparameter tuning
- Want best generalization
- Have time to tune LR carefully
- Training large models for long
For MNIST logistic regression:
- Adam: 92-93% accuracy, faster
- SGD: 91-92% accuracy, requires tuning
Category: Optimizers

Question: What is a confusion matrix?
Answer: Confusion matrix shows counts of true vs predicted classes.
Structure:
- Rows: True labels
- Columns: Predicted labels
- Diagonal: Correct predictions
- Off-diagonal: Errors (confusions)
What it reveals:
- Which classes are confused with each other
- Class-specific performance
- Common error patterns
For MNIST: Often see confusions between 4↔9, 3↔5, 3↔8, 7↔1
Category: Evaluation
Code: # Compute confusion matrix
cm = torch.zeros(10, 10, dtype=torch.int64)
model.eval()
with torch.no_grad():
for x, y in test_loader:
x, y = x.to(device), y.to(device)
preds = model(x).argmax(dim=1)
for true_label, pred_label in zip(y, preds):
cm[true_label, pred_label] += 1

Question: How do you visualize learned weights in logistic regression?
Answer: Weight visualization shows what each class's "template" looks like.
Interpretation:
- Red pixels: Positive weight (presence increases class score)
- Blue pixels: Negative weight (presence decreases class score)
- White pixels: ~Zero weight (neutral)
What it shows:
- Which pixels are important for each class
- Expected shape/pattern for each digit
- Model has learned interpretable features
Category: Evaluation
Code: # Extract weights
W = model.fc.weight.detach().cpu() # [10, 784]
# Visualize each class
fig, axes = plt.subplots(2, 5, figsize=(10, 4))
for cls in range(10):
ax = axes[cls // 5, cls % 5]
img = W[cls].view(28, 28) # Reshape to image
# Normalize for symmetric colormap
vmax = img.abs().max().item()
ax.imshow(img, cmap='seismic', vmin=-vmax, vmax=vmax)
ax.set_title(f'Class {cls}')
ax.axis('off')

Question: How does training data size affect model performance?
Answer: Performance vs data size for MNIST logistic regression:
Typical results:
- 10% data (~5k): 80-85% accuracy
- 25% data (~12.5k): 87-89% accuracy
- 50% data (~25k): 90-91% accuracy
- 100% data (50k): 92-93% accuracy
Key insights:
- More data consistently helps
- Diminishing returns after ~50%
- Even 10% gives reasonable performance
- Simple model (logistic regression) doesn't need huge data
Category: Evaluation

Question: How do noisy labels affect model performance?
Answer: Label noise corrupts training labels, causing the model to learn incorrect patterns.
Impact on MNIST (logistic regression):
- 0% noise: 92% accuracy (baseline)
- 10% noise: 88-89% accuracy (3-4% drop)
- 20% noise: 84-86% accuracy (6-8% drop)
Key findings:
- Label noise is more harmful than missing data
- Model learns from wrong labels → incorrect patterns
- Simple models (logistic regression) more robust than deep nets
- 10% noise → ~3-4% accuracy drop
Category: Evaluation
Code: # Simulate 10% label noise
class NoisyLabels(Dataset):
def __init__(self, base, noise_frac=0.1, num_classes=10):
self.base = base
n = len(base)
k = int(noise_frac * n)
self.noisy_idx = set(random.sample(range(n), k))
def __getitem__(self, idx):
x, y = self.base[idx]
if idx in self.noisy_idx:
# Random incorrect label
y = random.randint(0, self.num_classes-1)
return x, y

Question: What are common mistakes when using PyTorch CrossEntropyLoss?
Answer: Common CrossEntropyLoss mistakes:
1. Applying softmax before loss (WRONG):
CrossEntropyLoss already includes softmax internally!
2. Using one-hot labels (WRONG):
Targets should be class indices, not one-hot vectors
3. Wrong dimension (WRONG):
Logits should be [batch_size, num_classes]
4. Not flattening images (WRONG):
Must flatten 28×28 images to 784-dim vectors
Category: Common Mistakes
Code: # CORRECT
logits = model(x) # [batch, 10] raw scores
loss = nn.CrossEntropyLoss()(logits, y) # y: [batch] indices
# WRONG - double softmax
probs = F.softmax(model(x), dim=1)
loss = nn.CrossEntropyLoss()(probs, y)
# WRONG - one-hot labels
y_onehot = F.one_hot(y, 10)
loss = nn.CrossEntropyLoss()(logits, y_onehot)

Question: How do you debug a model that is not learning (loss not decreasing)?
Answer: Debugging checklist for poor convergence:
1. Check learning rate:
- Too high → loss explodes or oscillates
- Too low → very slow learning
- Try: 0.01, 0.1 for SGD; 1e-4, 1e-3 for Adam
2. Verify data normalization:
- Unnormalized data → poor gradients
- Check mean ≈ 0, std ≈ 1
3. Check loss function:
- Correct loss for task? (CrossEntropy for classification)
- Not applying softmax before CrossEntropyLoss?
4. Verify gradient flow:
- Check optimizer.zero_grad() is called
- Check loss.backward() is called
- Check optimizer.step() is called
Category: Debugging

Question: What are signs of overfitting and how do you fix it?
Answer: Overfitting symptoms:
- High train accuracy, low validation accuracy
- Train-val gap increases over epochs
- Validation loss increases while train loss decreases
Solutions for overfitting:
1. Regularization:
- Add L2: weight_decay=1e-4
- Add L1: manual penalty
2. More data:
- Collect more samples
- Data augmentation
3. Early stopping:
- Stop when validation accuracy plateaus
- Save best validation model
4. Simpler model:
- Logistic regression is already simple!
- Reduce features if too many
Category: Debugging

Question: What are best practices for model evaluation?
Answer: Evaluation best practices:
1. Use separate validation and test sets:
- Tune hyperparameters on validation
- Report final results on test
- NEVER touch test during development
2. Always use model.eval():
- Disables dropout, batch norm training mode
- Ensures reproducible evaluation
3. Disable gradients with torch.no_grad():
- Saves memory
- Faster computation
4. Track multiple metrics:
- Loss (for optimization)
- Accuracy (for interpretability)
- Confusion matrix (for error analysis)
Category: Best Practices
Code: # Proper evaluation
model.eval()
with torch.no_grad():
for x, y in test_loader:
x, y = x.to(device), y.to(device)
logits = model(x)
loss = criterion(logits, y)
acc = (logits.argmax(dim=1) == y).float().mean()

Question: What were the key findings from Assignment 1 experiments?
Answer: Assignment 1 experimental results:
L2 vs L1 regularization:
- L2 more stable and consistent
- L1 achieved feature selection but more volatile
- Both improved generalization over baseline
SGD vs Adam:
- Adam: faster convergence, higher accuracy (92-93%)
- SGD: required more tuning, 91-92% accuracy
- Adam recommended for MNIST logistic regression
Confusion matrix insights:
- Class 5 most confused (often misidentified as 3, 8)
- Common confusions: 4↔9, 3↔5, 3↔8, 7↔1
Data effects:
- 50% data → 90%+ accuracy (diminishing returns)
- 10% label noise → 3-4% accuracy drop
Category: Assignment Results

Question: What is the expected accuracy for logistic regression on MNIST?
Answer: Expected accuracies:
With proper implementation:
- Training: 93-95%
- Validation: 91-93%
- Test: 91-93%
Baseline (random guessing): 10%
If you're getting:
- > 93%: Excellent! Check for data leakage though
- 91-93%: Good, expected range
- 85-90%: Okay, but can improve (check LR, normalization)
- < 85%: Something wrong (check implementation)
- ~10%: Model not learning (check gradients, LR, loss)
Category: Assignment Results