Title: CMPUT328 Assignment 5 - Object Detection

Question: What are the key differences between R-CNN, Fast R-CNN, and Faster R-CNN?
Answer: R-CNN (2014): ~49 sec/image
- Selective Search for proposals
- Run CNN ~2000 times per image
- Very slow
Fast R-CNN (2015): ~2.3 sec/image
- Pass image through CNN only once
- ROI pooling on feature map
- Still uses selective search (bottleneck)
Faster R-CNN (2016): ~0.2 sec/image
- Replace selective search with RPN
- Fully neural, end-to-end trainable
- Uses anchor boxes
Category: Historical Evolution

Question: What is selective search and why did Faster R-CNN replace it?
Answer: Selective Search:
- Pre-deep learning algorithm
- Generates ~2000 region proposals per image
- Based on hierarchical grouping (color, texture, size)
- Not learnable, hand-crafted heuristics
- Slow (bottleneck in Fast R-CNN)
Replaced by:
- Region Proposal Network (RPN) in Faster R-CNN
- Neural network generates proposals
- Learnable, fast, end-to-end trainable
- Generates ~300 proposals
Category: Historical Evolution

Question: What is ROI pooling and why is it important?
Answer: ROI Pooling:
Converts variable-sized regions into fixed-size features
Process:
1. Project region proposal onto feature map
2. Divide region into grid (e.g., 2×2)
3. Max pool within each grid cell
4. Output: fixed-size feature vector
Importance:
- Enables variable-sized regions → fixed FC layer input
- Faster than R-CNN (no need to pass each region through CNN)
- Differentiable (can backpropagate)
Problem: Quantization causes slight misalignment
Solution: ROI Align (Mask R-CNN) uses bilinear interpolation
Category: Historical Evolution

Question: What are anchor boxes and why are they used?
Answer: Anchor Boxes:
Pre-defined bounding box shapes at each location
Why needed:
- Neural networks need fixed-size outputs
- Objects have variable sizes/shapes
- Solution: Pre-define K shapes at each location
Network predicts:
- Offsets from anchors (adjustments)
- Objectness score (does anchor contain object?)
Common configuration:
- 3 scales × 3 aspect ratios = 9 anchors per location
- Aspect ratios: 1:1, 1:2, 2:1 (square, tall, wide)
Used in: Faster R-CNN, YOLOv2-v7, RetinaNet
Category: Anchors & Region Proposals

Question: How does Region Proposal Network (RPN) work in Faster R-CNN?
Answer: RPN Architecture:
At each location in feature map:
1. Place K anchor boxes (different scales/ratios)
2. For each anchor, predict:
- Objectness score (1 value): object present?
- Box transforms (4 values): how to adjust anchor?
Output per location: K × 5 values
Post-processing:
1. Apply objectness threshold (e.g., > 0.5)
2. Transform anchors using predicted offsets
3. Apply NMS to remove duplicates
4. Keep top ~300 proposals
Training:
Two-stage: Train RPN, then train detection heads
Category: Anchors & Region Proposals

Question: Why does selective search not need anchors but RPN does?
Answer: Selective Search (no anchors):
- Bottom-up approach
- Based on image segmentation
- Can generate arbitrary shapes/sizes
- Not constrained by pre-defined shapes
- Downside: Not learnable, slow
RPN with Anchors:
- Top-down neural network approach
- Needs fixed-size outputs for predictions
- Anchors provide starting points
- Network predicts adjustments
- Upside: Learnable, fast, differentiable
Key insight: Neural nets need structure, anchors provide it
Category: Anchors & Region Proposals

Question: How does YOLOv1 work and what is its key innovation?
Answer: YOLOv1 Innovation:
Single-stage detector - predict boxes and classes in one pass
Architecture:
1. Divide image into S×S grid (e.g., 7×7)
2. Each grid cell predicts:
- B bounding boxes (x, y, w, h, confidence)
- C class probabilities
3. Output: S × S × (B×5 + C) tensor
Confidence score:
Pr(Object) × IoU(pred, truth)
Advantages:
- Very fast (> 10× faster than Faster R-CNN)
- Real-time performance (45+ FPS)
Disadvantages:
- Struggles with small objects
- Multiple objects in same grid cell
Category: YOLO

Question: What are the major improvements from YOLOv1 to YOLOv8?
Answer: YOLOv2 (2017):
- Batch normalization
- Anchor boxes
- Multi-scale training
YOLOv3 (2018):
- FPN (Feature Pyramid Network)
- 3-scale predictions
- Better small object detection
YOLOv4 (2020):
- CSPDarknet backbone
- Mosaic augmentation
- CIoU loss
YOLOv8 (2023):
- Anchor-free design
- Decoupled heads (classification + regression separate)
- C2f blocks
- Improved performance and speed
Category: YOLO

Question: What is the difference between anchor-based and anchor-free detectors?
Answer: Anchor-Based (YOLOv2-v7, Faster R-CNN):
- Pre-define K box shapes at each location
- Network predicts offsets from anchors
- Need to tune anchor sizes/ratios
- More hyperparameters
Anchor-Free (YOLOv8, CenterNet, FCOS):
- Direct prediction of box coordinates
- No pre-defined shapes
- Simpler, fewer hyperparameters
- Better generalization
YOLOv8 anchor-free approach:
- Predict box center and size directly
- No need for anchor clustering
- Easier to use, better performance
Category: YOLO

Question: How does CenterNet work as an anchor-free detector?
Answer: CenterNet: Objects as Points
Key idea: Represent each object as single center point
Architecture:
1. Backbone CNN → Feature map
2. Three prediction heads:
a) Heatmap: Gaussian peaks at object centers
b) Size head: Width and height at center
c) Offset head: Sub-pixel correction
Training:
- Ground truth: Gaussian heatmap around centers
- Focal loss for heatmap
- L1 loss for size and offset
Inference:
1. Find local maxima in heatmap
2. Read size and offset at each peak
3. Reconstruct bounding boxes
Advantages:
- No anchors needed
- No NMS needed (few duplicates)
- Simple and fast
Category: Modern Approaches

Question: What is DETR and how does it differ from traditional detectors?
Answer: DETR: Detection Transformer
Philosophy:
- Set prediction problem
- No hand-crafted components
- No anchors, no NMS
Architecture:
1. CNN backbone → Feature map
2. Flatten + positional encoding
3. Transformer encoder-decoder
4. N object queries → N predictions (class + bbox)
Object Queries:
- N learned embeddings (e.g., N=100)
- Each predicts one object or "no object"
- Transformer assigns queries to objects
Hungarian Matching:
- Bipartite matching between predictions and GT
- Finds optimal 1-to-1 assignment
- Loss computed on matched pairs only
Problem: Slow convergence (hundreds of epochs)
Category: Modern Approaches

Question: Why is Hungarian matching in DETR non-differentiable and how does training still work?
Answer: Non-Differentiable Problem:
- Hungarian algorithm uses discrete argmin/argmax
- No gradient through matching step
Training Still Works:
1. Matching done in torch.no_grad()
2. Matching provides indices (which pred matches which GT)
3. Loss computed using these indices (loss IS differentiable)
4. Gradients flow through predictions, not matching
5. Effectively: straight-through estimator
Issues:
- Noisy early supervision (random matches)
- Discontinuous loss when assignments flip
- Slow convergence
Solutions (later work):
- Deformable DETR: Multi-scale attention
- DN-DETR: Add noised GT queries
- Conditional DETR: Condition queries on spatial priors
Category: Modern Approaches

Question: What are the common bounding box formats and how to convert between them?
Answer: Formats:
1. XYXY: (x1, y1, x2, y2) - corners
2. XYWH: (x, y, w, h) - top-left + size
3. CXCYWH: (cx, cy, w, h) - center + size
4. YOLO: (cx, cy, w, h) normalized to [0,1]
Conversions:
XYWH to XYXY:
x2 = x + w, y2 = y + h
XYXY to YOLO (normalized):
cx = (x1 + x2) / (2 × width)
cy = (y1 + y2) / (2 × height)
w = (x2 - x1) / width
h = (y2 - y1) / height
Category: Bounding Boxes & IoU

Question: How do you compute IoU and what are typical thresholds?
Answer: IoU Formula: IoU = Intersection Area / Union Area Where: Union = Area_A + Area_B - Intersection Range: [0, 1] (0 = no overlap, 1 = perfect) Typical Thresholds: - IoU ≥ 0.5: Good match (standard for TP) - IoU ≥ 0.7: Strong match - IoU ≥ 0.9: Excellent match - IoU < 0.5: False positive COCO Evaluation: - mAP@0.5: Lenient - mAP@0.75: Strict - mAP@[0.5:0.95]: Average over multiple thresholds
Category: Bounding Boxes & IoU
Code: inter_area = max(0, min(x2_1, x2_2) - max(x1_1, x1_2)) * max(0, min(y2_1, y2_2) - max(y1_1, y1_2))
union = area1 + area2 - inter_area
iou = inter_area / (union + 1e-6)

Question: What is Non-Maximum Suppression (NMS) and when is it applied?
Answer: NMS Purpose:
Remove duplicate detections of same object
Algorithm:
1. Sort detections by confidence (high → low)
2. While detections remain:
a. Take highest confidence detection
b. Add to final output
c. Remove all with IoU > threshold (e.g., 0.45)
3. Return final detections
When Applied:
- Only during inference (NOT training)
- After confidence thresholding
- Per class separately
Hyperparameters:
- Confidence threshold (e.g., 0.25): filter low-conf
- IoU threshold (e.g., 0.45): suppress duplicates
Note: Some detectors (CenterNet, DETR) don't need NMS
Category: NMS

Question: What is the difference between confidence threshold and IoU threshold in NMS?
Answer: Confidence Threshold (e.g., 0.25):
- Applied BEFORE NMS
- Filters out low-confidence predictions
- Lower → more detections (higher recall, lower precision)
- Higher → fewer detections (lower recall, higher precision)
IoU Threshold for NMS (e.g., 0.45):
- Applied DURING NMS
- Controls duplicate suppression
- Lower → more aggressive (fewer duplicates, may remove valid)
- Higher → less aggressive (more duplicates)
Trade-offs:
- Both affect final precision-recall balance
- Confidence threshold affects detection sensitivity
- IoU threshold affects duplicate handling
Category: NMS

Question: What is the difference between TP, FP, and FN in object detection?
Answer: True Positive (TP): - Correct detection - IoU ≥ threshold (e.g., 0.5) - Correct class - Each GT can only match one prediction False Positive (FP): - Incorrect detection - IoU < threshold OR wrong class - Duplicate detection (already matched GT) False Negative (FN): - Missed object - No prediction matched this ground truth True Negative (TN): - Not applicable (infinite background) Matching: Usually greedy or Hungarian algorithm
Category: Evaluation Metrics

Question: How is Average Precision (AP) calculated?
Answer: AP Calculation:
1. Sort all detections by confidence (high to low)
2. For each detection:
- Determine if TP or FP (using IoU threshold)
- Compute cumulative precision and recall
3. Plot precision-recall curve
4. AP = Area under the interpolated P-R curve
Formula:
AP = Σ (Rₙ - Rₙ₋₁) × Pₙ
Where:
- Rₙ = recall at nth unique threshold
- Pₙ = precision at that threshold
Interpolation:
- PASCAL VOC: 11-point (r ∈ {0, 0.1, ..., 1.0})
- COCO: All-point (more accurate)
AP is computed PER CLASS
Category: Evaluation Metrics

Question: What is mAP and how does it differ from AP?
Answer: AP (Average Precision):
- Area under precision-recall curve
- Computed for ONE class
- Summarizes performance across confidence thresholds
mAP (mean Average Precision):
- Average of AP across ALL classes
- mAP = (1/N) × Σ AP_i for N classes
COCO Metrics:
- mAP or mAP@[0.5:0.95]: Average over IoU 0.5, 0.55, ..., 0.95
- mAP@0.5: IoU threshold = 0.5 (lenient)
- mAP@0.75: IoU threshold = 0.75 (strict)
- mAP_small, mAP_medium, mAP_large: By object size
Primary metric: mAP@[0.5:0.95] (COCO standard)
Category: Evaluation Metrics

Question: How do precision and recall trade-off in object detection?
Answer: Definitions:
Precision = TP / (TP + FP) - How many detections correct?
Recall = TP / (TP + FN) - How many objects found?
Trade-off controlled by confidence threshold:
Low Confidence Threshold:
- More detections (including low-confidence)
- Higher recall (find more objects)
- Lower precision (more false positives)
High Confidence Threshold:
- Fewer detections (only high-confidence)
- Lower recall (miss some objects)
- Higher precision (fewer false positives)
F1 Score balances both:
F1 = 2 × (Precision × Recall) / (Precision + Recall)
Ideal: High precision AND high recall (difficult!)
Category: Evaluation Metrics

Question: What is transfer learning and why is it important for object detection?
Answer: Transfer Learning:
Start with pre-trained weights, fine-tune on custom dataset
Pre-training:
- Large dataset (COCO: 80 classes, 120k images)
- Learn general features (edges, shapes, objects)
Fine-tuning:
- Custom dataset (often smaller)
- Adapt features to new domain/classes
Benefits:
1. Faster convergence (fewer epochs)
2. Better performance with less data
3. Features transfer across domains
4. Lower risk of overfitting
Strategy:
- Start with pre-trained weights (e.g., yolov8n.pt)
- Lower learning rate (0.001-0.01)
- Optional: Freeze backbone, train head only
Category: Training

Question: What data augmentation techniques are used in YOLO training?
Answer: Common Augmentations:
Mosaic (1.0):
- Combine 4 images into one
- Exposes model to more objects
- Standard in YOLOv4+
Horizontal Flip (0.5):
- 50% chance mirror image
- Good for symmetric objects
HSV Color Jitter:
- Vary hue, saturation, value
- Improves robustness to lighting
Scale (0.5):
- Zoom in/out (±50%)
- Handles different object sizes
Translation (0.1):
- Shift image by 10%
- Handles object positions
Rotation (0.0 for digits):
- Avoid for upright objects
- Use for orientation-invariant objects
Category: Training

Question: What are the two tasks in CMPUT 328 Assignment 5?
Answer: Part A: Single-class detection
- Fine-tune YOLOv8n on MNISTDD-RGB
- 1 class: generic "digit"
- Goal: Detect any digit (0-9) as one class
- Expected: mAP@0.5 ~ 90-95%, IoU ~ 80-85%
Part B: Multi-class detection
- Fine-tune YOLOv8n on MNISTDD-RGB
- 10 classes: digits 0-9
- Goal: Detect AND classify which digit
- Expected: mAP@0.5 ~ 85-90%, IoU ~ 75-80%
Dataset:
- 64×64 RGB images
- 1-3 MNIST digits per image
- Format: NPZ files with images and bboxes
Category: Assignment Specifics

Question: What is the "Mean IoU (matched)" metric used in Assignment 5?
Answer: Mean IoU (Matched):
Measures localization quality of correct detections
Algorithm:
For each image:
1. Run model → predictions
2. Compute IoU matrix (N_pred × N_gt)
3. Greedy matching:
- For each GT, find best prediction (max IoU)
- Match only if IoU ≥ 0.5
- Each prediction matched at most once
4. Compute mean IoU of matched pairs
5. If no matches, IoU = 0 for this image
Final: Average over all images
What it measures:
- How well model localizes detected objects
- Ignores missed detections (FN)
- Only evaluates matched detections
Category: Assignment Specifics
Code: for gt_idx in range(N_gt):
best_pred = argmax(iou_matrix[:, gt_idx])
best_iou = iou_matrix[best_pred, gt_idx]
if best_iou >= 0.5 and best_pred not in used:
matches.append(best_iou)
used.add(best_pred)

Question: How do you convert MNISTDD-RGB data to YOLO format for training?
Answer: Steps:
1. Load NPZ data:
images: (N, 64, 64, 3)
bboxes: (N, max_objs, 4) in XYXY format
2. For each image:
a) Save image as JPG
b) Convert boxes to YOLO format:
- XYXY → center (cx, cy), size (w, h)
- Normalize by image dimensions → [0, 1]
c) Save label file (one line per object)
3. Create dataset.yaml:
- paths: train/val images and labels
- class names (0-9 for digits)
Label format (per line):
class_id cx cy w h (all normalized)
Example: 3 0.500 0.500 0.200 0.300
Category: Assignment Specifics
Code: cx = (x1 + x2) / (2 * width)
cy = (y1 + y2) / (2 * height)
w = (x2 - x1) / width
h = (y2 - y1) / height

Question: How has object detection performance evolved on COCO?
Answer: Historical mAP@[0.5:0.95] on COCO:
Fast R-CNN (2015): 21%
Faster R-CNN (2016): 37%
Mask R-CNN (2017): 42%
YOLOv3 (2018): 47%
Cascade R-CNN (2019): 50%
DetectoRS (2020): 55%
DETR v2 (2023): 66%
Co-DETR (2024): 67%
Trends:
- Two-stage historically more accurate
- Single-stage catching up (YOLOv8+)
- Transformers (DETR) competitive
- Improvements slowing (diminishing returns)
Current SOTA: ~67% mAP (Co-DETR, Group DETR)
Category: Performance & Comparison

Question: What are the speed vs accuracy trade-offs in object detection?
Answer: Two-Stage Detectors (Faster R-CNN, Cascade R-CNN):
- Speed: 5-15 FPS
- Accuracy: Higher mAP
- Use case: When accuracy is critical
Single-Stage (YOLO, SSD, RetinaNet):
- Speed: 30-100+ FPS
- Accuracy: Lower but improving
- Use case: Real-time applications
Transformer-Based (DETR, Deformable DETR):
- Speed: 10-20 FPS
- Accuracy: Competitive
- Use case: Research, high-quality predictions
Model Size Trade-offs:
- YOLOv8n (nano): Fastest, lowest accuracy
- YOLOv8s/m (small/medium): Balanced
- YOLOv8l/x (large/xlarge): Highest accuracy, slower
Category: Performance & Comparison