Title: Assignment 6: Semantic and Instance Segmentation

Question: What is semantic segmentation?
Answer: Semantic segmentation classifies each pixel into a category.
Input: RGB Image $(H \times W \times 3)$
Output: Label Map $(H \times W)$
- Every pixel gets a class label
- Does NOT distinguish instances
- Example: All cars → "car"
Category: Fundamentals

Question: What is instance segmentation and how does it differ from semantic segmentation?
Answer: Instance = Semantic + Object Detection
Semantic: All cars → "car"
Instance: car₁, car₂, car₃
Formula: Instance Segmentation = Semantic Segmentation + Object Detection
Category: Fundamentals

Question: Why use Fully Convolutional Networks for segmentation?
Answer: FCN preserves spatial structure!
Advantages:
- Any input size
- Spatial info preserved
- Efficient computation
- End-to-end learning
Category: FCN
Code: # FCN (good)
x = self.conv(x) # Spatial preserved!
# FC (bad)
x = x.view(-1) # Spatial destroyed!

Question: What is transposed convolution?
Answer: Learnable upsampling operation
Formula: $\text{output_size} = (\text{input} - 1) \times \text{stride} - 2p + k + \text{out_pad}$
Example: $2\times2 \to 5\times5$ with $k=3, s=2$
Category: Transposed Convolution
Code: nn.ConvTranspose2d(64, 32, 3, stride=2, padding=1)
# [B,64,16,16] → [B,32,32,32]

Question: What are skip connections and why are they critical?
Answer: Connect encoder to decoder at same resolution
Why critical:
- Preserve spatial details
- Better gradients
- Sharp boundaries
- Combine low+high level features
Category: U-Net
Code: d4 = torch.cat([upsampled, e4], dim=1) # Skip!

Question: What is U-Net architecture?
Answer: Symmetric encoder-decoder with skips
Structure: Input → Encoder (down) → Bottleneck → Decoder (up) + Skips → Output
Key: Concatenation skip connections
Category: U-Net
Code: # Encoder
e1 = enc1(x) # 256×256
e2 = enc2(pool(e1)) # 128×128
# Decoder with skips
d1 = dec1(cat([up(d2), e1]))

Question: What is Mask R-CNN?
Answer: Extends Faster R-CNN with mask prediction
Pipeline: Backbone → RPN → RoI Align → 3 Heads (class, box, mask)
Loss: $L = L_{cls} + L_{box} + L_{mask}$
Category: Mask R-CNN

Question: Why is RoI Align better than RoI Pool?
Answer: RoI Align preserves spatial precision
RoI Pool: Quantizes coords (lossy)
RoI Align: Bilinear interpolation (precise)
Impact: ~3-5% AP improvement
Category: Mask R-CNN
Code: roi_align = torchvision.ops.roi_align(
features, boxes, output_size=7
)

Question: What is Cross-Entropy Loss for segmentation?
Answer: Standard loss for semantic segmentation
$$L_{CE} = -\sum_{pixels} \sum_{c} y_c \log(p_c)$$
Input: logits $[B, C, H, W]$
Target: indices $[B, H, W]$
Category: Loss Functions
Code: criterion = nn.CrossEntropyLoss()
loss = criterion(outputs, targets)

Question: What is Dice Loss?
Answer: Optimizes overlap directly
$$\text{Dice} = \frac{2|P \cap G|}{|P| + |G|}$$
$$L_{Dice} = 1 - \text{Dice}$$
Good for: Class imbalance, medical imaging
Category: Loss Functions
Code: intersection = (pred * target).sum()
dice = 2*intersection / (pred.sum() + target.sum())
loss = 1 - dice

Question: What is Focal Loss?
Answer: Focuses on hard examples
$$L_{Focal} = -\alpha(1-p_t)^{\gamma} \log(p_t)$$
Easy ($p_t \approx 1$): down-weighted
Hard ($p_t \approx 0$): full weight
Typical: $\alpha=0.25, \gamma=2$
Category: Loss Functions
Code: ce = F.cross_entropy(pred, target, reduction='none')
p_t = torch.exp(-ce)
focal = alpha * (1-p_t)**gamma * ce

Question: What is IoU (Intersection over Union)?
Answer: Most common segmentation metric
$$\text{IoU} = \frac{|P \cap G|}{|P \cup G|}$$
mIoU = Average IoU across all classes
Better than pixel accuracy for imbalance
Category: Evaluation
Code: intersection = (pred & target).sum()
union = (pred | target).sum()
iou = intersection / union

Question: Relationship between Dice and IoU?
Answer: Mathematically related metrics
$$\text{Dice} = \frac{2 \cdot \text{IoU}}{1 + \text{IoU}}$$
$$\text{IoU} = \frac{\text{Dice}}{2 - \text{Dice}}$$
Dice ≥ IoU (more forgiving)
Category: Evaluation
Code: def iou_to_dice(iou):
return 2*iou / (1+iou)
def dice_to_iou(dice):
return dice / (2-dice)

Question: How are labels formatted for segmentation?
Answer: Single-channel with class indices
Format:
Image: RGB $(H \times W \times 3)$
Label: Grayscale $(H \times W)$
Critical:
- Class indices (not one-hot!)
- dtype: torch.long
- Save as PNG (not JPEG)
Category: Data Preparation
Code: # CORRECT
labels = torch.zeros(H, W, dtype=torch.long)
labels[i,j] = class_id # 0 to C-1
# WRONG
labels = torch.zeros(H, W, C) # One-hot

Question: How to apply augmentation to image and mask?
Answer: CRITICAL: Same geometric transform for both!
Strategy:
1. Set random seed
2. Apply geometric to image
3. Reset same seed
4. Apply geometric to mask
5. Apply photometric to image only
Category: Data Preparation
Code: seed = np.random.randint(2**32)
random.seed(seed)
image = transform(image)
random.seed(seed) # SAME seed!
mask = transform(mask)

Question: Best upsampling method for segmentation?
Answer: Bilinear interpolation + Conv (recommended)
Options:
1. Transposed conv: Learnable, but artifacts
2. Bilinear: Smooth, not learnable
3. Bilinear + Conv: Best of both!
Category: Upsampling
Code: # BEST
nn.Sequential(
nn.Upsample(scale_factor=2, mode='bilinear'),
nn.Conv2d(64, 32, 3, padding=1)
)

Question: How to handle class imbalance in segmentation?
Answer: Multiple strategies
Solutions:
1. Weighted CE: $w_c = 1/\text{freq}_c$
2. Focal loss
3. Dice loss
4. Combined: CE + Dice
Best: $L = 0.5 L_{CE} + 0.5 L_{Dice}$
Category: Training
Code: class_weights = 1.0 / class_counts
criterion = nn.CrossEntropyLoss(weight=class_weights)
# Or combined
total_loss = 0.5*ce_loss + 0.5*dice_loss

Question: What is DeepLab v3+ key innovation?
Answer: Atrous convolution + ASPP
Atrous conv: Enlarges receptive field without pooling
ASPP: Multi-scale context with rates [1,6,12,18]
State-of-the-art on PASCAL VOC, Cityscapes
Category: State-of-the-Art
Code: nn.Conv2d(64, 64, 3, dilation=6)
# Receptive field: larger, params: same

Question: What is SAM (Segment Anything Model)?
Answer: Foundation model for zero-shot segmentation
Training: 11M images, 1B masks
Features:
- Promptable (point, box, mask)
- Zero-shot capability
- Interactive segmentation
Category: State-of-the-Art
Code: from segment_anything import SamPredictor
predictor = SamPredictor(sam)
masks, _, _ = predictor.predict(
point_coords=[[x, y]],
point_labels=[1] # foreground
)

Question: Most common segmentation implementation mistake?
Answer: Wrong label format!
WRONG: One-hot $(H,W,C)$
CORRECT: Class indices $(H,W)$
Other mistakes:
- Not calling model.eval()
- Different augmentations for image/mask
- No skip connections
- Ignoring class imbalance
Category: Common Pitfalls
Code: # CORRECT
labels = torch.zeros(H, W, dtype=torch.long)
labels[i,j] = class_id
loss = nn.CrossEntropyLoss()(outputs, labels)