Title: CMPUT 328 Assignment 4: Vision Transformers & LoRA

Question: What is a Vision Transformer (ViT)?
Answer: Vision Transformer (ViT) is a transformer architecture adapted for computer vision by treating image patches as tokens.
Key innovation:
- Applies self-attention directly to image patches
- No convolutions required
- Global receptive field from layer 1
Compared to CNNs:
- CNNs: local receptive fields, strong inductive biases
- ViT: global receptive field, minimal inductive biases
- ViT needs more data but scales better
Category: ViT Fundamentals

Question: How do you convert an image into patches for ViT?
Answer: Patch extraction divides the image into fixed-size non-overlapping patches.
For CIFAR-10 (32×32) with patch_size=4:
- Patches per dimension: 32 ÷ 4 = 8
- Total patches: 8 × 8 = 64 patches
- Each patch: 4×4×3 = 48 values
- Flattened dimension: 48
Process:
1. Divide image into patches using unfold()
2. Flatten each patch: [B, 64, 48]
3. Linear projection to embed_dim: [B, 64, 256]
Category: ViT Fundamentals
Code: # Patch extraction
patches = x.unfold(2, patch_size, patch_size)
.unfold(3, patch_size, patch_size)
# Flatten: [B, num_patches, C*P*P]
patches = patches.view(b, num_patches, -1)

Question: What is the purpose of the [CLS] token in ViT?
Answer: The [CLS] token is a learnable embedding prepended to the patch sequence for classification.
Purpose:
- Aggregates global information from all patches
- Final representation used for classification
- Similar to BERT's [CLS] token in NLP
How it works:
1. Prepend [CLS] token to patch embeddings
2. Pass through all transformer layers
3. Self-attention allows [CLS] to attend to all patches
4. Extract [CLS] output → classification head
Shape: [1, 1, embed_dim], expanded to [B, 1, embed_dim]
Category: ViT Architecture
Code: # Add CLS token
cls_tokens = self.cls_token.expand(batch_size, -1, -1)
x = torch.cat((cls_tokens, patch_embeddings), dim=1)
# After transformer layers
cls_output = x[:, 0] # Extract [CLS]
logits = self.mlp_head(cls_output)

Question: Why are positional embeddings necessary in ViT?
Answer: Transformers have no inherent spatial awareness - the model can't tell which patch came from which position.
Without positional encoding:
- All permutations of patches would give same output
- Model can't use spatial relationships
- Can't distinguish top-left from bottom-right
Solution - Positional embeddings:
- Learnable: Trained from scratch (not fixed sinusoidal)
- Added element-wise: x = patches + pos_embedding
- Shape: [1, num_patches+1, embed_dim]
- Include [CLS]: Position for [CLS] token also learned
Category: ViT Architecture
Code: self.pos_embedding = nn.Parameter(
torch.zeros(1, num_patches + 1, embed_dim)
)
# In forward pass
x = patch_embeddings + self.pos_embedding[:, :x.size(1)]

Question: What is self-attention and how does it work?
Answer: Self-attention allows each token (patch) to attend to all other tokens, learning relationships between different parts of the input.
Formula:
Attention(Q, K, V) = softmax(Q × K^T / √d_k) × V
Steps:
1. Compute Q, K, V: Linear projections of input
2. Calculate scores: Q × K^T (how much to attend)
3. Scale: Divide by √d_k (prevents large values)
4. Normalize: Softmax (attention weights sum to 1)
5. Weighted sum: Multiply by V (final output)
For vision: Each patch can attend to any other patch, capturing long-range dependencies.
Category: Self-Attention
Code: # Simplified self-attention
Q = x @ W_Q
K = x @ W_K
V = x @ W_V
scores = Q @ K.T / sqrt(d_k)
attn_weights = softmax(scores)
output = attn_weights @ V

Question: What is multi-head attention?
Answer: Multi-head attention runs multiple attention operations in parallel, allowing the model to attend to different aspects simultaneously.
For num_heads=8, embed_dim=256:
- Split 256 dimensions into 8 heads of 32 dimensions each
- Each head learns different relationships
- Concatenate outputs from all heads
- Final linear projection combines information
Benefits:
- Capture multiple types of relationships
- Some heads focus on color, others on shape, etc.
- More expressive than single attention
Category: Self-Attention
Code: self.attn = nn.MultiheadAttention(
embed_dim=256,
num_heads=8,
batch_first=True
)
# Each head dimension: 256 / 8 = 32

Question: What is the complete forward pass of a Vision Transformer?
Answer: ViT forward pass pipeline:
1. Patch extraction: Split image into patches [B, 64, 48]
2. Linear projection: Embed patches [B, 64, 256]
3. Add [CLS] token: Prepend [B, 65, 256]
4. Add positional encoding: x = x + pos_emb
5. Dropout: Regularization
6. Transformer blocks: Repeated N times (6 layers)
  - LayerNorm → Multi-head attention → Residual
  - LayerNorm → Feed-forward → Residual
7. Extract [CLS]: cls = x[:, 0]
8. Classification head: LayerNorm → Linear → logits
Category: ViT Architecture
Code: def forward(self, x):
patches = img_to_patch(x, self.patch_size)
tokens = self.patch_embed(patches)
cls_tokens = self.cls_token.expand(B, -1, -1)
x = torch.cat((cls_tokens, tokens), dim=1)
x = x + self.pos_embedding[:, :x.size(1)]
x = self.dropout(x)
for block in self.transformer:
x = block(x)
cls = x[:, 0]
return self.mlp_head(cls)

Question: What is Pre-LN vs Post-LN in transformers?
Answer: LayerNorm placement affects training stability:
Post-LN (original Transformer):
x = LayerNorm(x + Attention(x))
- Normalize after residual connection
- Less stable, harder to train deep models
Pre-LN (modern default):
x = x + Attention(LayerNorm(x))
- Normalize before attention/feed-forward
- More stable training
- Easier to train deep transformers
- Used in ViT
Category: ViT Architecture
Code: # Pre-LN (ViT uses this)
attn_input = self.norm1(x)
attn_out = self.attn(attn_input, attn_input, attn_input)
x = x + attn_out # Residual
ff_input = self.norm2(x)
x = x + self.ff(ff_input) # Residual

Question: What optimizer and learning rate schedule is used for training ViT?
Answer: AdamW optimizer with Cosine Annealing LR:
AdamW:
- Adam with decoupled weight decay
- Learning rate: 3e-4 (typical for ViT)
- Weight decay: 5e-5 (regularization)
Cosine Annealing LR:
- η_t = η_min + (η_max - η_min) × (1 + cos(πt/T)) / 2
- Gradually decreases LR following cosine curve
- Smooth decay over training
Gradient clipping: max_norm=1.0 (prevents exploding gradients)
Category: Training
Code: optimizer = optim.AdamW(
model.parameters(),
lr=3e-4,
weight_decay=5e-5
)
scheduler = optim.lr_scheduler.CosineAnnealingLR(
optimizer,
T_max=max_epochs
)

Question: Why does ViT perform worse than CNN on small datasets like CIFAR-10?
Answer: Inductive biases built into CNNs help with small datasets, but ViT lacks these biases.
CNN inductive biases:
- Locality: Nearby pixels are related (convolutions)
- Translation equivariance: Same features anywhere in image
- Hierarchical features: Edge → texture → object
ViT approach:
- Minimal inductive biases - learns from data
- Needs large datasets to learn spatial relationships
- More flexible but data-hungry
Results on CIFAR-10 (50k samples):
- CNN from scratch: ~85-90%
- ViT from scratch: ~75-80%
- Pre-trained ViT: ~90-95% ✓
Category: ViT vs CNN

Question: What is LoRA (Low-Rank Adaptation)?
Answer: LoRA is a parameter-efficient fine-tuning method that freezes pre-trained weights and injects small trainable matrices.
Concept:
W_new = W_frozen + B × A
- W: [d × k] original weight (FROZEN)
- A: [r × k] trainable matrix
- B: [d × r] trainable matrix
- r: rank (r << min(d, k))
Benefits:
- Memory efficient: Train <1% of parameters
- Modular: Swap adapters for different tasks
- Fast: Fewer parameters to update
- No inference overhead: Merge B×A into W
Category: LoRA
Code: lora_config = LoraConfig(
r=16, # Rank (bottleneck)
lora_alpha=32, # Scaling (2×r)
lora_dropout=0.05,
target_modules=[ # Which layers
"attn.c_attn", "attn.c_proj",
"mlp.c_fc", "mlp.c_proj"
]
)

Question: How many parameters does LoRA train compared to full fine-tuning?
Answer: LoRA dramatically reduces trainable parameters:
Example (GPT-2 decoder in ViT-GPT2):
- Total parameters: ~100M
- LoRA trainable: ~0.5M (0.5%)
- Frozen: ~99.5M
Calculation for one weight matrix:
Original: d × k parameters
LoRA: r × (d + k) parameters
For d=768, k=768, r=16:
- Original: 768 × 768 = 589,824
- LoRA: 16 × (768 + 768) = 24,576
- Reduction: ~96%
Category: LoRA
Code: model.decoder.print_trainable_parameters()
# Output:
# trainable params: 552,960 || all params: 124,439,808
# trainable%: 0.4442%

Question: What is the rank (r) in LoRA and how do you choose it?
Answer: Rank (r) is the bottleneck dimension in LoRA decomposition, controlling capacity vs efficiency.
Choosing r:
- r = 4-8: Very efficient, may underfit
- r = 16-32: Good balance (common choice) ✓
- r = 64+: More capacity, diminishing returns
Trade-off:
- Lower r: Fewer parameters, faster, less capacity
- Higher r: More parameters, slower, more capacity
Scaling factor (lora_alpha):
- Typically 2×r (e.g., r=16, alpha=32)
- Scales LoRA update: ΔW = (alpha/r) × B × A
Category: LoRA

Question: What is CLIP and how does it work?
Answer: CLIP (Contrastive Language-Image Pre-training) learns to match images with text descriptions.
Training:
- Trained on 400M image-text pairs from internet
- Contrastive loss: match correct pairs, separate incorrect pairs
- No class labels needed!
Architecture:
- Image encoder: ViT or ResNet
- Text encoder: Transformer
- Output: Aligned embedding space for images and text
Key capability: Zero-shot classification without seeing target dataset!
Category: CLIP
Code: # CLIP training (conceptual)
image_features = image_encoder(images)
text_features = text_encoder(texts)
# Contrastive loss
similarity = image_features @ text_features.T
loss = contrastive_loss(similarity, labels)

Question: How do you perform zero-shot classification with CLIP?
Answer: Zero-shot classification predicts classes without training on the target dataset.
Steps:
1. Encode class names:
text_features = encode_text(["a dog", "a cat", ...])
2. Encode test image:
image_features = encode_image(image)
3. Compute similarity:
similarity = image_features @ text_features.T
4. Predict:
prediction = argmax(similarity)
CIFAR-10 zero-shot accuracy: ~88-90%
(No CIFAR-10 training data used!)
Category: CLIP
Code: import clip
model, preprocess = clip.load("ViT-B/32")
# Class names
classes = ["airplane", "car", "bird", ...]
text_inputs = clip.tokenize(classes)
# Encode
text_features = model.encode_text(text_inputs)
image_features = model.encode_image(image)
# Cosine similarity
logits = 100.0 * image_features @ text_features.T
prediction = logits.argmax()

Question: What is PyTorch Lightning and why use it?
Answer: PyTorch Lightning is a high-level wrapper that organizes PyTorch code for cleaner, more maintainable training.
Benefits:
- Clean separation: Model code vs training logic
- Automatic: GPU handling, logging, checkpointing
- Reproducible: Consistent random seeds
- Less boilerplate: No manual train/eval loops
Key components:
- LightningModule: Define model + training logic
- Trainer: Handles training loop automatically
- Callbacks: ModelCheckpoint, LearningRateMonitor
Category: Implementation
Code: class ViT(pl.LightningModule):
def training_step(self, batch, batch_idx):
loss = self.criterion(self(batch[0]), batch[1])
self.log("train_loss", loss)
return loss
def configure_optimizers(self):
return optim.AdamW(self.parameters(), lr=self.lr)
trainer = pl.Trainer(max_epochs=30)
trainer.fit(model, train_loader, val_loader)

Question: What data augmentation is used for CIFAR-10 ViT training?
Answer: Standard augmentation for CIFAR-10:
Training transforms:
- RandomCrop(32, padding=4): Crop with padding for translation invariance
- RandomHorizontalFlip(): 50% chance flip (doubles effective dataset)
- ToTensor(): Convert to [0,1] range
- Normalize(mean, std): Zero mean, unit variance
CIFAR-10 normalization:
- Mean: (0.4914, 0.4822, 0.4465)
- Std: (0.2023, 0.1994, 0.2010)
Test transforms: Only ToTensor + Normalize (no augmentation)
Category: Implementation
Code: train_transform = transforms.Compose([
transforms.RandomCrop(32, padding=4),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
transforms.Normalize(
(0.4914, 0.4822, 0.4465),
(0.2023, 0.1994, 0.2010)
),
])

Question: What is mixed precision training and why use it?
Answer: Mixed precision trains with both FP16 (half precision) and FP32 (full precision) to speed up training.
How it works:
- Forward/backward: Use FP16 (faster, less memory)
- Weight updates: Use FP32 (numerical stability)
- Gradient scaling: Scale up gradients to prevent underflow
Benefits:
- 2-3× faster: FP16 operations are faster on modern GPUs
- Lower memory: Can use larger batch sizes
- Minimal accuracy loss: Careful scaling preserves precision
Category: Implementation
Code: # Enable mixed precision
scaler = torch.cuda.amp.GradScaler(enabled=True)
with torch.cuda.amp.autocast(enabled=True):
outputs = model(inputs)
loss = criterion(outputs, targets)
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()

Question: What were the key performance results from Assignment 4?
Answer: Assignment 4 Performance Summary:
CIFAR-10 Classification:
- ViT from scratch: ~75-80% test accuracy
- CNN (Assignment 3): ~85-90% test accuracy
- Pre-trained ViT (fine-tuned): ~90-95% test accuracy
- CLIP zero-shot: ~88-90% test accuracy
LoRA Image Captioning:
- Trainable parameters: <0.5% of total model
- Task: Generate captions for CIFAR-10 images
- Result: Successfully adapted model with minimal parameters
Category: Results

Question: What are the key takeaways about Vision Transformers?
Answer: Vision Transformer key insights:
Strengths:
- Scalability: Performance improves with more data and compute
- Transfer learning: Excellent when pre-trained on large datasets
- Global receptive field: Captures long-range dependencies from layer 1
- Interpretability: Attention maps show what model focuses on
Weaknesses:
- Data-hungry: Needs large datasets (>100k images) to excel
- Computational cost: O(n²) attention complexity
- Less inductive bias: Doesn't assume spatial locality like CNNs
Best use case: Pre-train on large dataset, fine-tune on target task
Category: Summary