Transfer Learning and Fine-Tuning

Transfer learning strategies: feature extraction, fine-tuning, domain adaptation, multi-task learning, prompt tuning, LoRA, and pretraining paradigms.

AI & ML Terms
8 entries
1 sections
CC0 — public domain

The data

Strategies

Name	Approach	When to use	Pros	Cons	Example	Notes
Feature Extraction (Frozen Backbone)	Use pretrained model as fixed feature extractor; train only new classification head	Small target dataset Target domain similar to source Limited compute resources	Fast training No catastrophic forgetting Minimal overfitting risk	Limited adaptation to target domain Features may not be optimal for new task	ResNet-50 pretrained on ImageNet → freeze all layers → train only final FC layer on custom dataset	null
Fine-Tuning (Full Model)	Unfreeze all layers; continue training entire model on target data with lower learning rate	Large target dataset Target domain differs from source Maximum performance needed	Full adaptation to target domain Best possible performance	Risk of catastrophic forgetting Requires more data Computationally expensive	BERT pretrained → fine-tune all 110M parameters on SQuAD QA dataset with LR=2e-5	null
Layer-wise Learning Rate Decay	Assign lower LR to earlier layers, higher LR to later layers during fine-tuning	Medium-sized target dataset Want to preserve generic low-level features	Balances preservation and adaptation Reduces forgetting of early features	Adds hyperparameter (decay rate) Requires tuning	LR(layer_i) = base_lr × decay^(depth - i); typical decay=0.9-0.95	null
Progressive Unfreezing	Start with frozen backbone + trainable head; gradually unfreeze layers from top to bottom	Small target dataset Want controlled adaptation Avoid catastrophic forgetting	Gradual adaptation reduces shock to pretrained weights Better stability	Slower training process More complex training loop	Phase 1: train head only. Phase 2: unfreeze last block + head. Phase 3: unfreeze all.	null
Domain Adaptation	Align feature distributions between source and target domains (e.g., adversarial training, MMD loss)	Source and target domains differ significantly Labeled target data scarce Domain shift is known issue	Explicitly addresses domain gap Works with unlabeled target data	More complex training Requires domain-specific techniques	DANN (Domain-Adversarial Neural Network): gradient reversal layer makes features domain-invariant	null
LoRA (Low-Rank Adaptation)	Inject trainable low-rank decomposition matrices into attention/FFN layers; freeze pretrained weights	Fine-tuning large models (LLMs, ViT) Memory-constrained environments Multiple downstream tasks from same base model	Dramatically fewer trainable parameters (0.1-1% of original) No inference latency overhead (merged into weights) Multiple LoRA adapters can share same base model	Slight accuracy drop vs full fine-tuning Rank hyperparameter to tune	LLaMA 7B: full fine-tuning = 7B params; LoRA (r=8) = 4.7M params (0.07%)	Used for most LLM fine-tuning (Alpaca, Vicuna, etc.)
Prompt Tuning / Prefix Tuning	Learn soft prompts (continuous vectors) prepended to input; freeze entire model	Very large models where even LoRA is expensive Multiple tasks from single model When prompt engineering is natural fit	Minimal trainable parameters (thousands vs billions) No model modification needed Easy to switch between tasks	Lower performance than full fine-tuning Only works well for very large models (10B+)	T5-11B: learn 100-token soft prompt (128-dim each) = 1.28M params vs 11B full model	null
Multi-Task Learning	Train single model on multiple related tasks simultaneously with shared backbone	Multiple related tasks available Data-scarce individual tasks Want shared representations	Improved generalization via inductive bias Single model for multiple tasks Data augmentation across tasks	Task interference possible Complex loss weighting Harder to debug	BERT pretraining: MLM + NSP simultaneously. MT-DNN: shared BERT + task-specific heads.	null

Fetch the same bytes

The static files are identical to what the API returns, but with no rate limit and no server round trip. Use the API when you want a query and a content type; use the files when you want to cache one document.

curl "https://yjtoon.com/api/dataset/transfer-learning?format=toon"

const res = await fetch(
  "https://yjtoon.com/static-data/dataset/transfer-learning.toon"
);
const toon = await res.text();

Rate limit: 120 requests per minute per IP, no key and no signup. API reference →

Transfer Learning and Fine-Tuning

The data

Strategies

Fetch the same bytes

Topics