Deep Learning Optimizers

Optimization algorithms for training neural networks: SGD, Momentum, NAG, Adam, AdamW, RMSprop, AdaGrad, and learning rate scheduling strategies.

AI & ML Terms
8 entries
1 sections
CC0 — public domain

The data

Optimizers

Name	Year	Update	Pros	Cons	Best for	Notes
SGD (Stochastic Gradient Descent)	1951 (Robbins-Monro)	θ = θ - α * ∇L(θ)	Simple, well-understood Good generalization No additional hyperparameters	Slow convergence on ill-conditioned problems Requires careful learning rate tuning No per-parameter adaptation	When generalization matters more than speed; convex or nearly convex problems	Still the gold standard for many vision tasks; often generalizes better than adaptive methods
SGD + Momentum	1964 (Polyak)	v = γv + α∇L(θ); θ = θ - v	Accelerates convergence in relevant direction Reduces oscillation Damps oscillations across ravines	Adds one hyperparameter (γ, typically 0.9) May overshoot minima	Most deep learning tasks; default choice for CNNs	Momentum term accumulates gradient in consistent directions; γ=0.9 is standard
NAG (Nesterov Accelerated Gradient)	1983	v = γv + α∇L(θ - γv); θ = θ - v	Lookahead correction prevents overshooting Better convergence rate than momentum Theoretically optimal for convex functions	Slightly more complex Marginal practical improvement over momentum	RNNs, problems with high curvature	Computes gradient at approximate future position rather than current position
AdaGrad (Adaptive Gradient)	2011 (Duchi et al.)	G = G + ∇L²; θ = θ - α/(√G + ε) * ∇L	Per-parameter learning rates No manual LR tuning per parameter Excellent for sparse features	Monotonically decreasing LR (can become too small) Not suitable for non-convex deep learning	Sparse features (NLP, recommender systems), convex optimization	Accumulates squared gradients; LR shrinks to zero eventually — problematic for deep nets
RMSprop (Root Mean Square Propagation)	2012 (Hinton)	E[g²] = γE[g²] + (1-γ)∇L²; θ = θ - α/(√E[g²] + ε) * ∇L	Fixes AdaGrad's vanishing LR problem Per-parameter adaptation Works well for RNNs	Adds γ hyperparameter (typically 0.9) Can still get stuck in local minima	RNNs, non-stationary objectives, online learning	Uses exponential moving average of squared gradients instead of cumulative sum
Adam (Adaptive Moment Estimation)	2014 (Kingma & Ba)	m = β₁m + (1-β₁)∇L; v = β₂v + (1-β₂)∇L²; θ = θ - α * m̂/(√v̂ + ε)	Combines momentum + RMSprop Bias correction for early steps Minimal hyperparameter tuning needed Works well out of the box	May generalize worse than SGD Memory overhead (2x gradients for m and v) Can converge to sharp minima	Default optimizer for most deep learning; fast prototyping; NLP, vision, RL	Default: β₁=0.9, β₂=0.999, ε=1e-8, α=0.001. Most widely used optimizer.
AdamW (Adam with Decoupled Weight Decay)	2017 (Loshchilov & Hutter)	θ = θ - α * (m̂/(√v̂ + ε) + λθ)	Proper weight decay implementation (decoupled from adaptive LR) Better generalization than Adam Standard for Transformer training	One more hyperparameter (weight decay λ) Still adaptive method generalization gap vs SGD	Transformers (BERT, GPT), vision transformers, any architecture with weight decay	Weight decay applied outside adaptive LR computation; default for most modern architectures
LAMB / LARS	2019	Trust ratio × adaptive step (layer-wise adaptive scaling)	Enables very large batch training (32K+) Layer-wise LR adaptation Stable at extreme batch sizes	More complex Less tested across architectures Overkill for small batches	Large-batch distributed training (TPU pods, multi-node GPU), BERT pretraining at scale	LARS (Layer-wise Adaptive Rate Scaling) and LAMB (Large Batch Adam) enable batch sizes that would diverge with standard Adam

Fetch the same bytes

The static files are identical to what the API returns, but with no rate limit and no server round trip. Use the API when you want a query and a content type; use the files when you want to cache one document.

curl "https://yjtoon.com/api/dataset/deep-learning-optimizers?format=toon"

const res = await fetch(
  "https://yjtoon.com/static-data/dataset/deep-learning-optimizers.toon"
);
const toon = await res.text();

Rate limit: 120 requests per minute per IP, no key and no signup. API reference →

Deep Learning Optimizers

The data

Optimizers

Fetch the same bytes

Topics