api: "YAML JSON TOON Database"
version: 1.0.0
format: toon
dataset:
  id: 396
  slug: deep-learning-optimizers
  title: "Deep Learning Optimizers"
  description: "Optimization algorithms for training neural networks: SGD, Momentum, NAG, Adam, AdamW, RMSprop, AdaGrad, and learning rate scheduling strategies."
  category: "AI & ML Terms"
  category_slug: ai-ml-terms
  tags: "deep-learning,optimizers,sgd,adam,adamw,rmsprop,learning-rate,momentum"
  view_count: 5
  created_at: 1781275786
  updated_at: 1781275786
data:
  optimizers
    [8]{name,year,update,pros,cons,best_for,notes}:
      "SGD (Stochastic Gradient Descent)","1951 (Robbins-Monro)","θ = θ - α * ∇L(θ)",["Simple, well-understood","Good generalization","No additional hyperparameters"],["Slow convergence on ill-conditioned problems","Requires careful learning rate tuning","No per-parameter adaptation"],"When generalization matters more than speed; convex or nearly convex problems","Still the gold standard for many vision tasks; often generalizes better than adaptive methods"
      "SGD + Momentum","1964 (Polyak)","v = γv + α∇L(θ); θ = θ - v",["Accelerates convergence in relevant direction","Reduces oscillation","Damps oscillations across ravines"],["Adds one hyperparameter (\u03b3, typically 0.9)","May overshoot minima"],"Most deep learning tasks; default choice for CNNs","Momentum term accumulates gradient in consistent directions; γ=0.9 is standard"
      "NAG (Nesterov Accelerated Gradient)",1983,"v = γv + α∇L(θ - γv); θ = θ - v",["Lookahead correction prevents overshooting","Better convergence rate than momentum","Theoretically optimal for convex functions"],["Slightly more complex","Marginal practical improvement over momentum"],"RNNs, problems with high curvature","Computes gradient at approximate future position rather than current position"
      "AdaGrad (Adaptive Gradient)","2011 (Duchi et al.)","G = G + ∇L²; θ = θ - α/(√G + ε) * ∇L",["Per-parameter learning rates","No manual LR tuning per parameter","Excellent for sparse features"],["Monotonically decreasing LR (can become too small)","Not suitable for non-convex deep learning"],"Sparse features (NLP, recommender systems), convex optimization","Accumulates squared gradients; LR shrinks to zero eventually — problematic for deep nets"
      "RMSprop (Root Mean Square Propagation)","2012 (Hinton)","E[g²] = γE[g²] + (1-γ)∇L²; θ = θ - α/(√E[g²] + ε) * ∇L",["Fixes AdaGrad's vanishing LR problem","Per-parameter adaptation","Works well for RNNs"],["Adds \u03b3 hyperparameter (typically 0.9)","Can still get stuck in local minima"],"RNNs, non-stationary objectives, online learning","Uses exponential moving average of squared gradients instead of cumulative sum"
      "Adam (Adaptive Moment Estimation)","2014 (Kingma & Ba)","m = β₁m + (1-β₁)∇L; v = β₂v + (1-β₂)∇L²; θ = θ - α * m̂/(√v̂ + ε)",["Combines momentum + RMSprop","Bias correction for early steps","Minimal hyperparameter tuning needed","Works well out of the box"],["May generalize worse than SGD","Memory overhead (2x gradients for m and v)","Can converge to sharp minima"],"Default optimizer for most deep learning; fast prototyping; NLP, vision, RL","Default: β₁=0.9, β₂=0.999, ε=1e-8, α=0.001. Most widely used optimizer."
      "AdamW (Adam with Decoupled Weight Decay)","2017 (Loshchilov & Hutter)","θ = θ - α * (m̂/(√v̂ + ε) + λθ)",["Proper weight decay implementation (decoupled from adaptive LR)","Better generalization than Adam","Standard for Transformer training"],["One more hyperparameter (weight decay \u03bb)","Still adaptive method generalization gap vs SGD"],"Transformers (BERT, GPT), vision transformers, any architecture with weight decay","Weight decay applied outside adaptive LR computation; default for most modern architectures"
      "LAMB / LARS",2019,"Trust ratio × adaptive step (layer-wise adaptive scaling)",["Enables very large batch training (32K+)","Layer-wise LR adaptation","Stable at extreme batch sizes"],["More complex","Less tested across architectures","Overkill for small batches"],"Large-batch distributed training (TPU pods, multi-node GPU), BERT pretraining at scale","LARS (Layer-wise Adaptive Rate Scaling) and LAMB (Large Batch Adam) enable batch sizes that would diverge with standard Adam"