Regularization Techniques in Machine Learning

Techniques to prevent overfitting: L1/L2 regularization, dropout, batch normalization, early stopping, data augmentation, label smoothing, and mixup.

AI & ML Terms
8 entries
CC0 — public domain

JSON7,916 B
YAML5,728 B · −28%
TOON5,174 B · −35%

Regularization techniques that reduce overfitting, with formula, effect, uses and notes: L1 and L2 penalties, dropout, batch normalization, early stopping, data augmentation, label smoothing and mixup or CutMix.

Use it when

Your model fits training data well but generalizes poorly.
You are choosing regularization for a neural network or linear model.

Watch out for

Regularization has hyperparameters that need tuning; too much causes underfitting.
Batch normalization's regularizing effect is a side effect, not its main purpose.

The data

Techniques

Name	Formula	Effect	Use cases	Notes	Mechanism
L1 Regularization (Lasso)	Loss + λ * Σ\|w_i\|	Drives weights to exactly zero (sparse solutions)	Feature selection Sparse models Compressed sensing When many irrelevant features	L1 penalty creates sparse weight vectors; useful for automatic feature selection	null
L2 Regularization (Ridge)	Loss + λ * Σw_i²	Shrinks weights toward zero but not exactly zero (smooth solutions)	General-purpose regularization Multicollinearity mitigation Default regularization for most models	Most common form; equivalent to Gaussian prior on weights; differentiable everywhere	null
Dropout	null	Prevents co-adaptation of neurons; ensemble of thinned networks	Fully connected layers in deep networks When overfitting is observed Standard in most deep learning architectures	Typical rates: 0.2-0.5 for FC layers, 0.1-0.2 for conv layers; scale activations by 1/(1-p) at test time (or use inverted dropout)	Randomly set fraction p of activations to zero during training
Batch Normalization	null	Reduces internal covariate shift; allows higher learning rates; slight regularization	Deep CNNs Very deep networks When training is unstable Enabling higher learning rates	Maintains running mean/variance for inference; can hurt performance with small batches or RNNs	Normalize activations per mini-batch: x̂ = (x - μ_batch) / √(σ²_batch + ε); y = γx̂ + β
Early Stopping	null	Prevents overfitting by stopping before model memorizes training data	Any iterative training algorithm When validation set available Neural networks, gradient boosting	Save best model weights during training; restore after stopping; patience=5-20 typical	Monitor validation loss; stop training when val_loss stops improving for patience epochs
Data Augmentation	null	Increases effective training set size; teaches invariances	Computer vision (standard practice) NLP (synonym replacement, back-translation) Audio (pitch shift, time stretch) When data is limited	Most effective regularization for vision; AutoAugment and RandAugment automate policy search	Apply label-preserving transformations to training data (rotation, flip, crop, color jitter for images)
Label Smoothing	null	Prevents overconfidence; improves calibration; reduces gap between train and test accuracy	Image classification (ResNet, EfficientNet) Machine translation When model is overconfident Knowledge distillation teacher	Typical ε = 0.1; prevents model from assigning probability 1 to any class; improves generalization	Replace hard labels (0/1) with soft targets: y_smooth = (1 - ε) * y_hard + ε / K
Mixup / CutMix	null	Linearizes training between examples; reduces memorization	Image classification When stronger regularization needed State-of-the-art vision models	Mixup: (x', y') = (λx_i + (1-λ)x_j, λy_i + (1-λ)y_j); CutMix often performs better by preserving local structure	Mixup: interpolate pairs of images and labels. CutMix: cut and paste patches

Fields

The shape of this dataset, read from the data itself: every column, its type and a real value. Agents can use this to plan a query without fetching the whole document.

Section	Field	Type	Example
techniques	`name`	string	L1 Regularization (Lasso)
techniques	`formula`	string	Loss + λ * Σ\|w_i\|
techniques	`effect`	string	Drives weights to exactly zero (sparse solution…
techniques	`use_cases`	array	[4 items]
techniques	`notes`	string	L1 penalty creates sparse weight vectors; usefu…
techniques	`mechanism`	string	Randomly set fraction p of activations to zero …

Fetch the same bytes

The static files contain the same reference rows as the API, with no API rate limit. Live metadata, such as view counts, may differ. Use the API for content negotiation; use the static files for a cacheable snapshot.

curl "https://yjtoon.com/api/dataset/regularization-techniques?format=toon"

const res = await fetch(
  "https://yjtoon.com/static-data/dataset/regularization-techniques.toon"
);
const toon = await res.text();

Rate limit: 120 requests per minute per IP, no key and no signup. API reference →