Model Training

Stochastic Gradient Descent

Quick Answer: An optimization algorithm that updates model weights using the gradient computed from a random subset (mini-batch) of training data rather than the entire dataset.

Stochastic Gradient Descent is an optimization algorithm that updates model weights using the gradient computed from a random subset (mini-batch) of training data rather than the entire dataset. The randomness makes training faster and can actually help the model find better solutions by escaping local minima.

Example

With 1 million training images, standard gradient descent would need to process all 1 million to make a single weight update. SGD with a batch size of 32 makes an update after every 32 images. The updates are noisier but 31,250 times more frequent, leading to much faster progress.

Why It Matters

SGD and its variants are how all neural networks learn. Every model from a simple classifier to GPT-4 was trained using some form of stochastic optimization. Understanding SGD helps you reason about training dynamics, hyperparameter choices, and why certain training configurations work better than others.

How It Works

The gradient descent spectrum runs from full-batch (compute gradients on all data, precise but slow), through mini-batch SGD (compute on a random subset, the practical default), to pure SGD (compute on a single example, very noisy). Mini-batch SGD with batch sizes of 32-512 is the standard in practice.

The noise in SGD is actually beneficial. It acts as implicit regularization, helping the model escape sharp minima (which tend to overfit) and find flat minima (which generalize better). This explains the counterintuitive finding that noisier training sometimes produces better models.

Modern optimizers build on SGD with additional features. Momentum accumulates gradient direction over time, smoothing out oscillations. Adam (Adaptive Moment Estimation) maintains per-parameter learning rates based on first and second moment estimates of gradients. AdamW adds proper weight decay. Learning rate schedulers (cosine annealing, warmup, reduce-on-plateau) adjust the learning rate during training.

Batch size interacts with learning rate: larger batches allow larger learning rates (the linear scaling rule). Gradient accumulation simulates large batches on limited GPU memory by accumulating gradients over multiple forward passes before updating.

For LLM training, distributed SGD across many GPUs introduces additional considerations: gradient synchronization, communication overhead, and the relationship between global batch size and convergence. Techniques like gradient compression and local SGD reduce communication costs.

Common Mistakes

Common mistake: Using a fixed learning rate throughout training instead of a scheduler

Use warmup + cosine decay or similar scheduling. Starting with a small learning rate and gradually increasing (warmup) followed by gradual decrease produces better convergence.

Common mistake: Choosing batch size without considering its interaction with learning rate and convergence

Follow the linear scaling rule: when increasing batch size by factor N, increase learning rate by factor N. Monitor both training loss and validation metrics when changing batch sizes.

Career Relevance

SGD and its variants are core knowledge for ML engineering roles. Understanding optimization is essential for training models, fine-tuning pre-trained models, and debugging training failures. It's a frequent interview topic and daily practical concern for anyone training neural networks.

Related Terms

Stay Ahead in AI

Join 1,300+ prompt engineers getting weekly insights on tools, techniques, and career opportunities.

Join the Community →