Gradient Descent

In depth

An optimization algorithm that trains a model by repeatedly nudging its weights in the direction that most reduces the loss, as given by the negative gradient. The size of each step is set by the learning rate. In practice models use stochastic or mini-batch gradient descent, which estimates the gradient from a small batch of examples instead of the full dataset, and optimizers like Adam and AdamW add momentum and per-parameter step sizes on top.

Primary source: arxiv.org