Hyperparameters

In depth

Settings chosen by the developer that control how a model learns, as opposed to the weights the model learns itself. Common examples are learning rate, batch size, number of epochs, weight decay, warmup steps, and, for LoRA, the rank and alpha. Small changes can decide whether a run converges, so teams usually sweep them on small runs before committing compute to a full training job.

Primary source: arxiv.org