Inference
Cross-Entropy Loss
The standard loss function for classification and language model training, measuring how much probability the model gave to the correct answer.
In depth
The standard loss function for classification and language model training, measuring how much probability the model gave to the correct answer. For an LLM it is the negative log probability of the actual next token, averaged over every position in the training data. In PyTorch, nn.CrossEntropyLoss takes raw unnormalized logits and applies the softmax internally. Perplexity is the exponential of the average cross-entropy, which is why lower loss and lower perplexity mean the same thing.
Primary source: docs.pytorch.org
Example
In practice, developers reach for Cross-Entropy Loss when they need the capability described above as part of an AI feature or workflow.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Inference.
FAQ
Common questions
What is Cross-Entropy Loss?
The standard loss function for classification and language model training, measuring how much probability the model gave to the correct answer.
Why does Cross-Entropy Loss matter for AI developers?
Cross-Entropy Loss sits in the Inference part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about Cross-Entropy Loss?
Developers Digest publishes tutorials and videos that cover Inference topics including Cross-Entropy Loss. Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Inference
Logits
The raw, unnormalized scores a language model's output layer assigns to every token in its vocabulary at each step, before softmax turns them into probabilities.
Inference
Next-Token Prediction
The training objective behind GPT-style language models: given a sequence of tokens, predict a probability distribution over the token that comes next.
Inference
Continued Pretraining
Running more next-token-prediction training on an already pretrained model using a large unlabeled corpus from a target domain, such as code, math, medicine, or a new language.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever