Training loss explained: minimising it, loss functions, validation curves, overfitting

Loss measures how wrong a model's predictions are, and training is the process of reducing it. But low training loss alone is not the goal; the model must generalise to new data, which validation loss reveals. FineTune Studio streams live loss charts over WebSockets during QLoRA fine-tuning and compares base and tuned models, so this page covers minimising loss, loss functions, training versus validation loss and curves, flat losses, Keras stopping, overfitting, soft labels and distillation temperature.

How do you minimise loss when training a model?

Use a suitable optimiser such as AdamW, tune the learning rate with warmup and a decay schedule, normalise inputs, choose a sensible batch size, and make sure data and labels are clean. Monitor validation loss, not just training loss.

Also asked as: how to minimize loss when training a model

If loss falls slowly, try a higher learning rate or check that gradients flow through all layers. If loss is unstable or explodes, lower the learning rate or use gradient clipping. Better data often reduces loss more than architecture changes. For fine-tuning, LoRA ranks and learning rates matter most. Remember that minimising training loss to near zero usually means memorising; the goal is the lowest validation loss on unseen data.

What functions can you use for training loss?

Common loss functions include mean squared error and Huber loss for regression, cross-entropy for classification, focal loss for imbalanced classes, contrastive and InfoNCE losses for embeddings, CTC loss for speech and handwriting, and next-token cross-entropy for language models.

Choose the loss that matches your task and output: probabilities need cross-entropy, ranking tasks use pairwise or listwise losses, and face recognition uses margin losses such as ArcFace. KL divergence compares probability distributions, useful in distillation. Custom losses can encode business priorities, such as penalising false negatives more. Frameworks such as PyTorch and Keras provide most standard losses. Plot loss alongside task metrics, since they can disagree, and test custom losses carefully.

What is the difference between training loss and validation loss?

Training loss is measured on the data the model learns from; validation loss is measured on held-out data it never trains on. Validation loss estimates how well the model generalises to new examples.

Also asked as: difference between training and validation loss

Training loss usually keeps falling as the model fits the data, while validation loss falls at first, then may rise once the model starts memorising. The gap between them indicates overfitting. Use validation loss to choose checkpoints, tune hyperparameters and decide when to stop. Keep a separate test set, untouched until the end, for a final honest evaluation, since repeated tuning on validation data can make it optimistic. Report both numbers when sharing results.

How do you read a training vs validation loss curve?

If both curves fall and stay close, training is healthy. If training loss keeps falling while validation loss rises, the model is overfitting. If both stay high, it is underfitting. Large spikes suggest learning rate or data problems.

Validation loss below training loss can occur when dropout or augmentation apply only during training, or when validation data is easier, so check for leakage too. Noisy curves often come from small batches or high learning rates; smoothing helps reading trends. FineTune Studio streams these curves live during fine-tuning, so problems can be caught early rather than after hours of wasted training. Save checkpoints near the validation minimum.

Why are training and validation loss not changing?

Flat losses usually mean the model is not learning: the learning rate is too low or extremely high, layers are frozen, gradients are zero because of a bug, labels are wrong or constant, or the optimiser step is missing in the training loop.

Also asked as: why my training and validation loss is not changing

Check basics: confirm the backward pass and optimiser step run each batch, gradients are non-zero, trainable parameters exist, and inputs are normalised and shuffled. Try overfitting a tiny batch, which a working setup should achieve quickly; if it cannot, there is a bug. Verify the loss function matches the output, such as cross-entropy with raw logits. Dead activations and poor initialisation can also stall learning. Logging gradient norms makes these problems visible.

How do you tell Keras to stop training based on loss?

Use the EarlyStopping callback, monitoring validation loss with a patience value and restore_best_weights set to true, so training stops when validation loss stops improving and keeps the best weights.

Also asked as: how to tell keras stop training based on loss value

To stop at a specific loss threshold, write a custom callback whose on_epoch_end method checks the logged loss and sets the model's stop_training attribute to true when the threshold is reached. Combine early stopping with ModelCheckpoint to save the best model to disk. Choose patience carefully: too small stops before noisy curves recover, while too large wastes time and risks overfitting. Log metrics to TensorBoard for easy comparison across runs.

What is overfitting in machine learning?

Overfitting happens when a model learns the training data too closely, including noise and quirks, so it performs well on training data but poorly on new data. A widening gap between training and validation performance reveals it.

Causes include too little data, overly complex models, too many training epochs and data leakage. In LLM fine-tuning, overfitting shows up as the model repeating training phrases, losing general abilities or answering only in the training format. Detect it by monitoring validation loss and testing on realistic held-out examples. FineTune Studio's base-versus-tuned comparison helps reveal when a fine-tuned model has lost useful general behaviour. Simple models trained on enough data often generalise surprisingly well.

How do you overcome overfitting in ML?

Add more or more varied data, use data augmentation, apply regularisation such as dropout and weight decay, stop training early based on validation loss, simplify the model, and use cross-validation to tune choices reliably.

Also asked as: what is overfitting in ml · how to overcome overfitting in ml

For fine-tuning language models, use fewer epochs, lower learning rates, smaller LoRA ranks and more diverse examples, and mix in general data to preserve broad skills. Remove duplicates and near-duplicates between training and validation sets, which hide overfitting. Ensembles and pretrained backbones also improve generalisation. Track experiments so you know which change helped. Always confirm improvements on a final test set, since repeated tuning against validation data can itself lead to overfitting the validation set.

What loss function should you use when labels are probabilities?

Use cross-entropy with soft targets, which accepts probability distributions as labels, or KL divergence between the predicted and target distributions. For independent per-label probabilities in multi-label tasks, use binary cross-entropy on each label.

Also asked as: what loss function to use when labels are probabilities

Soft labels appear in knowledge distillation, label smoothing, crowd-sourced annotations with disagreement, and mixup augmentation. Make sure predictions pass through softmax or sigmoid consistently with the loss, and avoid applying them twice. Check that target rows sum to one. PyTorch's cross-entropy loss supports probability targets directly, and Keras's categorical cross-entropy accepts them as well. Mean squared error on probabilities works but usually trains less effectively than cross-entropy for classification.

Why is temperature necessary in distillation loss?

Temperature softens the teacher's output probabilities, revealing how similar it considers different classes, information that hard labels and sharp probabilities hide. The student learns from these softer distributions, often generalising better than from labels alone.

Also asked as: why is temperature necessary in the loss function for model distillation

In Hinton and colleagues' 2015 formulation, both teacher and student logits are divided by a temperature greater than one before softmax, and the distillation loss compares them with KL divergence. Because softening shrinks gradients, the loss is multiplied by the temperature squared to keep its scale. Usually, distillation loss is combined with standard cross-entropy on true labels. Tune temperature on validation data, since values that are too high flatten useful distinctions.

Pranjul Rathour

GenAI Engineer · India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.

PortfolioInvite me to your campusHire meLinkedInGitHubXInstagramThreadsFacebookBlueskyDev.toBloggerHashnodeSCULTEmail