Training vs validation loss: oscillation, overfitting, schedulers and eval_loss errors

Loss curves are the main window into training: training loss shows how well a model fits its training data, and validation loss shows how well it generalises to data it has not seen. Their shapes reveal learning-rate problems, overfitting and bugs. FineTune Studio, which I built, streams live training loss over WebSockets during QLoRA runs for exactly this reason, so this page explains oscillation, the training and validation gap, overfitting in LLMs, stuck runs, schedulers and common logging errors.

What could an oscillating training loss curve represent?

Oscillation usually means the learning rate is too high, the batch size is too small, or the data is noisy or poorly shuffled. Some oscillation is normal with mini-batches; large, growing swings signal instability.

Diagnose step by step: plot a smoothed moving average to see the trend under the noise; lower the learning rate or add warmup; increase the effective batch size with gradient accumulation; check that data is shuffled, since batches ordered by class or length create periodic patterns; and look for bad examples that spike the loss. Gradient clipping helps with occasional explosions. If loss oscillates and rises, stop and fix the setup before wasting compute on a run that will not recover.

Validation loss vs training loss: what is the difference?

Training loss is computed on the data the model learns from, usually averaged during each epoch. Validation loss is computed on held-out data the model never trains on, measuring generalisation. The gap between them is your main overfitting signal.

Also asked as: val loss vs training loss

Healthy training shows both decreasing, with validation slightly above training. Validation lower than training can happen when dropout or augmentation is active only during training, or when training loss is averaged over an epoch while the model improves. A widening gap with rising validation loss means overfitting. Both high and flat means underfitting: the model is too small, the learning rate is wrong, or the features carry little signal. Always choose checkpoints by validation, not training, loss.

How do you read a training vs validation loss plot?

Plot both losses against steps or epochs on the same axes, often with a log scale for loss. Look at three things: whether both decrease, the gap between them, and the point where validation loss stops improving.

Common shapes: both falling together means keep training; validation flattening while training keeps falling means you are near the best checkpoint; validation rising means overfitting, so stop or regularise; both flat and high means underfitting or a bug; sudden spikes mean instability or bad data. In code, log both to TensorBoard or Weights & Biases, or plot the trainer's log history with matplotlib. Mark the best validation checkpoint on the plot for clarity.

What should you do if training loss decreases but validation loss does not?

That is overfitting or a mismatch between training and validation data. The model is memorising training examples instead of learning patterns that transfer. Stop at the best validation checkpoint and then improve data or regularisation.

Also asked as: what to do if training loss decreases but validation loss does not decrease

Fixes, roughly in order: use early stopping; add more or more diverse training data; apply data augmentation; increase dropout or weight decay; reduce model capacity or, for LoRA, lower the rank; lower the learning rate or train fewer epochs. Also check the split itself: if validation comes from a different distribution, or labels differ in style, the gap reflects the data, not the model. Leakage checks matter too, since duplicates make training look better than it is.

How do LLMs avoid overfitting?

During pre-training, LLMs see enormous, diverse datasets, often for about one pass, so they rarely overfit in the classic sense. Overfitting becomes a real risk during fine-tuning on small datasets, where a model can memorise examples within a few epochs.

Fine-tuning defences: parameter-efficient methods such as LoRA that update few weights; small learning rates with warmup; one to three epochs; dropout on LoRA layers; held-out evaluation with early stopping; and diverse, deduplicated data. Mixing in general instruction data helps prevent forgetting of broad skills. Evaluate on fresh prompts, not just validation loss, since a model can keep a low loss while copying phrasing from training data. FineTune Studio compares base and tuned outputs for this reason.

Why is training loss not decreasing when fine-tuning Qwen with Unsloth?

Common causes are a learning rate that is too low or too high, a wrong chat template, labels masked incorrectly so the model trains on nothing useful, LoRA targeting too few modules, or a dataset that is already easy for the model.

Also asked as: why is my training loss not decreasing while fine-tuning qwen with unsloth

Check the formatted training examples: apply the Qwen chat template your model expects and print a sample to confirm roles and special tokens are correct. If you train only on responses, verify that response masking actually leaves response tokens unmasked. Try a learning rate around 2e-4 for LoRA, target all linear layers, and confirm the count of trainable parameters is non-zero. Also check whether loss is already low at step one, which means the model already knows the task.

Should a learning rate scheduler step per batch or per epoch?

For modern training, especially transformers, step the scheduler per optimisation step, so warmup and decay are smooth. Per-epoch stepping suits older schedules such as step decay every few epochs, mostly in classic vision training.

Also asked as: should a learning rate scheduler adjust the learning rate by optimization step (batch) or by epoch

Hugging Face schedulers like linear or cosine with warmup are defined in optimisation steps and called after each optimizer step. With gradient accumulation, count optimizer updates, not micro-batches, when computing total steps. PyTorch's ReduceLROnPlateau, by contrast, steps on a validation metric, typically once per epoch. Whatever you choose, log the learning rate alongside loss, since many confusing loss curves turn out to be scheduler mistakes, such as decay finishing far too early.

What is the difference between training error and training loss?

Training loss is the value of the differentiable objective the optimiser minimises, such as cross-entropy. Training error is a task metric on the same data, such as the fraction of misclassified examples. They usually move together but are not the same.

Also asked as: training error vs training loss

Loss captures confidence: a classifier can predict the right class with low confidence, giving zero error but non-zero loss, or become more confident without changing any predictions, lowering loss while error stays the same. That is why you track both: loss to guide optimisation and spot instability, and error or accuracy to judge usefulness. For language models, perplexity, the exponential of the average loss, is often reported alongside task metrics.

Do we need zero training loss after achieving zero training error?

No. Zero training error means every training example is classified correctly; pushing loss to zero after that only makes predictions more confident on training data, which often harms generalisation and calibration.

Continuing to train can still help in some regimes, since margins grow and validation may improve for a while, but watch validation metrics rather than training loss. Techniques such as label smoothing deliberately keep training loss above zero to improve calibration. A 2020 method called flooding even keeps training loss near a small floor instead of driving it to zero. In practice, stop when validation performance peaks, regardless of what training loss is doing at that point.

Why is there no training log and a KeyError for eval_loss?

In the Hugging Face Trainer, eval_loss only exists when evaluation runs. A KeyError appears when settings such as load_best_model_at_end or metric_for_best_model reference eval_loss but no evaluation dataset or evaluation strategy is configured.

Also asked as: why no log for training model, and key_error for 'eval_loss

Fix it by passing an eval_dataset, setting eval_strategy (older versions call it evaluation_strategy) to steps or epoch, and matching save_strategy to it. Missing training logs usually mean logging_steps is larger than the total number of steps, or report_to is set to none without a console callback. Also confirm the model returns a loss: labels must be present in each batch, otherwise the trainer cannot compute it, and evaluation will silently report nothing useful.

Pranjul Rathour

GenAI Engineer · India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.

PortfolioInvite me to your campusHire meLinkedInGitHubXInstagramThreadsFacebookBlueskyDev.toBloggerHashnodeSCULTEmail