The most common fine-tuning mistake I see isn't a bad hyperparameter — it's skipping evaluation entirely and assuming the loss curve going down means the model got better at the task you actually care about.
Run the same prompts through both models
Before touching the fine-tuned model, save a fixed set of representative prompts and the base model's answers to them. After training, run the identical prompts through the tuned model. Side by side, not from memory — memory of "how the base model used to answer" is unreliable after a training run.
Score what actually matters for your task
A general benchmark score tells you very little about whether the model got better at your specific narrow task. Write five to ten task-specific questions with known correct answers, and check both models against those.
What a real regression looks like
Sometimes the tuned model is worse on general knowledge and better on your task — that's expected and fine. Sometimes it's worse on both, which means something in the dataset or hyperparameters needs fixing before you ship it. You only find out which one happened if you actually run the comparison.
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.
