More fine-tuning runs are ruined by formatting than by hyperparameters. The model trains happily on whatever string you give it — including the wrong one — and you only discover the problem when inference produces role tags in the output. Here is the map I wish I had when I started building FineTune Studio.
The formats you will meet
- Alpaca — instruction, optional input, output. Simple, single-turn, and still common for task datasets.
- ShareGPT — a conversations list with alternating human and assistant turns. Natural for chat data and multi-turn behaviour.
- Messages (OpenAI-style) — a list of role/content objects, now the closest thing to a standard and what most trainers accept directly.
What a chat template does
Models are trained with specific tokens marking who is speaking. The chat template is the recipe that turns a messages list into that exact string — system tags, turn separators, end-of-turn tokens. Use the template that ships with the base model's tokenizer. Writing your own means training the model to expect a format it will never see at inference, or vice versa.
Mistakes that look like bad hyperparameters
- Training on the whole sequence including the user turn, so the model learns to imitate questions as well as answers. Mask the prompt tokens and compute loss on assistant tokens only.
- A missing end-of-turn token, so the fine-tuned model never learns to stop and rambles at inference.
- Mixed formats in one dataset — half Alpaca, half chat — producing a model that is inconsistent about both.
- Whitespace and newline differences between training and inference templates. Tokenizers are unforgiving.
Validate before you train
FineTune Studio validates the dataset on upload and reports the actual problem — malformed JSON on line 412, an empty assistant turn, a role it does not recognise — instead of a stack trace three steps into training. Do the same in a script: parse every example, render it through the template, decode it back and read ten of them with your own eyes. The checklist is in validate your dataset before you burn a GPU hour.
Get the format right and half of fine-tuning's mystery disappears. What remains is data quality, which is a much better problem to have.
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.
