A project accumulating new labelled data every month faces a choice: retrain from the original base model each time, or keep fine-tuning the already-tuned model incrementally. The second is cheaper but carries a real risk.
The risk: compounding forgetting
Each additional round of fine-tuning on new data can erode capability learned in earlier rounds, especially if the new data's distribution differs even slightly from the old. This compounds silently across many rounds if never re-checked.
A safer approach
- Keep a fixed evaluation set spanning ALL prior rounds' concerns, not just the newest data, and check it after every continual fine-tune.
- Periodically retrain from the original base model on the full accumulated dataset, rather than purely incremental updates indefinitely.
- Version every checkpoint clearly so a regression can be traced to the specific round that caused it.
See catastrophic forgetting in fine-tuned models.
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.
