A student ran out of disk space on a free-tier GPU notebook because every epoch saved a full checkpoint. Most of those checkpoints were never going to be used.
A simple keep policy
- Keep the checkpoint with the best validation score, not the last one — training loss can keep dropping after validation performance peaks.
- Keep one early checkpoint as a sanity baseline to diff behaviour against.
- Delete the rest once you've confirmed the best one still evaluates well after quantization, if you quantize.
For LoRA specifically
Save only the adapter weights, not a full merged model, at every intermediate step — adapters are a few hundred megabytes versus several gigabytes for a merged model, and you can always merge the best one at the end.
See reading a loss curve during fine-tuning for how to spot the actual best checkpoint before you delete the others.
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.
