A fine-tuned model that scored well in evaluation is useless if it can't run within your deployment budget. Quantization shrinks it — the question is which format.
The three you'll actually encounter
- GGUF: for CPU and llama.cpp-style local inference; simplest to deploy, widest tool support, slightly lower throughput on GPU.
- AWQ: activation-aware quantization, strong quality retention at 4-bit, good GPU inference throughput.
- GPTQ: an earlier and still widely-supported 4-bit method; slightly behind AWQ on quality at the same bit-width in most comparisons.
How to choose
Deploying to a laptop or CPU-only server: GGUF. Deploying to a GPU and chasing throughput: AWQ first, GPTQ if your serving stack doesn't yet support AWQ. Always re-run your evaluation set after quantizing — the quality drop is usually small but not always negligible.
See merging LoRA adapters and serving for the step that usually comes right before this one.
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.
