Key takeaways
- GGUF packages quantized weights with architecture, tokenizer and template metadata in one file.
- Aligned tensors allow memory-mapped loading, and runtimes offload layers across CPU and GPU.
- Use GGUF for local inference, safetensors for training and FP8 for data-centre GPU serving.
- Q4_K_M is the usual default, with higher quants when memory allows.
- GGUF replaced the old GGML file format in 2023, while ggml remains the underlying tensor library.
GGUF is the single-file model format used by llama.cpp and tools built on it, such as Ollama and LM Studio. It packs a model's weights, usually quantized, together with metadata such as architecture, tokenizer and chat template, so models load quickly and run on ordinary hardware. I fine-tune models with QLoRA in FineTune Studio and run them locally, so this page explains how GGUF works, when to use it, how to choose models and quants, where files go, and how GGUF compares with GGML, safetensors and FP8.
What does GGUF mean for LLMs?
For LLMs, GGUF means a model packaged for local inference with llama.cpp-based tools. A GGUF file contains everything needed to run the model: weights, tokenizer, hyperparameters and prompt template, typically in a quantized form.
Also asked as: what does gguf mean in llm
The name is commonly expanded as GPT-Generated Unified Format, with GG linked to Georgi Gerganov, who created llama.cpp. In practice, seeing GGUF in a Hugging Face repository name tells you the model is ready for Ollama, LM Studio, llama.cpp or GPT4All, usually offered in several quantization levels. It is the dominant format for running open models on laptops, desktops and CPUs, while safetensors remains standard for training and GPU servers.
How does a GGUF file work?
A GGUF file has a header, a key-value metadata section, tensor descriptions and aligned tensor data. Metadata stores the architecture, context length, tokenizer vocabulary, special tokens and chat template; tensor entries record names, shapes, data types and offsets.
Also asked as: how does gguf work · how do gguf models work
Because tensor data is aligned, loaders can memory-map the file: the operating system pages weights into memory as needed instead of reading the whole file upfront, which makes loading fast. Each tensor can use a different quantization type, so sensitive layers may keep higher precision. New metadata keys can be added without breaking older readers, which is why GGUF replaced its predecessors. Tools such as gguf-py in llama.cpp let you inspect metadata and tensor types.
How do GGUF models run?
A llama.cpp-based runtime reads the GGUF metadata to build the model architecture, maps the quantized weights into memory, and runs inference on CPU or GPU backends such as CUDA, Metal or Vulkan, dequantizing blocks on the fly.
You choose how many layers to offload to the GPU; the rest run on the CPU, which lets models larger than VRAM still run, slowly. The runtime keeps a KV cache for the conversation, whose size grows with context length. Chat templates from metadata format prompts correctly. Ollama and LM Studio wrap this process with model management, a chat interface and an OpenAI-compatible API, so applications can use local GGUF models like hosted ones.
When should you use GGUF?
Use GGUF when you want to run models locally on laptops, desktops or CPU servers, especially with limited VRAM, through llama.cpp, Ollama or LM Studio. It suits private, offline and low-cost inference.
Also asked as: when to use gguf
GGUF is less suitable for training or fine-tuning, which use safetensors with PyTorch, and for high-throughput GPU serving, where engines such as vLLM or TensorRT-LLM with formats like FP8, AWQ or GPTQ usually deliver more requests per second. A common workflow: fine-tune in safetensors with QLoRA, merge the adapter, convert to GGUF with llama.cpp's conversion script, quantize, then run locally. That is how small custom models end up on ordinary machines.
Which GGUF model should you use?
Pick the model by task and hardware, then the GGUF version. For chat on a laptop, a 7 to 8 billion parameter instruct model is a good start; for coding, choose a coding model; for weaker machines, try 1 to 4 billion parameter models.
Also asked as: which gguf model to use
Download from reputable uploaders on Hugging Face, ideally the model creator or well-known quantizers, and read the model card for licence, context length and recommended settings. Prefer instruct or chat versions for assistants and base versions only for completion tasks. Check that your runtime supports the architecture, since brand-new models need recent llama.cpp versions. Test two or three candidates on your real prompts, comparing quality and speed, before settling on one.
Which GGUF quant should you use?
Q4_K_M is the usual default, balancing size and quality. Use Q5_K_M or Q6_K if you have spare memory and want better quality, Q8_0 for near-original quality, and Q3 or IQ quants only when memory is very tight.
Also asked as: which gguf quant to use
Quality loss grows sharply below about 4 bits, especially for smaller models, so a larger model at Q4 often beats a smaller model at Q8. Importance-matrix quants, marked imatrix or IQ, preserve quality better at low bits. Coding, maths and structured output tasks are more sensitive to quantization than casual chat, so test those carefully. Leave memory for the KV cache, which grows with context length, when deciding how large a quant fits.
Where should you put GGUF files?
It depends on the tool. LM Studio keeps models in its models folder, in publisher and model subfolders, or imports them with lms import. Ollama imports a GGUF through a Modelfile with a FROM line pointing to the file. llama.cpp loads any path you pass.
Also asked as: where to put gguf files
For text-generation-webui, place GGUF files in its models directory. In ComfyUI, GGUF image model weights, used through the ComfyUI-GGUF extension, typically go in the models unet folder. Keep large models on a fast SSD with plenty of space, and avoid duplicating files across tools; some users point several apps at one shared folder. After moving files, refresh or restart the app so it detects them, and keep the model card alongside for reference.
What is the difference between GGUF and GGML?
GGML is the tensor library behind llama.cpp, and also the name of the older model file format. GGUF replaced the GGML file format in August 2023, adding extensible metadata, a single self-contained file and better compatibility across versions.
Old GGML files broke when new model types or settings appeared, because the format lacked flexible metadata; GGUF stores architecture details, tokenizer and templates as key-value pairs, so new information can be added without breaking readers. Modern llama.cpp, Ollama and LM Studio no longer load old GGML model files, so convert or re-download them. The ggml library itself continues as the computation engine under llama.cpp and related projects.
What is the difference between GGUF and safetensors?
Safetensors is a safe, fast format from Hugging Face for storing tensors, used by PyTorch and Transformers for training and GPU inference, usually in 16-bit precision. GGUF is a llama.cpp format for local inference, usually quantized and bundled with metadata.
Safetensors checkpoints are often split into shards with separate config and tokenizer files, while GGUF puts everything in one file. Both avoid the code-execution risks of pickle files. Use safetensors for fine-tuning, research and serving with vLLM or Transformers; convert to GGUF for Ollama, LM Studio and CPU or mixed CPU-GPU inference. llama.cpp includes scripts that convert Hugging Face safetensors checkpoints into GGUF, followed by quantization to your chosen level.
What is the difference between GGUF and FP8?
GGUF is a file format; FP8 is a numeric precision, 8-bit floating point. FP8 weights are used for fast inference and training on recent GPUs with hardware FP8 support, usually stored as safetensors and served by engines like vLLM.
GGUF files typically use integer block quantization, such as 4 or 8 bits, designed for CPUs and consumer GPUs, while FP8 targets data-centre GPUs such as NVIDIA Hopper and newer, where it nearly doubles throughput compared with 16-bit with little quality loss. Choose FP8 for high-volume GPU serving and GGUF for local, memory-constrained or CPU inference. In image models such as Flux, both appear as options: FP8 for capable GPUs, GGUF quants for limited VRAM.
Sources
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.
