Key takeaways
- Quantization in machine learning stores weights and sometimes activations in fewer bits to save memory and speed up inference.
- Techniques include post-training quantization, quantization-aware training, weight-only and weight-and-activation schemes.
- In LLMs, 8-bit is nearly lossless, 4-bit is the popular sweet spot, and lower bits lose quality faster.
- Quantization noise is the error from rounding values to levels; it is roughly white for busy signals.
- In music, quantization snaps notes to a rhythmic grid, a feature that spread with drum machines and MIDI sequencers in the 1980s.
Quantization in machine learning means representing a model's numbers, its weights and sometimes activations, with fewer bits, such as 8-bit or 4-bit integers instead of 16 or 32-bit floats. Models become smaller and faster, with a small loss of accuracy when done well. FineTune Studio relies on it: QLoRA loads base models in 4-bit so fine-tuning fits on modest GPUs. This page covers quantization techniques, how it works in LLMs and affects them, a visual explanation, quantization noise and ADCs, and quantization in music.
What is quantization in machine learning?
Quantization maps high-precision numbers to a small set of low-precision values using a scale and sometimes a zero point, so each weight is stored as a small integer and converted back approximately during computation.
Benefits: less memory, lower bandwidth, faster inference on hardware with integer support, lower energy use, and deployment on phones, browsers and edge devices. The cost is quantization error, small approximations that can reduce accuracy. Good methods keep that loss small by choosing scales carefully, quantizing in small groups and protecting sensitive parts of the model. It is one of the most practical ways to make large models usable on ordinary hardware.
What are the main quantization techniques?
The main families are post-training quantization, applied to a trained model with little or no data, and quantization-aware training, where training simulates low precision so the model adapts. Within these, schemes differ in what they quantize and how.
Also asked as: what are quantization techniques
Key variations:
- Weight-only: weights in low bits, activations in higher precision; common for LLMs.
- Weight and activation: both quantized, often 8-bit, for maximum speed.
- Dynamic vs static: activation scales computed at runtime or calibrated in advance.
- Granularity: per-tensor, per-channel or per-group scales.
- Methods: GPTQ, AWQ, NF4 in QLoRA, llama.cpp K-quants, SmoothQuant and FP8.
How does quantization work in LLMs?
LLM weights are grouped, often in blocks of 32 to 128 values; each group gets a scale, and each weight is stored as a small integer. During inference, weights are dequantized on the fly, or used directly by specialised kernels, and multiplied with activations.
Since LLM inference is limited by memory bandwidth, moving fewer bytes per weight speeds up token generation. Advanced methods decide which weights deserve more care: AWQ protects channels with large activations, GPTQ corrects errors using second-order information, and importance matrices in llama.cpp guide I-quants. QLoRA's NF4 places levels to match the bell-shaped distribution of weights, reducing error for the same bit count.
How does quantization affect LLMs?
At 8 bits, most LLMs perform almost identically to the original. At 4 bits, quality drops slightly on most tasks, a widely accepted trade-off. Below 4 bits, quality falls faster, especially for small models, reasoning, coding and non-English languages.
Effects to watch: more factual slips, weaker multi-step reasoning, and reduced quality in less-represented languages, which matters for Hindi and other Indian languages. Larger models tolerate quantization better, so a large model at 4 bits often beats a smaller one at 8 bits. Always compare quantized and original models on your own evaluation set before deploying.
What does a visual guide to LLM quantization show?
Visual guides, such as Maarten Grootendorst's widely shared illustrated guide, show quantization as squeezing a wide range of numbers onto a short ruler of allowed values: each weight snaps to its nearest mark, and the gaps between marks are where error comes from.
They typically illustrate number formats such as FP32, FP16, BF16 and INT8 as bit layouts; symmetric and asymmetric mapping with scales and zero points; how outliers stretch the range and waste precision; why grouping helps; and how methods like GPTQ, AWQ and GGUF differ. Visual explanations are a great first step before reading papers, because the core ideas are geometric.
What is quantization noise?
Quantization noise is the error introduced when a continuous or high-precision signal is rounded to discrete levels, treated as noise added to the original signal. Its size depends on the step between levels: smaller steps, from more bits, mean less noise.
For a uniform quantizer, the noise power is roughly the step size squared divided by 12, and each additional bit improves the signal-to-noise ratio by about 6 decibels. In audio, quantization noise sets the noise floor of a recording; in images, it causes banding; in neural networks, it slightly perturbs computations. Techniques such as dithering and noise shaping manage it in audio.
Is quantization noise white?
Approximately, under common conditions. When a signal is complex and varies across many quantization levels, the error is nearly uniformly distributed and uncorrelated with the signal, so it behaves like white noise, with equal power across frequencies.
The approximation breaks down for simple or low-level signals, such as a pure tone near a few quantization steps, where error becomes correlated with the signal and produces harmonic distortion. Adding dither, a small amount of random noise before quantization, decorrelates the error and makes it behave like benign white noise, which is why audio engineers dither when reducing bit depth.
What is quantization noise in an ADC?
In an analogue-to-digital converter, quantization noise is the error from mapping each continuous voltage sample to the nearest of a finite number of digital codes. An N-bit ADC has 2 to the power N levels, and the error per sample lies within half a step.
Also asked as: what is quantization noise in adc
The theoretical signal-to-quantization-noise ratio for a full-scale sine wave is about 6.02 times N plus 1.76 decibels, so a 16-bit ADC reaches about 98 dB. Real ADCs also have thermal noise and nonlinearity, summarised as effective number of bits. Oversampling with noise shaping, as in sigma-delta converters, pushes quantization noise out of the band of interest for high-resolution audio.
What is quantization in music?
In music production, quantization moves recorded notes so they align to a rhythmic grid, such as sixteenth notes, correcting timing in MIDI performances or audio. It makes rhythms tighter, though too much can sound mechanical.
Digital audio workstations let you choose grid size, quantize strength, swing and groove templates that keep some human feel. Producers often quantize drums firmly and other parts lightly. In live performance software such as Ableton Live, launch quantization makes clips start on the beat. The concept mirrors signal quantization: continuous timing is snapped to discrete positions.
When did quantization in music start?
Timing quantization appeared with electronic sequencers and drum machines in the late 1970s and early 1980s, notably machines such as Roger Linn's LM-1, which offered timing correction and swing. The MIDI standard in 1983 and computer sequencers then made quantization common.
By the late 1980s and 1990s, software sequencers and digital audio workstations offered sophisticated quantization, groove templates and later audio quantization that moves recorded audio itself. The tight, quantized feel shaped genres such as electronic dance music and hip-hop, while producers also developed swing and humanise features to reintroduce natural timing.
Sources
- Hugging Face Transformers, Quantization overviewhuggingface.co
- Jacob et al., Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference (2017)arxiv.org
- Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023)arxiv.org
- Maarten Grootendorst, A Visual Guide to Quantizationnewsletter.maartengrootendorst.com
- Wikipedia, Quantization (signal processing)en.wikipedia.org
- Wikipedia, Quantization (music)en.wikipedia.org
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.
