Quantization in AI is a compression technique that stores a model’s weights, and sometimes its activations, in fewer bits so the model needs less memory. By moving from 16-bit weights down to 8-bit or 4-bit quantization, an 8B model drops from about 16GB to about 8GB or 4–5GB for weights alone, trading some accuracy for smaller footprints.
When I review model cards for local deployment, the same tags appear over and over: 8-bit, 4-bit, GPTQ, AWQ, and GGUF. Each label refers to a different way of representing numbers inside a neural network so the model fits on a single GPU, a laptop, or a smaller server instance.
How does model quantization shrink memory?
Model quantization shrinks memory by reducing the number of bits assigned to each parameter in a model’s weight matrices. As of November 2026, Hugging Face’s quantization overview notes that models are usually loaded in bf16, which takes 2 bytes per parameter. Using Hugging Face’s weights-only rule of thumb, an 8B-parameter model needs about 16GB for its weights in bf16, about 8GB at 8 bits, and about 4–5GB at 4 bits.
That 8B example is a weights-only rule of thumb, rather than a complete VRAM budget for running inference. Real serving memory also includes activations, runtime overhead, and the key-value cache that stores attention history across your prompt and output. If you plan to serve long contexts, our guide on what a KV cache is explains why memory grows alongside sequence length.
As basic storage arithmetic, each step down in bit width cuts the raw bytes needed per parameter:
| Precision format | Storage per parameter (arithmetic) | Relative weight footprint vs. BF16 |
|---|---|---|
| FP32 | 4 bytes | 2x |
| FP16 / BF16 | 2 bytes | 1x |
| FP8 / INT8 | 1 byte | 0.5x |
| FP4 / INT4 / NVFP4-style 4-bit | 0.5 bytes | 0.25x (plus format overhead) |
These byte counts describe raw storage arithmetic, not a benchmark of wall-clock speed or output quality. All else equal, going from BF16 to 8-bit halves weight memory, while going from BF16 to 4-bit cuts it to roughly one quarter, with the 4–5GB range on an 8B model accounting for quantization metadata and format overhead. As Hugging Face states, fewer bits cost some accuracy, and each quantization method makes a different tradeoff between memory, speed, and accuracy.
Two ways quantized models are loaded in practice
Quantized checkpoints enter an inference or fine-tuning pipeline through two distinct loading workflows, depending on whether compression happens on the fly or ahead of time.
- Quantizing a full-precision checkpoint while it loads. In Hugging Face Transformers, you pass a quantization configuration object to
from_pretrainedalongside a standard checkpoint. For example, the official documentation shows loadingmeta-llama/Llama-3.1-8B-Instructin 4 bits by passingBitsAndBytesConfig(load_in_4bit=True, ...). Most on-load methods skip calibration data, though GPTQ is the exception because it calibrates on a dataset as the model loads. - Loading an already-quantized checkpoint. When a checkpoint has already been compressed and saved, you load it without passing a new quantization configuration. Transformers reads the quantization details from the repository’s
config.jsonfile or from a named GGUF file. Methods such as AWQ, AutoRound, AQLM,compressed-tensors, and GGUF only support this workflow in Transformers: you start from a checkpoint quantized beforehand with the method’s own library, which is how many releases in open-weight models in 2026 are distributed.
Common quantization methods: bitsandbytes, GPTQ, AWQ, and GGUF
Common model quantization methods differ in when compression runs, which bit widths and hardware targets they support, and whether they target Transformers pipelines or standalone local runtimes.
| Method / format | Workflow in Transformers | Bit widths / numeric type | Hardware in HF docs | Key operational note |
|---|---|---|---|---|
| bitsandbytes | Quantize on load or load saved | 4-bit and 8-bit | CUDA, XPU, HPU, CPU | No calibration; supports fine-tuning |
| GPTQ | Pre-quantized or calibrate on load | 4-bit (int4 to fp16 on the fly) |
CUDA, XPU, CPU | Needs calibration dataset; use GPT-QModel |
| AWQ | Pre-quantized only | 4-bit | CUDA, XPU, CPU | Preserves salient weights via activations |
| GGUF | Pre-quantized only (gguf_file) |
1-bit to 8-bit | CUDA, MPS, XPU, CPU | Single file; standard in llama.cpp and Ollama |
| NVFP4 / Fine-grained FP8 | Quantize on load (hardware-specific) | 4-bit / 8-bit float | NVIDIA GPUs (NVFP4 on Blackwell) | Floating-point representation |
bitsandbytes
Hugging Face’s method selection guide recommends bitsandbytes when you want to fit a model in less memory with the least setup. It quantizes to 4 or 8 bits while loading without a calibration dataset, runs across CUDA, XPU, HPU, and CPU, and supports both saving and fine-tuning. Per the bitsandbytes documentation, 8-bit mode uses LLM.int8() and halves weight memory usage, while 4-bit mode powers QLoRA, which compresses weights to 4 bits while keeping a model trainable by inserting a small set of trainable low-rank adaptation (LoRA) weights.
GPTQ
GPTQ is a post-training quantization method that compresses each row of a weight matrix independently to minimize reconstruction error. According to Hugging Face’s GPTQ documentation, weights are quantized to int4 and restored to fp16 on the fly during inference, which “can save memory usage by 4x because the int4 weights are dequantized in a fused kernel rather than a GPU’s global memory,” while lower bitwidths take less time to communicate. Running GPTQ quantization requires a calibration dataset and a tokenizer, such as GPTQConfig(bits=4, dataset="c4", tokenizer=tokenizer) in the docs. Transformers no longer supports AutoGPTQ, directing users to GPT-QModel instead.
AWQ
AWQ, short for Activation-aware Weight Quantization, focuses on protecting the small subset of weights that matter most for output quality. As Hugging Face’s AWQ documentation explains, AWQ “preserves a small fraction of the weights that are important for LLM performance to compress a model to 4-bits with minimal performance degradation.” In Transformers, AWQ is pre-quantized only, loading checkpoints built with llm-awq or autoawq.
GGUF
GGUF is a single-file format that bundles a model’s weights, configuration, and tokenizer into one file rather than using a separate config.json, which is why Transformers asks for an explicit gguf_file argument. Across local tools such as llama.cpp and Ollama, GGUF is the format behind filenames tagged with labels like Q4_K_M, supporting 1-bit through 8-bit pre-quantized weights across CUDA, MPS, XPU, and CPU.
One major caveat applies specifically to Transformers: Hugging Face notes that most GGUF models are dequantized to full precision as they load in Transformers, using as much memory as an unquantized model, aside from a narrow exception using MPS and kernels on some Qwen3.5 models. That behavior belongs to the Transformers GGUF loader; local runtimes like llama.cpp and Ollama are built to execute quantized GGUF weights directly.
Low-bit floating-point formats: FP8, NVFP4, and MXFP8
Low-bit quantization also extends to floating-point formats, which differ from integer formats like INT8 and INT4. Hugging Face’s selection guide lists fine-grained FP8, MXFP4, and NVFP4, which quantizes to NVIDIA’s 4-bit float format on Blackwell GPUs while loading. Dozens of other schemes exist as well, including torchao, HQQ, and compressed-tensors, all cataloged in the Transformers selection guide.
What is the difference between weight-only quantization and full quantization?
Weight-only quantization compresses the stored parameter matrices while leaving activations and the KV cache in higher precision unless those components are quantized separately. Many popular 4-bit GGUF and weight-only inference pipelines store narrow weights in memory and dequantize or compute matrix multiplications in a wider format.
During long text generation, activations and the KV cache still consume GPU or system RAM alongside the weights. Hugging Face documents KV-cache quantization as a separate optimization step for long sequences. Architecture also shapes total memory: even in sparse designs covered in our Mixture of Experts explainer, all routed and shared experts must reside in memory even though only a fraction execute per token.
How does post-training quantization differ from native low-bit training?
Post-training quantization starts with a finished high-precision checkpoint and compresses it afterward, whereas native low-bit training trains and deploys the model directly in low-bit numeric formats. GPTQ, AWQ, most GGUF conversions, and on-load bitsandbytes fall into the post-training category, accepting some conversion error after pretraining and alignment finish.
Native low-bit training removes that post-training conversion step. A current example is Inkling 2 Mini, released by Thinking Machines Lab on November 20, 2026. According to the official Inkling 2 Mini announcement and model card, the 26.4B-parameter model has an approximate 20GB memory footprint and was trained with 4-bit NVFP4 in its expert layers on all forward passes from pretraining through reinforcement learning, plus 8-bit MXFP8 on the final four layers’ forward passes.
Thinking Machines Lab states: “Inkling 2 Mini’s weights are deployed in the same numeric precision they were trained on—20 layers in NVFP4 and 4 in MXFP8—so there is no conversion loss from quantization.” That statement means there is no accuracy loss from converting a finished BF16 checkpoint down to 4 bits after training, rather than a comparison against a separate BF16 twin. The team’s hardware guidance recommends Apple silicon with 32GB or more memory, GPUs with 32GB or more of VRAM, or a single NVIDIA DGX Spark.
Which quantization approach fits your workflow?
Choosing a quantization method depends on your runtime, hardware, and whether you need to fine-tune or only run inference. Because each method trades off memory, speed, and accuracy differently across tasks, no single bit width fits every workload.
- Choose bitsandbytes when you want to load an existing full-precision checkpoint in 4-bit or 8-bit with minimal setup, or when you plan to fine-tune with QLoRA.
- Choose GPTQ or AWQ when you want an already calibrated or activation-aware 4-bit checkpoint built for inference.
- Choose GGUF when you want a portable single-file checkpoint for local runtimes such as
llama.cppor Ollama. - Choose native low-bit checkpoints when a model developer ships weights trained directly in NVFP4, FP8, or MXFP8.
Always check the model’s license before converting or redistributing quantized weights, and leave memory headroom for the KV cache and runtime overhead. If your goal is lowering cloud inference bills rather than fitting weights onto local hardware, look at prompt and routing strategies in our guide on how to reduce AI API costs.
FAQ
Does 4-bit quantization always cut quality noticeably?
No single rule predicts the exact quality impact of 4-bit quantization across every model and task. Per Hugging Face’s quantization documentation, fewer bits cost some accuracy, and each method makes a different tradeoff between memory, speed, and accuracy. Methods like AWQ and GPTQ aim to limit error through activation awareness or calibration, while bit width, model architecture, and workload all shape the result.
Why does an 8B 4-bit model need more than 4–5GB of total VRAM?
The 4–5GB figure for a 4-bit 8B model is a weights-only rule of thumb from Hugging Face, not a full serving budget. Running a model also requires memory for activations, runtime overhead, and the KV cache that grows with prompt and generation length. Quantizing weights shrinks the static model footprint, while long contexts still demand separate memory planning.
What is the difference between GGUF and GPTQ or AWQ?
GGUF is a single-file format bundling weights, configuration, and tokenizer together for local runtimes such as llama.cpp and Ollama, supporting 1-bit to 8-bit pre-quantized checkpoints. GPTQ and AWQ are 4-bit quantization methods typically stored in directory-style checkpoints with a config.json file, where GPTQ calibrates per weight matrix row and AWQ preserves critical weights based on activations.
Does loading a GGUF file in Transformers save memory?
Most GGUF files do not save memory when loaded through Hugging Face Transformers. According to Hugging Face’s quantization overview, most GGUF models are dequantized to full precision during loading in Transformers, taking as much memory as an unquantized model, aside from a narrow MPS kernel exception on some Qwen3.5 models. Local apps like llama.cpp and Ollama avoid that Transformers behavior.
What is NVFP4 compared to INT4 quantization?
NVFP4 is NVIDIA’s 4-bit floating-point format supported on Blackwell hardware, whereas INT4 represents weights as 4-bit integers. Both formats store each parameter in 0.5 bytes, cutting raw weight storage similarly to integer 4-bit quantization. Floating-point formats like NVFP4 and 8-bit MXFP8 can also be used directly during training and deployment, as seen in Thinking Machines Lab’s Inkling 2 Mini.
Sources
- Hugging Face Transformers: Quantization Overview
- Hugging Face Transformers: Choose a Quantization Method
- Hugging Face Transformers: bitsandbytes
- Hugging Face Transformers: GPTQ
- Hugging Face Transformers: AWQ
- Thinking Machines Lab: Inkling 2 Mini Announcement
- Hugging Face Model Card: thinkingmachines/Inkling-2-Mini



