Evidence-first authority page
What Is Model Quantization?
A definition of model quantization, why it enables running large models on smaller hardware, and the accuracy trade-off involved.
Direct answer
Model quantization reduces the numerical precision used to store a model's parameters (for example, from 16-bit to 4-bit numbers), shrinking the model's memory footprint and often speeding up inference — at the cost of some accuracy loss, which varies by model and quantization level and needs to be tested for your specific use case rather than assumed.
Last reviewed 2026-08-29
This page remains noindex while its page-level source receipts are completed and verified. It is available for review, not presented as publication-ready authority.
Why lower precision saves resources
A model's parameters are numbers; storing each one with fewer bits directly reduces the total memory required to hold the model and can reduce the computation needed to run it, particularly relevant for running models on consumer hardware or reducing cloud inference costs at scale.
The accuracy trade-off is real but variable
Aggressive quantization can measurably degrade output quality, but the degree varies significantly by model architecture, quantization technique, and task — some models tolerate 4-bit quantization with minimal quality loss on many tasks, while others degrade more noticeably. Test the specific quantized model against your actual use case rather than assuming a fixed quality cost.
Why this matters for local and on-premise deployment
Quantization is often the deciding factor in whether a capable model can run on available local hardware at all — see what is a local AI agent and on-premise AI agents for the broader deployment trade-offs this connects to.