Authors: Sirine Ayadi, Sándor Daróczi, Stephan Günneman, Bertrand Charpentier
Published at TMLR 2026 & ICML 2026 SCALE Workshop
We study reliability bit-level scaling laws for quantized LLMs to find the optimal precision that maximizes the reliability under a fixed bit budget. Our reliability evaluation covers uncertainty, calibration, and robustness to 15 natural input perturbations. We find that 4-bit precision offers the best reliability-efficiency tradeoff across tasks, model families, and quantization methods.
Real users type with tpyos, slang, emoji :), and miXeD CasE. We implement 15 natural input perturbations on the character- and word-level to evaluate model robustness.
Overview of our character-level and word-level input perturbations. Illustrated is an example where perturbations with intensity level 1 are applied to a standard question prompt.
Radar plots of the accuracy (Top) and AUCROC (Entropy) (bottom) across all 15 character-level and word-level perturbations for two intensities. We evaluate the base LLaMa-3-8B model and five 4-bit quantization methods. Quantized models can provide more reliable uncertainty estimates under natural perturbations compared to their base counterparts, while maintaining close performance.
We characterize trends in reliability as the total number of bits scales. We model a metric as a function of total bits using a log quadratic scaling law.
Bit-level scaling trends of the accuracy and AUCROC (Entropy) on TriviaQA. We use four base models (blue): LLaMA-3.2-1B, LLaMA-3.2-3B, LLaMA-3-8B, and LLaMA-3-70B, and their corresponding quantized variants using six quantization methods and different bitwidths.
Full code release comign soon. Star this repo to get notified when it drops.
The release will include:
- Model quantization and loading scripts
- Reliability evaluation pipeline
- KL divergence evaluation scripts
- Hydra configs for all experiments in the paper
- Notebooks to reproduce the figures
MIT — see LICENSE for details.