XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression

• Paper
• Code

Abstract

Large language models require substantial memory during generation because KV cache grows with context length. XQuant is a training-free, plug-and-play framework for KV cache quantization that combines (1) a computationally light data-free calibration method and (2) cross-layer KV cache compression. The method reaches sub-1.4-bit equivalent quantization and reports stronger trade-offs than prior methods such as KIVI-2bit and AsymKV-1.5bit on TruthfulQA and LongBench.

BibTeX

@inproceedings{yang2025xquant,
    title={XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression},
    author={Yang, Haoqi and Yao, Yao and Li, Zuchao and Qi, Baoyuan and Guoming, Liu and Zhao, Hai},
    booktitle={Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
    publication={EMNLP},
    pages={9796--9811},
    year={2025},
    url={https://arxiv.org/abs/2510.11236},
    code={https://github.com/brinenick511/XQuant}
}