Breaking the 1.58-bit Barrier for Ternary LLMs
4 hours ago
- Ternary LLMs store weights as -1, 0, or +1, conventionally costing 1.585 bits per weight, but practical packing (five-trit) rounds to 1.625 bits per weight.
- Measurement of 29 ternary models shows zeros account for up to 51.5% of weights, motivating a distribution-adaptive approach.
- Introduces BITCOS, a layout combining a presence bitmap and compacted sign vector, achieving 2 - z bits per weight (z = zero density), reaching 1.485 bits per weight on the sparsest model.
- BITCOS outperforms five-trit packing in 26 of 29 tested models and is compatible with efficient unpacking on AVX-512, AVX2, and Intel Xe2 GPUs.
- Realized throughput gain up to 1.28× in matrix-vector multiplication kernels, and end-to-end LLM inference improvement up to 1.18× on CPUs and 1.27× on GPUs.