BITCOS Cuts Ternary LLM Storage Below 1.585 Bits by Tracking Zeros

A 14 September preprint from Evangelos Georganas, Alexander Heinecke and Pradeep Dubey introduces BITCOS, a storage layout for ternary large language models that gets below the textbook 1.585-bit limit by exploiting zero sparsity. arXiv

The paper, titled 'Breaking the 1.58-bit Barrier for Ternary LLMs' and posted to arXiv on 16 September 2026, covers models whose weights are limited to three values: -1, 0 and +1. The standard reference cost for three values is log2 3, about 1.585 bits per weight. In practice, the authors state, the common format packs five ternary weights into one byte, which works out to 1.625 bits per weight.

The core finding is about distribution. The authors measured 29 ternary models and found zeros make up to 51.5% of all weights. Fixed five-weight packing cannot use that imbalance. It uses the same space whether a tensor has many zeros or few.

BITCOS uses a different layout that adapts to that imbalance. It pairs a dense presence bitmap, which records zero versus non-zero, with a compacted sign vector that records plus or minus only for the non-zero weights. Think of it as a seating chart that first marks which seats are taken, then lists names only for those seats. The cost is 2 - z bits per weight for zero share z. Memory traffic dominates decode, so as z rises the footprint drops below fixed packing.

On compression, BITCOS uses less space than five-weight packing in 26 of 29 tested models. It reaches 1.485 bits per weight on the sparsest model. That is the lowest point in the sweep, not an average. Three denser models do not benefit under this scheme.

Smaller storage alone does not speed up inference. Unpacking has to stay cheap. The paper presents optimized BITCOS unpacking sequences for AVX-512, AVX2 and Intel Xe2 GPUs. Those cover mainstream server CPU SIMD paths and one current GPU target. The kernels are tuned for ternary matrix-vector multiplication, the operation that sets token-by-token decode throughput.

The paper reports up to 1.28x gain over production state-of-the-art ternary matrix-vector multiplication kernels at real-world zero densities. It then reports end-to-end inference across 5 platforms, with decode throughput improving by up to 1.18x on CPUs and 1.27x on GPUs. arXiv

In my view, the shift here is from fixed packing to a layout that follows sparsity. Ternary quantization had already narrowed the value range to three options. BITCOS narrows the work to what is stored. The presence bitmap is dense and predictable to scan. The sign vector shrinks as sparsity rises. That structure fits vector unpack and masked load logic.

The broader context for deployment teams is that the 26-of-29 result matters more than the 1.485 figure. It suggests the method follows naturally occurring sparsity rather than one tuned checkpoint. Teams would still need to measure per model. Dense ternary tensors will not gain. Sparse ones gain in both capacity and bytes moved per token, which is why CPU and GPU decode both move. For memory-constrained serving, that combination is often more useful than a lower bit number on its own.