Huawei's MACaQ Aims to Speed Up Quantization for Local GGUF Models Across Text, Image, Video and Audio

Huawei CSL has published MACaQ, short for Model Agnostic Calibrated Quantization, a toolkit that produces GGUF models for local inference across text, image, video and audio. The GitHub repository, dated 11 November 2026, describes the project as high-quality GGUF for local inference and lists llama.cpp, vLLM, Diffusers and PyTorch as the runtimes its output targets.

GGUF is the file format llama.cpp uses to load models on consumer hardware. Quantization stores a model's weights at lower numeric precision, so the model takes less memory, at some cost in accuracy.

On the text side, MACaQ supports 2-bit, 3-bit and 4-bit GGUF formats for text generation under llama.cpp. Its vLLM integration uses W4A4 as the default inference mode, meaning 4-bit weights paired with 4-bit activations (the intermediate values a model computes while it runs). The repository is explicit that the vLLM path is the default, which places activation quantization, not only weight quantization, at the center of the pipeline.

The headline throughput figure concerns the quantization job itself. According to the repository, MACaQ can quantize a 122B-parameter model in 64 minutes using 32.5 GB of device memory. The same material claims lower perplexity than MR-GPTQ and DuQuant++ and, on Llama-3-8B, matching their best perplexity with 11x less quantization time. Perplexity is a standard measure of how well a language model predicts text, and lower is better.

The most detailed benchmark concerns Qwen3.8-27B served through vLLM. The repository gives a perplexity of 2.8994 for MACaQ W4A4 against 2.8463 for BF16, a 16-bit floating-point baseline, a gap of about 0.05 points. It also reports 2.33x prefill (processing the input prompt) and 2.58x decode (generating output tokens) speedups for the same configuration. These are the numbers a practitioner will compare against their own evaluation harness.

The broader context here is the familiar trade between memory footprint, throughput and output quality that anyone who has deployed a quantized model already manages. The W4A4 figures sit on that trade curve: a small perplexity penalty in exchange for substantial gains in both prefill and decode throughput. Whether that penalty is acceptable depends on the workload. A perplexity delta on a single model family is a starting point for evaluation, not a verdict.

In my view, the more consequential detail for local deployment is the calibration cost. A quantization pass that fits a 122B-parameter model into 32.5 GB of device memory and completes in about an hour moves the calibration step from a cluster job toward a single well-provisioned workstation. That matters most for teams that serve fine-tuned or domain-specific checkpoints and need to requantize them often, because calibration has historically been the bottleneck that made per-checkpoint quantization expensive.

The multimodal scope is also worth noting. Most GGUF tooling has grown out of text-only language models, and the repository positions MACaQ as covering image, video and audio models, with Diffusers listed alongside llama.cpp and vLLM. Whether those non-text paths reach the same quality and speed profile as the text benchmarks is a question the published material answers only in part. Anyone adopting the toolkit for generative media should test those paths directly.

For engineers who maintain local inference stacks, the practical next step is straightforward. Run the released quantization pipeline against a model already in production, compare perplexity and latency under your own serving configuration, and decide from those measurements whether the speedups hold for your workload.