Quantization Techniques: GGUF/AWQ/GPTQ Benchmark Comparison

Chapter 28: Quantization Techniques: GGUF/AWQ/GPTQ Compared

Quantization is the art of trading "slightly lower precision" for "dramatically reduced resource requirements." For Hermes Agent, this isn't an academic topic — it's what makes running an 8B model on a MacBook or a 70B model on consumer-grade GPUs possible. This chapter benchmarks three mainstream quantization approaches on their principles, measured performance, and recommended configurations.

28.1 Why Quantization Matters

Large language model weights are typically stored as FP32 or FP16. For a 70B parameter model:

FP32:   70B × 4 bytes  = 280 GB  (completely impractical)
FP16:   70B × 2 bytes  = 140 GB  (requires 2×A100 80GB)
INT8:   70B × 1 byte   = 70 GB   (fits on one A100, barely)
INT4:   70B × 0.5 byte = 35 GB   (achievable on consumer hardware)

Quantization maps floating-point weights to low-bit integer representations, trading limited precision loss for major gains in memory efficiency and inference speed.

Three Quantization Approaches at a Glance

28.2 GGUF: The Flexible Multi-Platform Format

Technical Principle

GGUF (GPT-Generated Unified Format) is llama.cpp's file format, paired with its group quantization algorithm:

Process:
1. Divide weight matrices into groups (typically 32 or 128 weights per group)
2. Compute per-group scale factor and zero point
3. Quantize weights to target bit width
4. At inference: quantized_value × group_scale + zero_point ≈ original weight

K-Quant improvement (the "K" in Q4_K_M):
- Apply higher precision to important layers (attention, embeddings)
- Apply lower precision to less critical layers
- Achieves significantly better accuracy at the same average bit width

GGUF Quantization Levels

from llama_cpp import Llama

# Q4_K_M: recommended for personal GPU
llm = Llama(
    model_path="./hermes-3-llama-3.1-70b.Q4_K_M.gguf",
    n_gpu_layers=-1,   # load all layers to GPU
    n_ctx=8192,
    n_batch=512,
    verbose=False
)

# CPU offload when VRAM is insufficient
llm_hybrid = Llama(
    model_path="./hermes-3-llama-3.1-70b.Q4_K_M.gguf",
    n_gpu_layers=20,   # 20 layers on GPU, rest on CPU
    n_ctx=4096,
    n_threads=8
)

28.3 AWQ: Activation-Aware Quantization

Technical Principle

AWQ's core insight: not all weights are equally important. By analyzing activation value distributions, AWQ identifies "salient weights" that most affect model output, then protects them with higher effective precision:

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "NousResearch/Hermes-3-Llama-3.1-70B"
quant_path = "hermes-3-70b-awq"

model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM"
}

model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)

AWQ with vLLM (Recommended Production Setup)

python -m vllm.entrypoints.openai.api_server \
  --model hermes-3-70b-awq \
  --quantization awq \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.95 \
  --max-model-len 32768 \
  --host 0.0.0.0 \
  --port 8080

28.4 GPTQ: Reconstruction-Based Quantization

GPTQ is based on Optimal Brain Quantization (OBQ): process weights layer by layer, quantizing each column while using the Hessian matrix to compensate remaining weights for the introduced error.

from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

quantize_config = BaseQuantizeConfig(
    bits=4,
    group_size=128,
    desc_act=True,  # sort by activation (better accuracy, slower quantization)
)

model = AutoGPTQForCausalLM.from_pretrained(model_name, quantize_config)
examples = get_calibration_data(tokenizer)
model.quantize(examples)
model.save_quantized("hermes-3-8b-gptq-4bit-128g")

28.5 Precision Loss Benchmarks on Hermes Agent Tasks

8B Model Comparison (Hermes-3-Llama-3.1-8B)

70B Model Comparison (Hermes-3-Llama-3.1-70B)

AWQ vs GPTQ vs GGUF Q4_K_M (8B, head-to-head)

Key conclusions:

• Accuracy: AWQ > GGUF Q4_K_M > GPTQ (small gap, ~0.1–0.5 PPL)
• Speed: AWQ ≈ GPTQ > GGUF (pure GPU, AWQ/GPTQ are 50–70% faster)
• Flexibility: GGUF supports CPU offload; AWQ/GPTQ require GPU

28.6 Recommended Configurations by VRAM Budget

Decision Matrix

Quick Format Selection Guide

Should I use GGUF, AWQ, or GPTQ?

Do you have pure NVIDIA GPU and prioritize inference speed?
  → Yes → AWQ + vLLM (batch), or GPTQ (broad compatibility)
  → No  → GGUF

Do you need CPU to participate in inference (memory offload)?
  → Yes → Must use GGUF
  → No  → AWQ or GPTQ also viable

Are you running on Apple Silicon?
  → Yes → GGUF (Metal acceleration, only practical option)

Is accuracy more important than speed?
  → Yes → Q5_K_M or Q8_0 (GGUF), or AWQ
  → No  → Q4_K_M (GGUF)

Need to serve multiple users with OpenAI-compatible API?
  → Yes → AWQ/GPTQ + vLLM (better concurrency)
  → No  → GGUF + llama.cpp server is fine

28.7 Summary

The three quantization techniques each suit different scenarios:

• GGUF: Most flexible, supports CPU offload and Apple Silicon — the go-to for individual users and mixed hardware environments
• AWQ: Best accuracy INT4 option, fastest on pure GPU, recommended for production GPU servers
• GPTQ: Most mature toolchain, broadest compatibility — a reliable alternative to AWQ
• Precision loss: Q4_K_M causes ~2.4% PPL increase on 8B models and ~2.1% reduction in Function Calling success — acceptable for most scenarios
• 70B + Q4_K_M: Agent task success still reaches 87.2%, making it the ideal choice when hardware is constrained

Discussion Questions

1. Quantization "precision loss" varies significantly by task type. Why are math reasoning tasks (GSM8K) more sensitive to quantization, while text generation tasks are more robust?

2. AWQ requires a calibration dataset. If your business data distribution differs significantly from the general calibration data (WikiText), how should you handle this? How much would re-calibrating with domain data improve accuracy?

3. The benchmark shows Q4_K_M inference speed (28 tok/s) is slower than Q8_0 (24 tok/s). This seems counterintuitive — why would a smaller data type be slower? Explain the likely cause.

4. For Function Calling, which requires precise JSON output, is there a way to improve accuracy specifically for that task without raising global quantization precision?