Skip to content
dphnAIPublic

About

Lossless compression of BF16 models to 12-bits

Resources

Stars

32 stars

Watchers

0 watching

Forks

Repository files navigation

nf12

NF12 is a lossless, fixed-length 12-bit encoding of BF16 weight matrices, based on "NyoomFloat12: Accelerating LLM Inference via Lossless 12-bit Weight Compression" (Liberman, Fang, Zhang, Dao, Fu, Together AI, 2026). This library implements this data type efficiently on NVIDIA GPUs, from the Ampere generation up until Blackwell, on AMD RDNA4 (gfx1200 / gfx1201) via HIP, and on Apple Silicon GPUs through Metal for PyTorch MPS and MLX (see docs/METAL.md).

This library's bit and memory layout (tiled) are not compatible with the one described in the paper. You can read the spec for more info on the format.

Requirements

  • NVIDIA GPU with SM80 or newer (Ampere and later), an AMD RDNA4 GPU (gfx1200 / gfx1201), or an Apple Silicon Mac (macOS 14+).
  • CUDA toolkit 13.0+ for NVIDIA, ROCm with amdclang++ for AMD, or Xcode with its Metal Toolchain (xcodebuild -downloadComponent MetalToolchain) for Apple.
  • A C++ compiler, CMake 3.26+, and Ninja for builds.
  • uv for Python environment management and builds.
  • Python 3.10+ and PyTorch 2.10+ with CUDA, ROCm or MPS support for the Python package; optionally MLX 0.32+ on a Mac.
  • Weight shapes: N (out_features) a multiple of 16, K (in_features) a multiple of 64.

Install

uv pip install nf12

Linux x86-64 and ARM64 wheels include kernels for SM80, 86, 89, 90, 100, 103, 110, and 120. They bundle the CUDA 13 runtime and require a compatible NVIDIA driver. There are no macOS wheels yet; build the Metal backend from source.

From source:

uv sync --locked                          # all supported architectures
NF12_CUDA_ARCHS=120 uv sync --locked      # one architecture for development
uv sync --locked --extra mlx              # macOS: Metal for PyTorch MPS and MLX

To defer compilation until the first native operation:

NF12_BUILD_NATIVE=OFF uv sync --locked

You can build the C library alone with CMake:

cmake -B build/c -G Ninja -DNF12_BUILD_TORCH=OFF -DNF12_CUDA_ARCHS=120
cmake --build build/c

AMD RDNA4 (ROCm)

The HIP backend supports gfx1200 and gfx1201 with a C API, and the full python/nf12 / torch.ops.nf12 API: the torch extension compiles the same csrc/torch/ops.cpp as the CUDA build and backs it with the HIP kernels, so there is one op layer rather than a second ROCm implementation. That covers encode/decode, linear (including the bf16t twin), pack_bf16t, embedding and the whole MoE op set (moe_align, moe_linear, moe_combine, moe_silu_mul, moe_decode, _fused_moe). _fused_moe at T == 1 takes a native two-kernel fused route instead of the composed one; see csrc/hip/README.md.

Validated on ROCm 10.1.0 with HIP 7.16.26385 on Radeon AI PRO R9700 (gfx1201). See the validated toolchain and test instructions.

The package follows the CUDA build's NF12_BUILD_NATIVE switch:

Setting ROCm wheel contents Compilation
ON (default) HIP libraries and sources During the package build
OFF Sources only JIT on first native use, cached per user

NF12_ENABLE_HIP enables the HIP backend; it does not control when compilation happens. With ROCm PyTorch installed, the top-level build enables HIP by default. For a native ROCm wheel, build in the environment containing the ROCm PyTorch wheel and the build tools:

python -m pip install 'scikit-build-core>=0.10' ninja
python -m pip wheel --no-build-isolation --no-deps .

An isolated build resolves its own PyTorch wheel and may select CUDA or CPU instead. The torch extension must be built against the ROCm wheel that loads it.

Build from source with the ROCm toolkit, a C++ compiler, CMake 3.26+ and Ninja.

cmake -S . -B build/hip -G Ninja \
  -DNF12_ENABLE_HIP=ON -DNF12_BUILD_TORCH=ON \
  -DNF12_HIP_ARCHS=gfx1201 \
  -DCMAKE_HIP_COMPILER=/opt/rocm/bin/amdclang++
cmake --build build/hip --target nf12_torch

Use -DNF12_HIP_ARCHS=gfx1200 for a gfx1200 GPU. To build only the C library, set -DNF12_BUILD_TORCH=OFF and build --target nf12_hip. When an installed wheel has no libraries, nf12._jit builds both from its packaged sources.

Fused linear kernels support batches of 1–256 tokens. For larger batches, the torch API decodes bounded slices and runs BF16 GEMMs; the raw C API uses native M chunks with caller-owned workspace. See the ROCm validation report for measured performance.

See the shared ABI design for the CUDA/HIP boundary.

See the HIP documentation for R9700 benchmark results and kernel implementation details.

Python

import torch
import nf12

w = torch.randn(4096, 4096, dtype=torch.bfloat16, device="cuda")
nw = nf12.encode(w)
assert torch.equal(nf12.decode(nw).view(torch.int16), w.view(torch.int16))

x = torch.randn(2, 7, 4096, dtype=torch.bfloat16, device="cuda")
y = nf12.linear(x, nw)  # preserves leading dimensions

On a Mac, use device="mps": the same API runs on the Metal backend. MLX arrays work too, through the same functions or nf12.mlx (which does not import torch):

import mlx.core as mx
import nf12.mlx as nm

w = mx.random.normal((4096, 4096)).astype(mx.bfloat16)
nw = nm.encode(w)  # nf12.encode(w) routes MLX arrays here as well
y = nm.linear(mx.random.normal((7, 4096)).astype(mx.bfloat16), nw)

Build with the mlx extra (uv sync --locked --extra mlx) for MLX.

Use nf12.NF12Linear.from_linear(layer) to convert a BF16 CUDA or MPS nn.Linear. Decoding recovers the original weights. GEMM outputs can differ from cuBLAS due to accumulation order.

Mixture of experts

w13 = nf12.encode_moe(gate_up)  # BF16 [E, 2I, K], gate rows first
w2 = nf12.encode_moe(down)      # BF16 [E, K, I]
# x: [T, K], router weights and IDs: [T, topk]
out = nf12.fused_moe(x, w13, w2, topk_weights, topk_ids)

Embedding and tied heads

nw = nf12.encode(model.embed_tokens.weight)  # BF16 [V, H]
x = nf12.embedding(token_ids, nw)           # [*token_ids.shape, H]

embed = nf12.NF12Embedding.from_weight(nw)
lm_head = nf12.NF12Linear.from_weight(nw)    # shares the encoded weights

Benchmarks

Show benchmark tables

Serving throughput vs BF16

NF12 / BF16 output tokens per second. Each cell shows concurrency 1 / 8 / 32. Above 1× favors NF12. HTTP generation, 128 input / 256 output tokens, 68 server runs with alternating BF16/NF12 order. PyTorch 2.14, CUDA 13.0.

GPU Qwen3-8B Granite 1B MoE Qwen3-30B-A3B
CMP 170HX (SM80) 1.24× / 1.16× / 1.07× 0.93× / 1.08× / 1.01× NF12 only*
RTX A6000 (SM86) 1.30× / 1.24× / 1.15× 1.04× / 1.16× / 1.18× —
RTX 6000 Ada (SM89) 1.07× / 1.14× / 1.23× 1.03× / 1.17× / 1.16× —
H200 (SM90) 1.02× / 1.01× / 0.86× Unstable 1.09× / 1.10× / 0.93×
B200 (SM100) 1.07× / 0.95× / 0.81× 0.95× / 0.80× / 0.78× 0.90× / 0.83× / 0.87×
Jetson Thor (SM110) 1.32× / 1.34× / 1.28× 1.10× / 1.14× / 1.14× NF12 only*
RTX PRO 6000 Blackwell (SM120) 1.25× / 1.18× / 1.21× 1.22× / 1.14× / 1.06× 1.27× / 1.23× / 1.26×

*Qwen3-30B fit only with NF12 under the tested memory caps: 75% on CMP 170HX, 40% on Thor. BF16 was not attempted there. —: excluded on the 48 GB GPUs. H200 Granite timings were unstable, so no pooled speedup is reported.

The serving harness used a benchmark-only nf12.fused_moe adapter. The baseline used native BF16 backends, including FlashInfer TRT-LLM on B200. Prefill results are mixed. Greedy outputs can differ; these are not accuracy results.

Loaded weight memory

Model BF16 GiB NF12 GiB
Qwen3-8B 15.27 11.49
Granite 1B MoE 2.50 1.89
Qwen3-30B-A3B 56.88 42.80

Single-token MoE kernels

NF12 layer latency in µs (speedup over the previous composed NF12 path, not BF16). Real weights, uniform routing, cold CUDA graphs rotating at least 2 GiB. Median of two run medians. PyTorch 2.15, CUDA 13.2.

GPU Qwen3-30B-A3B DeepSeek-V2-Lite
H200 23.2 (1.37×) 30.5 (1.24×)
B200 18.9 (1.48×) 22.1 (1.49×)
RTX PRO 6000 Blackwell 38.9 (1.20×) 59.6 (1.10×)

Reference

License

Apache-2.0.

About

Lossless compression of BF16 models to 12-bits

Resources

Stars

32 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages