CUDA: add Q2_0 (ternary, group-64) weight support#96
Open
Andgihat wants to merge 1 commit into
Open
Conversation
Wire GGML_TYPE_Q2_0 through the CUDA weight path: dequantize, MMVQ
and MMQ (load-tiles + dp4a/mma) with the {-1,0,+1,+2} symbol mapping
and sequential 2-bit layout. Register the type in convert, getrows,
supports_op, the MMQ per-arch configs and the template instances.
Blackwell inherits the config via the ampere fall-through. Verified
on a group-64 GGUF: GPU perplexity matches the CPU reference
(6.7145 vs 6.7143 over 6 chunks of 512).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the CUDA weight path for
GGML_TYPE_Q2_0(the group-64 ternary format, id 42) so ternary GGUFs — e.g. PrismML's Ternary-Bonsai-27B group-64 — run on GPU. v0.4.0 already ships the CPU reference for this type; only the CUDA kernels were missing (the tree wiredGGML_TYPE_Q2_0Sbut notGGML_TYPE_Q2_0).Provenance
The kernels are ported from upstream ggml-org#25707 ("CUDA: add Q2_0 support", by @khosravipasha), which targets the same group-64 layout. Credit for the kernel design is upstream's. This PR adapts that patch to the v0.4.0 tree so the format works on BeeLlama before ggml-org#25707 merges upstream; it should become redundant once you rebase past that merge.
Changes
dequantize.cuh,vecdotq.cuh(vec_dot_q2_0_q8_1),mmq-load-tiles.cuh(load_tiles_q2_0): the(c-1) ∈ {-1,0,+1,+2}symbol mapping over the sequential 2-bit layout.convert.cu,getrows.cu,mmvq.cu,mmq.cu/mmq.cuh,common.cuhtype traits,ggml-cuda.cusupports_op: type registration.mmq-instance-q2_0.cu+ generator entry.Q2_0S.Verification
-DCMAKE_CUDA_ARCHITECTURES=120(Blackwell / RTX 50-series),-DGGML_CUDA_FORCE_MMQ=ON.-ngl 99) vs 6.7143 (CPU,-ngl 0) over 6×512-token chunks — so both the MMVQ (decode) and MMQ (prefill) paths are correct, with no silent divergence.Happy to adjust naming/structure to your preference, or to hold this if you'd rather wait for the upstream merge.