[planetf1]

Pull Request

Issue

Fixes #1735

Description

On llama-server, Granite Switch adapter functions that read documents gave wrong answers, with no error. Same server, same 3b preview checkpoint, main against this PR:

• rag.check_answerability with the answer in the document: "unanswerable" on main, "answerable" here.
• rag.find_citations: [] on main, the right citation here.
• rag.flag_hallucinated_content on a sentence the document supports word for word: "unfaithful" on main, with an invented reason ("the population of Paris is actually around 2.2 million"). Here it's "faithful".

Adapter functions that take no documents (check_certainty, guardian_check, and decisions on the 8b) already worked on llama-server and give the same results either way. Adapter activation was never the problem: llama-server renders the Switch chat template and honours chat_template_kwargs.adapter_name.

The PR also documents the route. The released 8b and 30b have published GGUFs that llama-server serves directly; the 3b preview, which carries the RAG and core adapter functions, has none, so the PR adds a script that converts and serves it. That gives a Mac or any GPU-less box a supported way to run these adapter functions.

What changed in the request

Two parts of the intrinsic request went in fields only vLLM reads:

• Documents went out as a top-level documents field, which llama-server drops, so the template never saw them. They now also go in chat_template_kwargs.
• Output schemas went out as vLLM's structured_outputs, which llama-server ignores, so adapters decoded unconstrained. The schema now also goes out as the OpenAI-standard response_format, which llama-server enforces.

On vLLM these are duplicates it folds back together, and a live before/after run shows no change (see Testing). A caller's own chat_template_kwargs.documents or response_format still takes precedence. Decision adapters carry no schema, so neither change reaches them.

Serving a Switch model

Which checkpoint to serve depends on the adapter functions you want, because no published checkpoint has all of them. Both routes use port 8081, so they run beside the base-Granite server on 8080:

• Decisions and guardian-core: IBM's published GGUF, llama-server -hf ibm-granite/granite-switch-4.1-8b-GGUF:Q4_K_M --alias ibm-granite/granite-switch-4.1-8b --port 8081 --jinja (~6.4 GB). --alias makes the server report the name Mellea's adapter discovery and the example expect.
• RAG, core and guardian: the 3b preview, which has no published GGUF. scripts/start_llamacpp_switch.sh converts it to an f16 GGUF and serves it (~7 GB download, ~9 GB GGUF).

For the script, later runs reuse the cache and are ready in seconds. The converter runs in its own venv, with transformers overridden to 5.x because the converter's 4.57.6 pin can't load current Switch tokenizers. It's pinned to llama.cpp build b11675, the first with ggml-org/llama.cpp#30621. Current Switch checkpoints use MultiSwitch, and older converters, v0.7.0 included, write a block count that llama-server refuses to load (ggml-org/llama.cpp#30617).

Docs, examples and marker

• docs/docs/advanced/intrinsics.md gets a "Granite Switch on llama-server" section beside the local-checkpoint one. It covers prerequisites, the two start commands and the backend snippet. The backend note, the decisions section, tutorial 09 and both example READMEs link to it.
• docs/examples/granite-switch/answerability_llamacpp.py (3b preview) and docs/examples/intrinsics/decisions_llamacpp.py (8b, from feat(intrinsics): one-token decisions (decide/choose/score) #1742, now aligned) skip with the start command when the right server isn't up. Neither downloads weights.
• A new llamacpp_switch marker is gated on a llama-server at LLAMACPP_SWITCH_BASE_URL (default port 8081). llamacpp keeps meaning the base-Granite server.
• GraniteChatCompletion accepts identical documents in both places, so it validates the requests the backend now sends.

Limitations

• The script converts to f16 only. Quantized output is tested only through IBM's published 8b Q4_K_M.
• Tested on llama-server v0.6.0 and v0.9.0. The granite-switch architecture has been in llama.cpp releases since v0.1.0.
• No single Switch checkpoint has every adapter function, so an app needing decisions and RAG checks runs two servers (on two ports) and two backends.
• Standalone aLoRA adapters on llama-server (a base GGUF plus per-adapter files) stay out of scope, per enhancement(backends): run Granite Switch adapter functions on llama-server (llama.cpp) #1735.

Testing

• Tests added to the respective file if code was changed

• New code has 100% coverage if code was added

• Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

• llama-server before/after (v0.9.0, Apple Silicon, 3b preview, main vs this PR): the three results above. check_certainty (0.895) and guardian_check (0.99998) agree on both arms.

• vLLM before/after (v0.30.0, 3b preview, main vs this PR): answerability (both labels), citations, hallucination detection and check_certainty give identical output on both arms, and guardian_check differs by 4e-6. With a 20-sentence document, the answerability prompt is 335 tokens on both arms, so the duplicate documents render once.

• Decisions on llama-server, published GGUF (granite-switch-4.1-8b-GGUF:Q4_K_M via -hf, the documented command): decisions_llamacpp.py passes, directly and under pytest. decide gives 0.901, choose picks payments at 0.999, score gives 1.70, and label mass is at least 0.999 on every call. guardian_check scores 0.999994 on a biased prompt and 6.0e-5 on a benign one.

• Decisions on llama-server, converted (8b f16 from the script, converted from an empty cache, 40 blocks): the same example passes, with decide 0.904, choose 0.999, score 1.68, and guardian_check 0.99995 against 2.3e-5. So Q4_K_M moves the scores by under 0.02.

• 3b preview on llama-server: answerability_llamacpp.py passes, and query clarification returns a well-formed clarification. Each example skips with its start command when the server holds the other model.

• Earlier runs (llama-server 0.6.0): all 12 Switch io.yaml schemas compile through response_format, and constraining leaves the logprob-scored adapters' scores unchanged to about 1e-5. The converter install resolves the same transformers from any directory.

• Unit. In test/backends/test_openai_intrinsics_unit.py (48 pass), the documents and response_format tests fail without their fix. Separate tests pin that a caller's own documents, structured_outputs or response_format win, and that a decision request carries no response_format. test/test_example_collection.py pins that llamacpp and llamacpp_switch each gate only on their own server, and fails if the Switch gate is removed.

• Full suite, uv run pytest test/ -m "not qualitative" at d8ef166: 5341 passed, 205 skipped, 0 failed. test_adapter_versions_unchanged was deselected: it compares the pinned granitelib SHAs with the live Hub repos, which have since moved (README edits, plus a new decision/ directory in the core repo), so it fails whatever this PR does, and CI skips it.

• The final commit changes only docs, the example's docstring and skip messages, and script comments; test/test_example_collection.py and the intrinsics unit tests (70) pass on it.

• pre-commit passes on every commit, and shellcheck -S style is clean on the script.

Attribution

• AI coding assistants used

---

Adding a new component, requirement, sampling strategy, or tool?

If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.

• Component
• Requirement
• Sampling Strategy
• Tool

NOTE: Please ensure you have an issue that has been acknowledged by a core contributor and routed you to open a pull request against this repository. Otherwise, please open an issue before continuing with this pull request.

[planetf1]

Moving to draft until #1742 (one-token decisions) merges. This PR then needs a rebase (both PRs add blocks in _generate_from_intrinsic) and a live decision check against an 8b Switch GGUF, since none of the checks here exercise the decision adapter. Will mark ready after that.

[planetf1]

this is ready for review

[planetf1]

I propose holding this until #1700 (shim removal) merges, then rebasing once.

#1700 rewrites _generate_from_intrinsic in openai.py, where this PR changes the docstring near 1087 and adds a documents mirror plus response_format fallback near 1279, and rewrites test_openai_intrinsics_unit.py, where this PR adds tests near 701. Resolving that once, after #1700, beats resolving it twice.

Not a hard dependency: this merges cleanly against main today.

[ajbozarth]

Some feedback from Claude: looks good overall. A few items inline; I'll leave the approval call to other reviewers.

[ajbozarth]

"A caller's own response_format wins" only holds on vLLM for json_schema. vLLM applies response_format on top of structured_outputs via dataclasses.replace, so a caller's json_object plus the io.yaml json fails StructuredOutputsParams validation ("only one kind of structured outputs constraint"), and a caller's text leaves the io.yaml constraint in place. test_user_response_format_not_overwritten uses json_object, the case that errors. main behaves the same, so it isn't a regression, but either drop the io.yaml-derived structured_outputs when the caller passes response_format, or scope this comment to llama-server and json_schema.

[ajbozarth]

Nothing in Mellea runs the backend's outgoing request through GraniteChatCompletion (rewriter.transform uses ChatCompletion, before the mirror), and the outside users I could find import granite-common's copy. So this relaxation has no caller. Can it be dropped, or what is it for?

Related but pre-existing: the kwargs-only branch below (line 498) assigns self.documents rather than self.extra_body.documents, so _documents() returns None and the documents are lost.

[ajbozarth]

LLAMACPP_SWITCH_PORT only applies to the script; the llama-server -hf command needs --port. Line 171 also already says both routes use 8081, so these two sentences could merge.

[ajbozarth]

nit: this says tested on v0.6.0 (line 28 too), while the docs and PR description say v0.6.0 and v0.9.0.

[ajbozarth]

nit: readers copying this example don't need the fix history. Something like # Both calls depend on the server actually rendering the documents. would do.