Equivalence test with power: llama.cpp vs Ollama on the same weights #1590

Follow-up to #1586 (results: evals/runtime_screen/RESULTS.md, PR #1589).

Why

llama.cpp was faster than Ollama on the same GGUF weights: prefill 1.21x, and decode 1.11x on qwen3:8b and 1.38x on gemma4. It was still not adopted, because #1586's output-equivalence rule failed on one guardrail:

• Sufficiency: equivalent (qwen3:8b +0.013 [−0.012, +0.044]; gemma4 delta 0.000).
• evals: choose the concept-extraction model with an annotation-free, fast screen #1557 replay coverage lift: unchanged (+0.0000).
• G1 precision: point −0.010, CI [−0.185, +0.113] against a −0.05 floor. This is an underpowered interval, not evidence of a loss.

Since the weights are byte-identical, any difference comes from the engine (sampling, kernels, KV-cache quantization). That is testable directly.

Proposal (pre-register before any run)

1. Equivalence with power. TOST with a margin fixed in advance on:

   • the coverage lift;
   • sufficiency accuracy;
   • G1, or a direct output-agreement metric that does not need gold precision judgments, such as per-window object-set Jaccard against Ollama at temperature 0.

   Size the sample (meetings × repeats) so the expected CI width is under 2× the margin. Never touch the thesis evaluation series or their reserves.

2. End-to-end benefit. Measure the wall time for ingesting one meeting and for answering one Ask question, Ollama vs llama.cpp, on the 32 GB floor emulation. Pre-register a minimum gain (proposed: ≥15 % less total time). Otherwise the speedup does not justify the switch.

3. Delivery, only if 1 and 2 pass:

   • document llama.cpp as a supported opt-in through backend: openai-compatible (no code change);
   • a separate ADR, superseding ADR-0057, decides whether the desktop app should bundle it as the default. That ADR is for the maintainer to merge.

Out of scope

MLX. Its gemma4 sufficiency lost (−0.019 [−0.044, −0.002]) on a different 4-bit build without KV quantization. Revisit when the 32 GB co-residency matters (both models plus bge-m3 fit in 23.56 GiB), or when a closer quantization is available.