models/llm: Qwen3-0.6B CoreML — LLM-on-ANE prefill/decode test

[Alex-Wengg]

First llm-class conversion in mobius. Empirically tests the LLM-on-ANE thesis from the research lead in #76 (knowledge/coreml/ane-cpu-scheduled-matmul.md), on M5 Pro with standalone Qwen/Qwen3-0.6B.

Result: the prefill/decode ANE split is real

The two graphs divide exactly as Surgical Inference §6.3 predicts — the ANE admission cliff is in-graph state mutation, not the transformer math.

Graph	Compute units	Outcome	Latency
Decode (stateful, in-graph KV mutation)	CPU_AND_NE	ANE REJECTED — ANECCompile -14	—
Decode	CPU_AND_GPU	OK, coherent output	27 tok/s, RTFx 1.80×
Prefill (stateless, seq=128)	CPU_AND_NE	ANE ACCEPTED — 1918/1919 device-assigned ops on the Neural Engine	12.5 ms/128 tok
Prefill	CPU_ONLY	OK	46.0 ms (ANE ≈ 4× over CPU)
Prefill	CPU_AND_GPU	OK	9.3 ms

Bottom line: an LLM can run on the ANE — for prefill (compute-bound, cache-free, Dense-Static). Decode's KV-cache mutation is ANE-rejected and belongs on GPU/CPU, where it is comfortably real-time (RTFx 1.80× > 1). This is the split-phase placement the note predicts.

Nuance: on this M5 Pro the (strong) GPU is marginally ahead of the ANE for prefill via standard Core ML (9.3 vs 12.5 ms). The reviewer's "ANE beats GPU on prefill" claim rests on the private CPU-scheduled matmul path, which this standard export does not use — untested here.

Contents

• convert-coreml.py — stateful decode decoder (KV cache as MLState), adapted from the in-repo stt/qwen3-asr-0.6b graph.
• convert-prefill.py — stateless prefill graph (fixed seq len, no state) — the ANE-positive case.
• benchmark.py — decode tok/s + RTFx (vs 15 tok/s real-time drain).
• probe-placement.py — ANE-admission + MLComputePlan per-op device breakdown.
• Build artifacts (1.1 GB .mlpackage) gitignored; not HF-uploaded.

Notes

• Validates the knowledge(coreml): ANE CPU-scheduled matmul research lead #76 knowledge note with real device data; merge order doesn't matter (this branch references the note, doesn't edit it).
• coreml-cli couldn't profile this package (its Python 3.14 venv fails to load libmodelpackage) — placement was obtained via MLComputePlan in the working 3.11 env instead. Possible follow-up to fix the tool's env.