Skip to content

Measure ADR-0076's sufficiency model swap on the 32 GB floor #1587

Closed

Description

@jasonssdev

Part of #1556. Refs #1560, #1585, ADR-0076.

Problem

The PR #1585 bar passed only on the 48 GB development machine, where qwen3:8b and gemma4:26b-a4b stayed resident together. On the 32 GB product floor (24 GB budget) they do not fit together: the Ollama server log puts the pair at about 24.1 GiB. So every query there would swap the two models twice (check, then answer). That cost is unmeasured, and the bar written for #1585 did not separate the host machine from the floor.

Measurement

Repeat the #1585 protocol, which is 3 interleaved sequences per arm of one cold question then 5 warm, with one loaded model enforced. Use a second Ollama server bound to another port with OLLAMA_MAX_LOADED_MODELS=1 and the same model store, and do not touch the user's server. Apply the same bar: warm median at most 1.5x of arm (a), and the cold first question at most 30 s.

Done when

Activity

  1. jasonssdev commented on Oct 20, 2026

    @jasonssdev
    OwnerAuthor

    Recorded in #1589 (evals/runtime_screen/RESULTS.md). ADR-0076's bar passes with one chat model resident, so no superseding ADR.

    Configuration A: a second Ollama server with OLLAMA_MAX_LOADED_MODELS=1, where only qwen3:8b stayed resident after each question.

    • Cold first question: 23.88 s against 22.96 s for arm (a); the bar is 30 s.
    • Warm median: 18.58 s against 19.34 s, a ratio of 0.96; the bar is 1.5.

    Configuration B, a 24 GB budget through OLLAMA_GPU_OVERHEAD, is void. Ollama never evicted: all three models stayed resident after 17 of 18 arm-(b) questions, because ollama ps puts gemma4 at 985 MB.

    As an estimate, not a measurement: with bge-m3 resident on the floor, arm (b) would pay two swaps of about 3.3 s plus 1.1 s, giving roughly 16.4 / 14.24 = 1.15. That is still under 1.5, provided the floor's page cache keeps both weight files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions