Skip to content

exllamav3: optional request-class admission gate (default off) - #521

Open
dfein38347g wants to merge 1 commit into
theroyallab:mainfrom
dfein38347g:lane-admission
Open

dfein38347g wants to merge 1 commit into
theroyallab:mainfrom
dfein38347g:lane-admission

Conversation

@dfein38347g

@dfein38347g dfein38347g commented Nov 15, 2026 •

Copy link
Copy Markdown

Is your pull request related to a problem? Please describe.

Yes. On a single-GPU exllamav3 server with a MoE model (experts streaming from
RAM), two concurrent long generations halve each other: they decode in the same
batch, and the expert stream to the GPU constantly thrashes as it switches
between the two hot expert sets. Meanwhile, short requests that share the model
(e.g. auxiliary tooling with a 60s timeout, fail-open) have to wait for a slot
while the long sessions occupy all of them.

This is not a queue-ordering problem: the scheduler skips jobs that don't
fit and proceeds to consider the ones that do, so a short job that fits in the
remaining cache launches alongside the long one. What the stock config cannot
express is per-class concurrency — max_batch_size is a total cap and
max_skips works in the other direction. There is no knob for "at most one
long job, plus one more slot for everything else".

I hit this with Qwen Flash Next EXL3 on one RTX 3090 (250k KV cache,
max_batch_size: 2): two ~100k-context agentic sessions thrashed the expert
stream, and a <5k-token tool request with a 60s timeout regularly missed its
deadline while the two long sessions held both slots.

Why should this feature be added?

It is the smallest mechanism that fixes both complaints without touching the
engine or replacing its scheduler:

  • Deployments that want it set one admission: block and a
    max_batch_size that has a spare slot.
  • Deployments that do not set it take the identical code path as today. The
    gate is constructed but disabled; its context manager never awaits and
    holds no state. There is no if enabled branch in the request path.
  • The config key is additive (Optional[Dict[str, Any]] = None), so pydantic
    exclude_none=True means no existing deployment's backend kwargs change.

It composes with the engine's existing batching rather than pre-empting jobs.
Held requests are never rejected; hold_timeout expiry warns and proceeds
(fail-open).

Examples

model:
  max_batch_size: 2          # two engine slots; one per lane in this shape
  admission:
    default: short
    classes:
      long:
        prompt_tokens_gte: 32768   # inclusive
        max_concurrent: 1
      short:
        max_concurrent: 1
    hold_timeout: 30         # seconds; warns and proceeds, never rejects
  • long requests (prompt ≥ 32768 tokens) serialise against each other, so the
    engine decodes one long session at a time.
  • The short lane keeps its own slot: a second long request that would fit
    in the residual cache does not start and squeeze a short one out of it.
  • A second long waits in the long FIFO; it is not rejected.
  • Requests with no declared max_tokens and no matching predicate take
    default.
  • Omitting admission: (or {}) disables the gate entirely.

A smaller live shape (10k boundary) is what I run on the 3090 today.

Additional context

How it works:

  • admission.py is a stdlib-only async gate. Requests are classified on the
    client's declared max_tokens (not the engine-resolved value, which
    auto-expands to the whole window) and prompt length.
  • Each class has a FIFO and a capacity. A released permit is handed directly
    to the next live waiter; a fail-open wake owns no permit, so the count can
    never exceed its limit.
  • The permit is held for the whole generate_gen body, including the
    AssertionError → ContextLengthExceededError path, so no exit can strand
    it. Client disconnect (DisconnectHandler.abort_event) abandons a queued
    hold without consuming a permit. wait_for_jobs releases all holds so an
    unload cannot hang on them.
  • Integration is five small points in model.py: import, the class attribute
    and its per-instance default, gate construction at load, abort_all on
    unload, and one async with around the existing generation block.
    config_models.py gets the field; without it, pydantic's default
    extra="ignore" drops the YAML key before the backend sees it.
  • admission.py is also covered by tests/test_admission.py (stdlib
    unittest, offline): disabled no-op, inclusive boundaries, FIFO per class,
    short-never-behind-long, hold_timeout fail-open, cancellation/disconnect
    release, abort_all, config validation, and a strict logger-signature test
    (tabbyAPI's XLogger takes a single message, not printf args).

Scope and known limits (so they are not filed as bugs):

  • The class is decided once, at admission. A request that starts short and
    grows long stays in its original class.
  • The gate serialises classes; it does not size the engine. It only helps if
    max_batch_size has a slot for each class you want to run concurrently. It
    warns once if the class capacities exceed max_batch_size.
  • It covers the chat/completions path (generate_gen). The
    explore_continuations / JobGroup path builds jobs directly and is not
    gated.
  • It is a config/tabby_config option. It is not exposed in the load-options UI
    or in /v1/model/load; I can add that in a follow-up if wanted.

Testing:

  • tests/test_admission.py: 67 tests, stdlib unittest, no model or GPU.
  • Verified live on one 3090 (Qwen Flash Next EXL3, max_batch_size: 2, one
    long + one short) through llama-swap; the gate is in the request path.
  • I have not published an A/B throughput table yet — the PR claims
    behaviour (two longs do not co-decode; a short job keeps its slot against a
    long-job pile-up), not a tok/s number. I can add measurements if you want
    them before merge.

Reviewer notes:

  • Review backends/exllamav3/model.py with git diff -w. The raw diff is a
    re-indent; the logical change is ~54 added lines.
  • admission.py is self-contained and has no tabbyAPI imports, so it can be
    reviewed and tested on its own.
  • Happy to split this into (1) the gate module + tests and (2) the model
    wiring if that is easier to review.

Add a config-gated, per-class concurrency gate in front of the exllamav3
request path. A deployment can split requests into classes (by prompt
length and the client-declared max_tokens) and cap each class separately,
so short queries are not blocked by a parked long session and two long
sessions do not co-decode on one GPU.

Omitting the admission block leaves the gate disabled: it is a no-op
context manager that never awaits and holds no state. The ModelConfig
field is additive and optional, so exclude_none=True keeps existing
backend kwargs unchanged.

A held request is never rejected; hold_timeout expiry warns and proceeds.
Cancellation, client disconnect, and model unload all release holds.

Covered by tests/test_admission.py (67 stdlib unittest cases, offline).
@turboderp

Copy link
Copy Markdown
Collaborator

This seems incorrect. Do you maybe not have output_chunking enabled in the config? A request wouldn't normally occupy the whole cache unless it's actually grown to use all of it. And until it does, other jobs are allowed to use the remainder of the cache. It's specifically not a FIFO queue.

Suppose you have a 128k cache, and three jobs come in, in order:

A: 100k token prompt
B: 80k token prompt
C: 2k token prompt

A will launch on the first iteration, reserving 102k tokens (prompt + one chunk). B will then be skipped since it doesn't fit in the cache. But rather than block there, the generator proceeds to consider C, which does fit as a 2k prompt + 2k initial response allocation.

So the two active jobs are now using 106k, and job B stays in the pending queue. Eventually A will reach the requeue boundary and restart itself as a new job with a 102k-token prompt. At this point it takes priority over B for performance reasons; processing B now would mean evicting some of A's context and prefilling again afterwards.

But until A has actually filled up the whole cache, there's still room for more short jobs. Unless you explicitly disable concurrency by setting max_batch_size: 1. But I don't know why you would do that. Note that max_skips only counts towards the number of times a job was actually skipped over by a shorter job that was behind it in the queue. It's there to prevent long jobs from getting delayed forever if short jobs keep coming in.

@dfein38347g

Copy link
Copy Markdown
Author

Yes, output_chunking was on — and you're right that short jobs don't queue behind long ones. Let me explain the case I'm actually solving.

I run Qwen Flash Next (MoE) on a 3090: experts streaming from RAM, 250k KV cache, max_batch_size: 2. The normal load is two agentic sessions at ~100k context each. When both are active, the engine runs them in the same batch and I can watch the expert stream to the GPU constantly thrash as it switches between the two hot expert sets — that's what halves both sessions.

On top of that, a memory-extraction/injection tool hits the same model with short requests (<5k tokens), 60s timeouts, fail-open. When the two 100k sessions occupy both slots, that small request has to wait for one of them to free up — the wait blows past 60s and the tool gives up.

So this isn't about queue ordering. It's two lanes with unequal context gates: at most one "long" job (a second long one waits in the queue) plus a separate slot for "short" jobs. With that, the engine decodes only one 100k session at a time, short requests still run concurrently, and almost all of my short queries come back inside the 60s timeout.

@dfein38347g

Copy link
Copy Markdown
Author

I've also rewritten the PR description along the lines of your point — the "cannot overtake" framing was a miscommunication on my side (the scheduler is skip-and-consider, not FIFO). The actual problem is slot occupancy: two 100k sessions holding both slots, plus MoE expert-stream thrashing while they co-decode. The description now says that and frames the ask as per-class concurrency rather than queue ordering.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants