Repository navigation
exllamav3: optional request-class admission gate (default off) - #521
dfein38347g wants to merge 1 commit into
Conversation
Add a config-gated, per-class concurrency gate in front of the exllamav3 request path. A deployment can split requests into classes (by prompt length and the client-declared max_tokens) and cap each class separately, so short queries are not blocked by a parked long session and two long sessions do not co-decode on one GPU. Omitting the admission block leaves the gate disabled: it is a no-op context manager that never awaits and holds no state. The ModelConfig field is additive and optional, so exclude_none=True keeps existing backend kwargs unchanged. A held request is never rejected; hold_timeout expiry warns and proceeds. Cancellation, client disconnect, and model unload all release holds. Covered by tests/test_admission.py (67 stdlib unittest cases, offline).
653bf30 to
6cadf70
Compare
|
This seems incorrect. Do you maybe not have Suppose you have a 128k cache, and three jobs come in, in order: A: 100k token prompt A will launch on the first iteration, reserving 102k tokens (prompt + one chunk). B will then be skipped since it doesn't fit in the cache. But rather than block there, the generator proceeds to consider C, which does fit as a 2k prompt + 2k initial response allocation. So the two active jobs are now using 106k, and job B stays in the pending queue. Eventually A will reach the requeue boundary and restart itself as a new job with a 102k-token prompt. At this point it takes priority over B for performance reasons; processing B now would mean evicting some of A's context and prefilling again afterwards. But until A has actually filled up the whole cache, there's still room for more short jobs. Unless you explicitly disable concurrency by setting |
|
Yes, output_chunking was on — and you're right that short jobs don't queue behind long ones. Let me explain the case I'm actually solving. I run Qwen Flash Next (MoE) on a 3090: experts streaming from RAM, 250k KV cache, On top of that, a memory-extraction/injection tool hits the same model with short requests (<5k tokens), 60s timeouts, fail-open. When the two 100k sessions occupy both slots, that small request has to wait for one of them to free up — the wait blows past 60s and the tool gives up. So this isn't about queue ordering. It's two lanes with unequal context gates: at most one "long" job (a second long one waits in the queue) plus a separate slot for "short" jobs. With that, the engine decodes only one 100k session at a time, short requests still run concurrently, and almost all of my short queries come back inside the 60s timeout. |
|
I've also rewritten the PR description along the lines of your point — the "cannot overtake" framing was a miscommunication on my side (the scheduler is skip-and-consider, not FIFO). The actual problem is slot occupancy: two 100k sessions holding both slots, plus MoE expert-stream thrashing while they co-decode. The description now says that and frames the ask as per-class concurrency rather than queue ordering. |
Is your pull request related to a problem? Please describe.
Yes. On a single-GPU exllamav3 server with a MoE model (experts streaming from
RAM), two concurrent long generations halve each other: they decode in the same
batch, and the expert stream to the GPU constantly thrashes as it switches
between the two hot expert sets. Meanwhile, short requests that share the model
(e.g. auxiliary tooling with a 60s timeout, fail-open) have to wait for a slot
while the long sessions occupy all of them.
This is not a queue-ordering problem: the scheduler skips jobs that don't
fit and proceeds to consider the ones that do, so a short job that fits in the
remaining cache launches alongside the long one. What the stock config cannot
express is per-class concurrency —
max_batch_sizeis a total cap andmax_skipsworks in the other direction. There is no knob for "at most onelong job, plus one more slot for everything else".
I hit this with Qwen Flash Next EXL3 on one RTX 3090 (250k KV cache,
max_batch_size: 2): two ~100k-context agentic sessions thrashed the expertstream, and a <5k-token tool request with a 60s timeout regularly missed its
deadline while the two long sessions held both slots.
Why should this feature be added?
It is the smallest mechanism that fixes both complaints without touching the
engine or replacing its scheduler:
admission:block and amax_batch_sizethat has a spare slot.gate is constructed but disabled; its context manager never awaits and
holds no state. There is no
if enabledbranch in the request path.Optional[Dict[str, Any]] = None), so pydanticexclude_none=Truemeans no existing deployment's backend kwargs change.It composes with the engine's existing batching rather than pre-empting jobs.
Held requests are never rejected;
hold_timeoutexpiry warns and proceeds(fail-open).
Examples
longrequests (prompt ≥ 32768 tokens) serialise against each other, so theengine decodes one long session at a time.
shortlane keeps its own slot: a secondlongrequest that would fitin the residual cache does not start and squeeze a
shortone out of it.longwaits in thelongFIFO; it is not rejected.max_tokensand no matching predicate takedefault.admission:(or{}) disables the gate entirely.A smaller live shape (10k boundary) is what I run on the 3090 today.
Additional context
How it works:
admission.pyis a stdlib-only async gate. Requests are classified on theclient's declared
max_tokens(not the engine-resolved value, whichauto-expands to the whole window) and prompt length.
to the next live waiter; a fail-open wake owns no permit, so the count can
never exceed its limit.
generate_genbody, including theAssertionError→ContextLengthExceededErrorpath, so no exit can strandit. Client disconnect (
DisconnectHandler.abort_event) abandons a queuedhold without consuming a permit.
wait_for_jobsreleases all holds so anunload cannot hang on them.
model.py: import, the class attributeand its per-instance default, gate construction at load,
abort_allonunload, and one
async witharound the existing generation block.config_models.pygets the field; without it, pydantic's defaultextra="ignore"drops the YAML key before the backend sees it.admission.pyis also covered bytests/test_admission.py(stdlibunittest, offline): disabled no-op, inclusive boundaries, FIFO per class,short-never-behind-long,
hold_timeoutfail-open, cancellation/disconnectrelease,
abort_all, config validation, and a strict logger-signature test(tabbyAPI's
XLoggertakes a single message, not printf args).Scope and known limits (so they are not filed as bugs):
grows long stays in its original class.
max_batch_sizehas a slot for each class you want to run concurrently. Itwarns once if the class capacities exceed
max_batch_size.generate_gen). Theexplore_continuations/JobGrouppath builds jobs directly and is notgated.
or in
/v1/model/load; I can add that in a follow-up if wanted.Testing:
tests/test_admission.py: 67 tests, stdlibunittest, no model or GPU.max_batch_size: 2, onelong + one short) through llama-swap; the gate is in the request path.
behaviour (two longs do not co-decode; a short job keeps its slot against a
long-job pile-up), not a tok/s number. I can add measurements if you want
them before merge.
Reviewer notes:
backends/exllamav3/model.pywithgit diff -w. The raw diff is are-indent; the logical change is ~54 added lines.
admission.pyis self-contained and has no tabbyAPI imports, so it can bereviewed and tested on its own.
wiring if that is easier to review.