Repository navigation
Moreh features on v1.5.0 - #4
Merged
Merged
Conversation
…ernetes-sigs#37) Adapted for v1.5 (upstream df9f890): - Keep ITL/TTFT/TPOT Histogram + Gauge definitions (metrics.go). - Move per-token ITL measurement into the predictedlatency plugin's per-token hook (processTokenForLatencyPrediction); RecordRequestITL is called from latencypredictor_helper.go instead of handlers/server.go because upstream's Pluggable Parser Framework (00edcef) replaced the SSE parsing that used to live in the handler. - Move per-request average ITL gauge flush into the predictedlatency plugin's EndOfStream path (requestcontrol_hooks.go). - TTFT/TPOT base metrics are recorded by upstream's handler path, so no direct changes to handlers are needed here. Signed-off-by: HyunKyun Moon <mhg5303@gmail.com>
…API test case (kubernetes-sigs#39) Adapted for v1.5 (upstream df9f890): - SSE newline fix applied to the existing "response streaming: SSE token counting" test case in hermetic_test.go (upstream kept this test inline — it was not moved to common_tests.go). - Added new "response streaming: Responses API token counting" test case after the SSE test. Exercises the Responses API streaming path (event: response.output_text.delta / event: response.completed). - harness.go 1-line format-string fix from the original commit is intentionally excluded per maintainer direction. Signed-off-by: HyunKyun Moon <mhg5303@gmail.com>
…sigs#40) Add inference_objective_output_tokens histogram assertion to both streaming test cases in hermetic_test.go: - "response streaming: SSE token counting": sum=10 (Chat Completions). - "response streaming: Responses API token counting": sum=20 (Responses API output_tokens=20). Mirrors the assertion already present in grpc_test.go for the gRPC streaming path. No upstream-adaptation changes needed. Signed-off-by: HyunKyun Moon <mhg5303@gmail.com>
…able SD selection (kubernetes-sigs#47) Adapted for v1.5 (upstream df9f890): - Upstream 5bdd01d promoted SaturationDetector to a pluggable extension point (fwkflowcontrol.SaturationDetector interface). - RunningRequestsDetector moved from pkg/epp/saturationdetector/framework/plugins/runningrequestsdetector/ to pkg/epp/framework/plugins/flowcontrol/saturationdetector/runningrequests/ to follow the new plugin convention. - Rewrote as fwkplugin.Plugin implementing Saturation(ctx, []datalayer.Endpoint) float64 (the new interface). - Registered the plugin with RunningRequestsDetectorType "running-requests-detector" via fwkplugin.Register in runner.go, alongside utilization and concurrency detectors. - The original config schema (SaturationDetector.Type switch), configloader buildSaturationDetectorType/buildRunningRequestsDetectorConfig functions, and runner switch block are not needed — upstream's registry-based plugin loading via pluginRef handles all detectors uniformly. - To select this detector, set spec.parameters in the plugin config (maxConcurrencyPerPod, cacheTTL) and reference by pluginRef. Signed-off-by: HyunKyun Moon <mhg5303@gmail.com>
Squashed for v1.5 (upstream df9f890) — combines PR #3 commits b35264e + d24a2d9 + b1ecd43 + febaa7c into a single commit because upstream's datalayer plugins-runtime separation (dc2f1fe) and race fix (a1f0b01) reshape the files these four commits touched, making a clean per-commit split impractical. What this commit delivers: - PoolInfo.EndpointSetHealthy(ep, healthy bool) interface added (pkg/epp/datalayer/factory.go) so the collector can report per-endpoint health. Datastore implements it to flip a pod in/out of the PodList filter. - Collector rewrite: - Added poolInfo field + NewCollector(poolInfo) signature. - Aggregates poll errors with errors.Join across sources and calls EndpointSetHealthy(endpoint, pollErrs == nil) after each tick. - Keeps upstream's lastPollErrors/lastExtractErrors change-only logging + extractor dispatch. - Runtime.NewEndpoint forwards poolInfo to NewCollector (pkg/epp/datalayer/runtime.go). - Datastore adds unhealthyEndpoints sync.Map to track endpoints marked unhealthy. PodList filters them out; Clear/PodDelete/podResyncAll remove them alongside the pods map. Recovery flips them back automatically when the next successful poll arrives. - Backend metrics refresh loop (legacy path) reports health via EndpointSetHealthy when FetchMetrics succeeds/fails. - PodMetricsFactory tracks active endpoints by NamespacedName via a sync.Map and returns nil from NewEndpoint when a collector is already running for that endpoint — prevents a pod-update event from re-spawning a goroutine for an endpoint whose existing collector is still handling recovery. This subsumes d24a2d9's "prevent re-add on pod update" guard. - FakeDataSource Poll-error propagation migrated into source/mocks/data_source_mock.go: when SetErrors is set for an endpoint, Poll returns that error so the health-check path triggers. - All downstream test stubs (FakeOddMetricsDataStore, FakeRefresherDataStore, datalayer/logger_test stubs) implement the new EndpointSetHealthy no-op method. Signed-off-by: HyunKyun Moon <mhg5303@gmail.com>
…ubernetes-sigs#41) Adapted for v1.5 (upstream df9f890): - PR kubernetes-sigs#41 (MAF-19473) added RawBody/ResponseBodyJSON/CurrentStreamingBody/ PrepareDataPlugin 404-error-propagation + Director Responses dispatch + plugin_executor hook. These are all superseded by upstream's Pluggable Parser Framework (00edcef via LLMRequestBody.Body / ParsedBody) and are therefore intentionally dropped in this merge. - However, upstream's ResponsesRequest does NOT include a Store field. The downstream scheduler's ResponsesStorePlugin reads body.Responses.Store to decide whether to persist the response. Preserving just this one field as a Moreh-local diff on the upstream type keeps the plugin working without further changes downstream. - Consider proposing this field to upstream in a follow-up so the next merge has one less local delta. Signed-off-by: HyunKyun Moon <mhg5303@gmail.com>
Add AudioURL (audio_url) multimodal content block type alongside existing InputAudio (input_audio). vLLM 0.18+ strictly validates audio_url via pydantic ChatCompletionContentPartAudioParam, rejecting requests that carry an audio_url type without the matching field.
bongwoobak
reviewed
Apr 27, 2026
| typeTTFTSLOViolation = "ttft_slo_violation" | ||
| typeTTFTSLOThreshold = "ttft_slo_threshold" // Moreh MAF-19466 | ||
|
|
||
| typeITL = "itl" // Moreh 5701fa97 |
Member
There was a problem hiding this comment.
(Leaving this here as there’s no other place to put it)
It seems the part for collecting ITL when the predictedlatency plugin is disabled in hie#37 (per-token measurement in HandleResponseBodyModelStreaming of pkg/epp/handlers/response.go + EndOfStream aggregation in pkg/epp/handlers/server.go) is missing.
…ing (kubernetes-sigs#37)" This reverts commit 7d36f1c.
Signed-off-by: HyunKyun Moon <mhg5303@gmail.com>
…tdata/inferencepool-e2e.yaml that strict CRD decoding rejects Signed-off-by: HyunKyun Moon <mhg5303@gmail.com>
hyeongyun0916
force-pushed
the
merge-150
branch
from
April 28, 2026 07:47
036f38c to
a444834
Compare
nulledge
approved these changes
Apr 29, 2026
| ImageURL ImageBlock `json:"image_url"` | ||
| InputAudio AudioBlock `json:"input_audio"` | ||
| AudioURL AudioURLBlock `json:"audio_url"` | ||
| VideoURL VideoBlock `json:"video_url"` |
There was a problem hiding this comment.
FYI) vLLM v0.17.* requires the audio_url field when contentType=audio_url.
Without this change, the multi-modal test was failed.
Also, llm-d-kv-cache should support audio_url [Link].
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
merge-150 commits (top of df9f890, oldest → newest)
harness.go)Storefield preservedHandleResponseBodyso they work without the predictedlatency plugin. Addsrequest_itl_secondsHistogram (only the gauge existed in row 1).spec.endpointPickerRef.namespacefromtest/testdata/inferencepool-e2e.yaml— strict CRD decoding rejects it and was preventingmake test-e2e's BeforeSuite from creating the InferencePoolSkipped commits
[DONE]detectionOutcome distribution
Storefield preserved)11 source commits → 10 merge-150 commits (7 ports + 1 revert + 2 review fixes).