@@ -21,6 +21,8 @@ specific language governing permissions and limitations under the License.
2121| [ ` env ` ] ( #env ) | Print environment info for bug reports. |
2222| [ ` schema ` ] ( #schema ) | Inspect a pipeline's ` __call__ ` signature without downloading weights. |
2323| [ ` run ` ] ( #run ) | Run a pipeline locally or in a Hugging Face Sandbox. |
24+ | [ ` serve ` ] ( #serve ) | Serve a pipeline over HTTP with the Generative Media Spec API. |
25+ | [ ` generate ` ] ( #generate ) | Send a generation to a Generative Media Spec server and save its outputs. |
2426| [ ` custom_blocks ` ] ( #customblocks ) | Package a local ` ModularPipelineBlocks ` subclass for the Hub. |
2527| [ ` fp16_safetensors ` ] ( #fp16safetensors ) | Convert a checkpoint to fp16 ` .safetensors ` . |
2628| [ ` skills ` ] ( #skills ) | Install pre-authored skill bundles into your AI coding agent. |
@@ -287,6 +289,148 @@ diffusers-cli run -m black-forest-labs/FLUX.1-dev --dtype bf16 \
287289hf sandbox kill < id>
288290```
289291
292+ ## ` serve `
293+
294+ Serve a pipeline over HTTP. The server implements the Generative Media Spec (GMS, draft ` gms/0.1 ` ): clients
295+ discover what the model accepts, create a generation, follow it while it runs, and read the results. It needs
296+ the ` serve ` extra:
297+
298+ ``` bash
299+ pip install " diffusers[serve]"
300+ diffusers-cli serve --model black-forest-labs/FLUX.1-dev --dtype bf16
301+ ```
302+
303+ ` serve ` takes the same loading and optimization flags as [ ` run ` ] ( #run ) (` --dtype ` , ` --device-map ` ,
304+ ` --cpu-offload ` , ` --compile ` , ` --lora ` , ...), plus ` --host ` (default ` 127.0.0.1 ` ) and ` --port ` (default ` 8000 ` ).
305+ The server has no authentication, so keep it on a trusted network.
306+
307+ [ Modular repos] ( ../modular_diffusers/overview ) are detected the same way as in ` run ` and served in-process. They
308+ report no step progress, and a cancel request only takes effect once the pipeline call returns.
309+
310+ | Flag | Effect |
311+ | ---| ---|
312+ | ` --default-seed ` | Seed used for requests that do not set one. Without it, each such request gets a random seed. |
313+ | ` --enable-cors ` | Allow cross-origin requests from any origin, so a browser app on another host can call the server. |
314+ | ` --log-level ` | Logging level for diffusers and the HTTP server (` debug ` , ` info ` , ` warning ` , ` error ` , ` critical ` ). |
315+
316+ ` GET /health ` returns ` {"status": "ok"} ` , or ` 503 ` when an SGLang or vLLM-Omni backend has exited. Every response
317+ carries an ` x-request-id ` header: the one sent with the request, or a generated one.
318+
319+ Generations run one at a time. Creating one returns immediately with a ` queued ` generation; poll it, subscribe
320+ to its events, or ask the server to hold the connection with ` Prefer: wait=<seconds> ` :
321+
322+ ``` bash
323+ # What the model accepts: tasks, inputs, parameters with types and defaults.
324+ curl localhost:8000/v1/models/black-forest-labs/FLUX.1-dev
325+
326+ # Create a generation and wait up to 120 seconds for it.
327+ curl localhost:8000/v1/generations -H ' Prefer: wait=120' -d ' {
328+ "model": "black-forest-labs/FLUX.1-dev",
329+ "inputs": {"prompt": "an astronaut riding a horse"},
330+ "parameters": {"width": 1024, "height": 1024, "steps": 28, "seed": 42}
331+ }'
332+
333+ # Follow step progress as server-sent events, or poll the generation.
334+ curl -N localhost:8000/v1/generations/< id> /events
335+ curl localhost:8000/v1/generations/< id>
336+
337+ # Cancel a queued or running generation, or delete it together with its files.
338+ curl -X POST localhost:8000/v1/generations/< id> /cancel
339+ curl -X DELETE localhost:8000/v1/generations/< id>
340+ ```
341+
342+ A completed generation lists its outputs with a download URL each (or inline base64 with
343+ ` "response_format": "b64_json" ` ), the seed that was used, and the model revision. Outputs are kept under
344+ ` --output-dir ` and expire after one hour. Images are PNG by default (` "output_format": "image/jpeg" ` or
345+ ` "image/webp" ` to change it), videos MP4, audio WAV. Media inputs are passed as
346+ ` {"type": "image", "url": "https://..."} ` or ` {"type": "image", "base64": "...", "media_type": "image/png"} ` .
347+
348+ Validation is strict: an unknown parameter, a value outside its declared range, or a missing input is rejected
349+ with a JSON pointer to the offending field.
350+
351+ ``` json
352+ {"error" : {"code" : " invalid_request" , "message" : " task 'text_to_image' declares no parameter 'cfg'" , "pointer" : " /parameters/cfg" }}
353+ ```
354+
355+ ### Manifests
356+
357+ What a model accepts is described by its manifest. ` serve ` looks for one in this order:
358+
359+ 1 . the file passed with ` --manifest <path> ` ;
360+ 2 . a ` gms.json ` published in the model repo, served byte-for-byte;
361+ 3 . a manifest derived from the pipeline's ` __call__ ` signature.
362+
363+ A derived manifest is a best-effort reading of the signature: scalar arguments become parameters, prompts and
364+ media arguments become inputs, and arguments with a core GMS name are renamed (` num_inference_steps ` becomes
365+ ` steps ` , ` num_images_per_prompt ` becomes ` num_outputs ` , ` generator ` becomes ` seed ` ). It carries no value ranges.
366+ Write a ` gms.json ` when you need ranges, presets, or a duration in seconds.
367+
368+ When a manifest uses a name that differs from the pipeline argument, map it per task under ` x-diffusers ` :
369+
370+ ``` json
371+ "x-diffusers" : {"inputs" : {"caption" : " prompt" }, "parameters" : {"cfg" : " guidance_scale" }}
372+ ```
373+
374+ A task is only hosted when every input it requires maps to a pipeline argument. A parameter the pipeline cannot
375+ set is rejected with ` not_implemented ` unless the request sends its default value.
376+
377+ ### Adapters
378+
379+ A request can apply LoRAs for that generation only. They are loaded before the call and unloaded after it:
380+
381+ ``` bash
382+ curl localhost:8000/v1/generations -d ' {
383+ "model": "black-forest-labs/FLUX.1-dev",
384+ "inputs": {"prompt": "a tiny cat"},
385+ "adapters": [{"type": "lora", "path": "hf:alvdansen/littletinies", "scale": 0.8}]
386+ }'
387+ ```
388+
389+ Request adapters are turned off when the server starts with ` --compile ` or ` --lora ` ; such requests fail with
390+ ` not_implemented ` .
391+
392+ ### SGLang and vLLM-Omni backends
393+
394+ ` --backend sglang ` and ` --backend vllm ` run the model in [ SGLang Diffusion] ( https://docs.sglang.io/ ) or
395+ [ vLLM-Omni] ( https://docs.vllm.ai/projects/vllm-omni ) instead of in-process. ` serve ` starts ` sglang serve ` or
396+ ` vllm serve --omni ` as a child process on a local port, waits for it to become healthy, and translates GMS
397+ requests to the engine's OpenAI-compatible image and video routes. Clients only ever talk to the GMS endpoints
398+ of ` serve ` . The engine must be installed separately.
399+
400+ ``` bash
401+ diffusers-cli serve --model Qwen/Qwen-Image --backend sglang --backend-args ' --num-gpus 2'
402+ diffusers-cli serve --model Wan-AI/Wan2.2-T2V-A14B-Diffusers --backend vllm
403+ ```
404+
405+ The pipeline flags (` --dtype ` , ` --cpu-offload ` , ` --compile ` , ...) do not apply to these backends; pass the
406+ engine's own flags with ` --backend-args ` . Compared with the in-process backend, they serve image and video
407+ outputs only, accept ` image ` as the only media input, expose the parameters both engines share (size, steps,
408+ guidance scale, seed, number of frames, fps, number of images), report no step progress, and apply no request
409+ adapters.
410+
411+ ## ` generate `
412+
413+ Send a generation to a server started with [ ` serve ` ] ( #serve ) , or to any other Generative Media Spec server, and
414+ save its outputs. Unlike [ ` run ` ] ( #run ) , ` generate ` loads no pipeline: it reads the model's discovery document,
415+ submits the request, polls the generation until it finishes, and downloads the files.
416+
417+ ``` bash
418+ diffusers-cli generate --url http://localhost:8000 --inputs ' {"prompt": "a cat on the moon"}'
419+
420+ diffusers-cli generate --url http://localhost:8000 \
421+ --inputs ' {"prompt": "make the fur grey", "image": "cat.png"}' \
422+ --parameters ' {"steps": 8, "seed": 0}' --output outputs/
423+ ```
424+
425+ ` --inputs ` and ` --parameters ` use the names the server declares, not pipeline argument names (` steps ` , not
426+ ` num_inference_steps ` ). An image, mask, video or audio input can be an http(s) URL or a local file path; local
427+ files are sent inline as base64. ` --task ` , ` --preset ` , ` --adapters ` and ` --output-format ` map to the request
428+ fields of the same name. ` --model ` is only needed when the server hosts more than one model.
429+
430+ Outputs are saved to ` ~/.diffusers/cli/generate/outputs/<generation id>/ ` unless ` --output ` names a directory.
431+ ` --token ` sends a bearer token in the ` Authorization ` header, for servers behind an authenticating proxy; no
432+ token is sent by default. Interrupting the command cancels the generation on the server.
433+
290434## ` custom_blocks `
291435
292436Package a local ` ModularPipelineBlocks ` subclass for upload to the Hub. Reads a Python file, AST-scans it for
0 commit comments