Skip to content

Commit 260ae4a

Browse files
committed
add diffusers serve and generate commands
1 parent ae2e4c7 commit 260ae4a

13 files changed

Lines changed: 3166 additions & 1 deletion

File tree

‎.ai/skills/diffusers-cli/SKILL.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@ description: >
1616
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
1717
| `run` | Run any `DiffusionPipeline` or `ModularPipeline`. Forwards `--pipeline-kwargs` verbatim, saves output by detecting its runtime type, optionally runs on HF Jobs via `--remote`. |
1818
| `schema` | Print the input schema for a pipeline repo (kwarg names, types, defaults, descriptions). **No weights downloaded** — only the small index file. |
19+
| `serve` | Serve a pipeline over HTTP with the Generative Media Spec API: model discovery, queued generations, polling or SSE progress. In-process, or behind SGLang / vLLM-Omni via `--backend`. |
20+
| `generate` | Send a request to a Generative Media Spec server (such as `serve`) and save the outputs. Loads no pipeline; takes `--url`, `--inputs` and `--parameters` with the server's declared names. |
1921
| `custom_blocks` | Package a local `ModularPipelineBlocks` subclass for the Hub. |
2022
| `env` | Print versions of diffusers + torch + transformers + accelerate + safetensors + CUDA + GPU info. Use when investigating environment issues, dtype/precision support, or building bug reports. |
2123

@@ -33,6 +35,8 @@ The other commands are small enough that `diffusers-cli <command> --help` is the
3335

3436
```bash
3537
diffusers-cli schema --help
38+
diffusers-cli serve --help
39+
diffusers-cli generate --help
3640
diffusers-cli custom_blocks --help
3741
diffusers-cli env --help
3842
```

‎docs/source/en/using-diffusers/cli.md‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ specific language governing permissions and limitations under the License.
2121
| [`env`](#env) | Print environment info for bug reports. |
2222
| [`schema`](#schema) | Inspect a pipeline's `__call__` signature without downloading weights. |
2323
| [`run`](#run) | Run a pipeline locally or in a Hugging Face Sandbox. |
24+
| [`serve`](#serve) | Serve a pipeline over HTTP with the Generative Media Spec API. |
25+
| [`generate`](#generate) | Send a generation to a Generative Media Spec server and save its outputs. |
2426
| [`custom_blocks`](#customblocks) | Package a local `ModularPipelineBlocks` subclass for the Hub. |
2527
| [`fp16_safetensors`](#fp16safetensors) | Convert a checkpoint to fp16 `.safetensors`. |
2628
| [`skills`](#skills) | Install pre-authored skill bundles into your AI coding agent. |
@@ -287,6 +289,148 @@ diffusers-cli run -m black-forest-labs/FLUX.1-dev --dtype bf16 \
287289
hf sandbox kill <id>
288290
```
289291

292+
## `serve`
293+
294+
Serve a pipeline over HTTP. The server implements the Generative Media Spec (GMS, draft `gms/0.1`): clients
295+
discover what the model accepts, create a generation, follow it while it runs, and read the results. It needs
296+
the `serve` extra:
297+
298+
```bash
299+
pip install "diffusers[serve]"
300+
diffusers-cli serve --model black-forest-labs/FLUX.1-dev --dtype bf16
301+
```
302+
303+
`serve` takes the same loading and optimization flags as [`run`](#run) (`--dtype`, `--device-map`,
304+
`--cpu-offload`, `--compile`, `--lora`, ...), plus `--host` (default `127.0.0.1`) and `--port` (default `8000`).
305+
The server has no authentication, so keep it on a trusted network.
306+
307+
[Modular repos](../modular_diffusers/overview) are detected the same way as in `run` and served in-process. They
308+
report no step progress, and a cancel request only takes effect once the pipeline call returns.
309+
310+
| Flag | Effect |
311+
|---|---|
312+
| `--default-seed` | Seed used for requests that do not set one. Without it, each such request gets a random seed. |
313+
| `--enable-cors` | Allow cross-origin requests from any origin, so a browser app on another host can call the server. |
314+
| `--log-level` | Logging level for diffusers and the HTTP server (`debug`, `info`, `warning`, `error`, `critical`). |
315+
316+
`GET /health` returns `{"status": "ok"}`, or `503` when an SGLang or vLLM-Omni backend has exited. Every response
317+
carries an `x-request-id` header: the one sent with the request, or a generated one.
318+
319+
Generations run one at a time. Creating one returns immediately with a `queued` generation; poll it, subscribe
320+
to its events, or ask the server to hold the connection with `Prefer: wait=<seconds>`:
321+
322+
```bash
323+
# What the model accepts: tasks, inputs, parameters with types and defaults.
324+
curl localhost:8000/v1/models/black-forest-labs/FLUX.1-dev
325+
326+
# Create a generation and wait up to 120 seconds for it.
327+
curl localhost:8000/v1/generations -H 'Prefer: wait=120' -d '{
328+
"model": "black-forest-labs/FLUX.1-dev",
329+
"inputs": {"prompt": "an astronaut riding a horse"},
330+
"parameters": {"width": 1024, "height": 1024, "steps": 28, "seed": 42}
331+
}'
332+
333+
# Follow step progress as server-sent events, or poll the generation.
334+
curl -N localhost:8000/v1/generations/<id>/events
335+
curl localhost:8000/v1/generations/<id>
336+
337+
# Cancel a queued or running generation, or delete it together with its files.
338+
curl -X POST localhost:8000/v1/generations/<id>/cancel
339+
curl -X DELETE localhost:8000/v1/generations/<id>
340+
```
341+
342+
A completed generation lists its outputs with a download URL each (or inline base64 with
343+
`"response_format": "b64_json"`), the seed that was used, and the model revision. Outputs are kept under
344+
`--output-dir` and expire after one hour. Images are PNG by default (`"output_format": "image/jpeg"` or
345+
`"image/webp"` to change it), videos MP4, audio WAV. Media inputs are passed as
346+
`{"type": "image", "url": "https://..."}` or `{"type": "image", "base64": "...", "media_type": "image/png"}`.
347+
348+
Validation is strict: an unknown parameter, a value outside its declared range, or a missing input is rejected
349+
with a JSON pointer to the offending field.
350+
351+
```json
352+
{"error": {"code": "invalid_request", "message": "task 'text_to_image' declares no parameter 'cfg'", "pointer": "/parameters/cfg"}}
353+
```
354+
355+
### Manifests
356+
357+
What a model accepts is described by its manifest. `serve` looks for one in this order:
358+
359+
1. the file passed with `--manifest <path>`;
360+
2. a `gms.json` published in the model repo, served byte-for-byte;
361+
3. a manifest derived from the pipeline's `__call__` signature.
362+
363+
A derived manifest is a best-effort reading of the signature: scalar arguments become parameters, prompts and
364+
media arguments become inputs, and arguments with a core GMS name are renamed (`num_inference_steps` becomes
365+
`steps`, `num_images_per_prompt` becomes `num_outputs`, `generator` becomes `seed`). It carries no value ranges.
366+
Write a `gms.json` when you need ranges, presets, or a duration in seconds.
367+
368+
When a manifest uses a name that differs from the pipeline argument, map it per task under `x-diffusers`:
369+
370+
```json
371+
"x-diffusers": {"inputs": {"caption": "prompt"}, "parameters": {"cfg": "guidance_scale"}}
372+
```
373+
374+
A task is only hosted when every input it requires maps to a pipeline argument. A parameter the pipeline cannot
375+
set is rejected with `not_implemented` unless the request sends its default value.
376+
377+
### Adapters
378+
379+
A request can apply LoRAs for that generation only. They are loaded before the call and unloaded after it:
380+
381+
```bash
382+
curl localhost:8000/v1/generations -d '{
383+
"model": "black-forest-labs/FLUX.1-dev",
384+
"inputs": {"prompt": "a tiny cat"},
385+
"adapters": [{"type": "lora", "path": "hf:alvdansen/littletinies", "scale": 0.8}]
386+
}'
387+
```
388+
389+
Request adapters are turned off when the server starts with `--compile` or `--lora`; such requests fail with
390+
`not_implemented`.
391+
392+
### SGLang and vLLM-Omni backends
393+
394+
`--backend sglang` and `--backend vllm` run the model in [SGLang Diffusion](https://docs.sglang.io/) or
395+
[vLLM-Omni](https://docs.vllm.ai/projects/vllm-omni) instead of in-process. `serve` starts `sglang serve` or
396+
`vllm serve --omni` as a child process on a local port, waits for it to become healthy, and translates GMS
397+
requests to the engine's OpenAI-compatible image and video routes. Clients only ever talk to the GMS endpoints
398+
of `serve`. The engine must be installed separately.
399+
400+
```bash
401+
diffusers-cli serve --model Qwen/Qwen-Image --backend sglang --backend-args '--num-gpus 2'
402+
diffusers-cli serve --model Wan-AI/Wan2.2-T2V-A14B-Diffusers --backend vllm
403+
```
404+
405+
The pipeline flags (`--dtype`, `--cpu-offload`, `--compile`, ...) do not apply to these backends; pass the
406+
engine's own flags with `--backend-args`. Compared with the in-process backend, they serve image and video
407+
outputs only, accept `image` as the only media input, expose the parameters both engines share (size, steps,
408+
guidance scale, seed, number of frames, fps, number of images), report no step progress, and apply no request
409+
adapters.
410+
411+
## `generate`
412+
413+
Send a generation to a server started with [`serve`](#serve), or to any other Generative Media Spec server, and
414+
save its outputs. Unlike [`run`](#run), `generate` loads no pipeline: it reads the model's discovery document,
415+
submits the request, polls the generation until it finishes, and downloads the files.
416+
417+
```bash
418+
diffusers-cli generate --url http://localhost:8000 --inputs '{"prompt": "a cat on the moon"}'
419+
420+
diffusers-cli generate --url http://localhost:8000 \
421+
--inputs '{"prompt": "make the fur grey", "image": "cat.png"}' \
422+
--parameters '{"steps": 8, "seed": 0}' --output outputs/
423+
```
424+
425+
`--inputs` and `--parameters` use the names the server declares, not pipeline argument names (`steps`, not
426+
`num_inference_steps`). An image, mask, video or audio input can be an http(s) URL or a local file path; local
427+
files are sent inline as base64. `--task`, `--preset`, `--adapters` and `--output-format` map to the request
428+
fields of the same name. `--model` is only needed when the server hosts more than one model.
429+
430+
Outputs are saved to `~/.diffusers/cli/generate/outputs/<generation id>/` unless `--output` names a directory.
431+
`--token` sends a bearer token in the `Authorization` header, for servers behind an authenticating proxy; no
432+
token is sent by default. Interrupting the command cancels the generation on the server.
433+
290434
## `custom_blocks`
291435

292436
Package a local `ModularPipelineBlocks` subclass for upload to the Hub. Reads a Python file, AST-scans it for

‎setup.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,8 @@
146146
"opencv-python",
147147
"timm",
148148
"flashpack",
149+
"fastapi",
150+
"uvicorn",
149151
]
150152

151153
# this is a lookup table with items like:
@@ -253,6 +255,7 @@ def run(self):
253255
extras["nvidia_modelopt"] = deps_list("nvidia_modelopt[hf]")
254256
extras["sdnq"] = deps_list("sdnq")
255257
extras["flashpack"] = deps_list("flashpack")
258+
extras["serve"] = deps_list("fastapi", "uvicorn")
256259

257260
extras["dev"] = extras["quality"] + extras["test"] + extras["training"] + extras["docs"] + extras["torch"]
258261

‎src/diffusers/commands/diffusers_cli.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,10 @@
2020
from .custom_blocks import CustomBlocksCommand
2121
from .env import EnvironmentCommand
2222
from .fp16_safetensors import FP16SafetensorsCommand
23+
from .generate import GenerateCommand
2324
from .run import RunCommand
2425
from .schema import SchemaCommand
26+
from .serve import ServeCommand
2527
from .skills import SkillsCommand
2628

2729

@@ -47,8 +49,10 @@ def main():
4749
EnvironmentCommand.register_subcommand(commands_parser)
4850
FP16SafetensorsCommand.register_subcommand(commands_parser)
4951
CustomBlocksCommand.register_subcommand(commands_parser)
52+
GenerateCommand.register_subcommand(commands_parser)
5053
RunCommand.register_subcommand(commands_parser)
5154
SchemaCommand.register_subcommand(commands_parser)
55+
ServeCommand.register_subcommand(commands_parser)
5256
SkillsCommand.register_subcommand(commands_parser)
5357

5458
# Let's go

0 commit comments

Comments
 (0)