diff --git a/CHANGELOG.md b/CHANGELOG.md index be5b971d..b0614bf3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,12 +2,18 @@ ## v0.7.0-beta.x (see [genai versions](https://crates.io/crates/genai/versions)) +- `!` API CHANGE - `Error::HttpError` adds a `headers: Box` field carrying the response headers of failed streaming HTTP calls (e.g., `retry-after`, `retry-after-ms`, `x-should-retry`), matching the non-streaming `webc::Error::ResponseFailedStatus`, so downstream retry layers can honor provider-requested retry delays. Exhaustive matchers/constructors of `Error::HttpError` must add the `headers` field (or match with `..`). - `!` API CHANGE - `Tool` adds the public `custom_format: Option` field for provider-native freeform custom-tool formats. Downstream `Tool` struct literals must add `custom_format: None`, or preferably migrate to `Tool::new(...)` and builder methods. `Tool::with_custom_format(...)` is the new builder API. +- `!` API CHANGE - `ToolResponse` adds the public `parts: Option>` field for binary tool-result attachments (e.g., screenshots produced by agentic tools). Downstream `ToolResponse` struct literals must add `parts: None`, or preferably migrate to `ToolResponse::new(...)` and the new `ToolResponse::with_parts(...)` / `ToolResponse::append_binary(...)` builders. Image parts serialize natively where the wire supports them (Anthropic `tool_result`, Bedrock Converse `toolResult`, OpenAI Responses `function_call_output`) and ride in a follow-up user message elsewhere (OpenAI Chat Completions-compatible providers, Gemini, Ollama). Non-image parts are skipped with a warning, and text-only tool responses keep their exact previous serialization on every adapter. +- `!` BEHAVIOR CHANGE - A `ContentPart::ToolResponse` embedded in an Assistant-role message now fails serialization with the existing `Error::MessageContentTypeNotSupported` (no new error variant) on every adapter: OpenAI Chat Completions and Responses, Anthropic, Bedrock Converse, Gemini, and Ollama native — including the delegating providers that reuse those serializers (Vertex Claude models, opencode_go, minimax, github_copilot, ollama_cloud, ...). Previously Anthropic, Bedrock Converse, and Gemini silently dropped the embedded response, Ollama garbled it into the assistant `content`, and the OpenAI serializers silently dropped it. No provider wire represents a tool result authored by the assistant, so the shape has no faithful translation; the error's cause points at the supported Tool-role message shape (e.g., `ChatMessage::from(ToolResponse)`). The crate-wide rule is now: supported shapes are translated faithfully, unsupported shapes fail loudly, nothing vanishes. +- `+` `Client` adds per-request exec hooks, following the resolver idiom (dedicated types with sync/async function variants, installed via the `ClientBuilder`, stored in the `ClientConfig`). `ClientBuilder::with_payload_interceptor(...)` / `with_payload_interceptor_fn(...)` set a `PayloadInterceptor` called on each chat exec call (streaming and non-streaming) with the target `ModelIden` and the serialized provider payload (`serde_json::Value`) before the HTTP request is built — returning `Some(value)` replaces the payload, `None` keeps it unchanged. `ClientBuilder::with_response_observer(...)` / `with_response_observer_fn(...)` set a `ResponseObserver` called with the `ModelIden`, response `StatusCode`, and `HeaderMap` as soon as the HTTP response arrives and before its body/stream is consumed — including on 4xx/5xx responses (on the streaming path, the send is lazy, so the observer fires during the first stream poll, before the status check and the `Error::HttpError` construction). Chat exec paths only (`exec_chat` / `exec_chat_stream`); embeddings and model listing are not hooked. - `+` New Providers: - AtlasCloud - default env: `ATLASCLOUD_API_KEY`, Adapter: OpenAI, endpoint: `https://api.atlascloud.ai/v1/` (activated on the `atlascloud::` namespace) (PR #259) - Qwen Cloud - default env: `QWEN_CLOUD_API_KEY`, Adapter: OpenAI, endpoint: `https://dashscope-intl.aliyuncs.com/compatible-mode/v1/` (activated on the `qwen_cloud::` namespace) - Kimi - default env: `KIMI_API_KEY`, Adapter: OpenAI, endpoint: `https://api.moonshot.ai/v1/` (activated on the `kimi::` namespace or `kimi` model prefix, moonshot.ai) - Anthropic: + - `+` Serialize `ToolResponse.parts` image attachments as base64 `image` blocks inside the `tool_result` content array (after the text block). Text-only tool responses keep the legacy plain-string `content`. Non-image parts are skipped with a warning, since Anthropic `tool_result` content only accepts text and image blocks. + - `^` Support URL image sources in user messages and tool results: `BinarySource::Url` images now serialize natively as `{"type": "image", "source": {"type": "url", "url": ...}}` blocks instead of being silently omitted with a warning (the Anthropic Messages API natively supports URL image sources). Base64 image serialization is unchanged. Delegating providers that reuse the Anthropic serializers (minimax, the `baidu-coding-anthropic` namespace, Vertex Claude models, opencode_go minimax models) inherit this automatically; whether a given gateway accepts URL sources is provider-side. - `+` Expose streaming SSE ping messages as provider-neutral `ChatStreamEvent::Heartbeat` events, allowing callers to distinguish a live long-running stream from a stall. (PR #271) - `+` Add prompt caching on tools via `Tool::with_cache_control`, and make request-level `ChatOptions::with_cache_control` automatically apply a cache breakpoint to the static (tools+system) prefix, which was previously ignored. `Ephemeral24h` is documented as clamped to Anthropic's max `1h` TTL. - `+` Support the `extra_body` `ChatOptions` field, merging extra request body fields. ([#255](https://github.com/jeremychone/rust-genai/pull/255)) @@ -21,6 +27,9 @@ - Fable and Mythos omit `thinking`, because it is always on and cannot be explicitly disabled. - The Anthropic `-zero` model suffix is canonical, while `-none` remains a backward-compatible alias. Both map to `Zero` and are stripped. - OpenAI: + - `-` Chat Completions and Responses: a `ToolResponse` embedded in a User-role message is now serialized instead of silently dropped (text and all). This user-embedded shape (the Anthropic-style form where tool results ride as user content blocks) is extracted into proper `role:"tool"` messages / `function_call_output` items (`custom_tool_call_output` for custom tool calls) emitted before the carrying user message, with images folded into that same user message (`image_url` / `input_image` blocks, no label); a user message left empty by the extraction is omitted. Text/placeholder and custom-output rules match Tool-role serialization, and `call_id`s are serialized as-is (provider-side validation, as elsewhere). The Ollama native serializer (shared by `ollama_cloud`) gets the same extraction — it previously garbled the shape (the response text was inserted as the user `content`, where sibling text parts overwrote it, and image parts were lost): the user-embedded `ToolResponse` now becomes a `role:"tool"` message before the carrying user message, its images ride the existing labeled follow-up user image message (native base64 `images` array), and the same empty-user-message omission applies; in the same stroke, the Ollama Tool-role path now emits one `role:"tool"` message per `ToolResponse` when a Tool-role message carries several, in part order (previously each response's text overwrote the previous, keeping only the last), with their images still batched into that single labeled follow-up user image message. A `ToolResponse` embedded in an Assistant-role message fails loudly instead — see the crate-wide BEHAVIOR CHANGE entry above. + - `+` Chat Completions: `ToolResponse.parts` images ride in a follow-up `user` message (`image_url` blocks), batched across a run of consecutive tool messages; the `tool` message keeps its text, or the `"(see attached image)"` placeholder when the result is image-only. Applies to all OpenAI-compatible providers sharing this serializer. + - `+` Responses: `ToolResponse.parts` images serialize natively as `input_image` items in the `function_call_output` `output` array (after the `input_text` item). Custom tool-call outputs stay raw strings (with the `"(see attached image)"` / `"(no tool output)"` placeholder rules); their images ride in a follow-up `user` message input item, batched across a run of consecutive tool messages. - `+` Support OpenAI Responses freeform custom tools with grammar-constrained raw-string input. Custom tools serialize as `type: "custom"`, custom tool-call input streams incrementally, and round-trips as `custom_tool_call` / `custom_tool_call_output` items. (PR #266) - `^` Capture `cache_write_tokens` from prompt-cache usage and normalize it to `Usage.prompt_tokens_details.cache_creation_tokens` for Chat Completions and Responses API payloads. - `^` GPT-5.6 and later now use cache opt-in only. Here is how to opt in: (see [PR #260](https://github.com/jeremychone/rust-genai/pull/260)) @@ -34,14 +43,19 @@ - `+` Sanitize JSON Schema for structured responses and strict tools. (PR #263) - `!` Apply the `ReasoningEffort::None` to `ReasoningEffort::Zero` rename mechanically, while preserving provider-specific keyword mappings. - Gemini: + - `+` `ToolResponse.parts` images ride in a follow-up `user` turn (`inline_data` / `file_data`) emitted after the merged `functionResponse` turn; the `functionResponse` keeps its text, or the `"(see attached image)"` placeholder when the result is image-only. Also applies to Vertex (Google publisher). - `^` Forward JSON Schema raw via `responseJsonSchema` and `parametersJsonSchema`. (PR #257) - `!` Map `ReasoningEffort::Zero` to a budget of `0`, which might be rejected by the provider on some models. - `-` Protect known model names such as `deepseek-r1-zero` from reasoning suffix stripping by using a whitelist in `from_model_name()`. - Bedrock: + - `+` `ToolResponse.parts` images serialize natively as `image` blocks inside the Converse `toolResult` content array (after the text block). - `!` Apply the `ReasoningEffort::None` to `ReasoningEffort::Zero` rename mechanically, with behavior unchanged. +- Ollama: + - `+` `ToolResponse.parts` images (base64 only) ride in a follow-up `user` message via the native `images` array; the `tool` message keeps its text, or the `"(see attached image)"` placeholder when the result is image-only. - Cross-provider adapters: - `^` Move messages after tools in JSON payloads for better prompt cache utilization. (PR #262) - OpenTelemetry: + - `-` Fix `otel` feature compilation, by covering the `CacheBreakpointNoEligibleContent` error variant in the `error.type` derivation (broken since v0.7.0-beta.18). - `+` Add optional OpenTelemetry GenAI semantic-convention instrumentation behind the new `otel` feature, off by default, using a pure `tracing` bridge with no extra dependencies. - Auto-instruments `exec_chat`, `exec_chat_stream`, and `exec_embed` as `gen_ai.*` spans, including operation, provider, request params, server address/port, usage tokens, finish reasons, response id/model, streaming time-to-first-chunk, and `error.type`. Prompt and response content capture is opt-in via `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT`. Adds opt-in `genai::otel` helpers for agent, workflow, and tool spans, plus the evaluation-result event. Export by wiring `tracing-opentelemetry` in the application. See `docs/otel.md` and `examples/c12-otel.rs`. diff --git a/dev/specs/spec-chat.md b/dev/specs/spec-chat.md index f28942c9..e7ba894c 100644 --- a/dev/specs/spec-chat.md +++ b/dev/specs/spec-chat.md @@ -34,7 +34,7 @@ The module exports the following key data structures: - **Tooling:** - `Tool`: Metadata and schema defining a function the model can call. - `ToolCall`: The model's invocation request for a specific tool. - - `ToolResponse`: The output returned from executing a tool, matched by call ID. + - `ToolResponse`: The output returned from executing a tool, matched by call ID. Carries text `content` plus optional binary `parts` (e.g., screenshots); see `spec-tool.md` for the per-adapter image serialization matrix and the mapping of user-embedded tool responses (the Anthropic-style shape, serialized natively or extracted to standalone tool messages/items per wire). A `ToolResponse` embedded in an Assistant-role message is rejected with a hard error on every adapter — no provider wire supports assistant-authored tool results; use a Tool-role message. - **Metadata:** - `Usage`, `PromptTokensDetails`, `CompletionTokensDetails`: Normalized token usage statistics. diff --git a/dev/specs/spec-client.md b/dev/specs/spec-client.md index cedd505d..63d4488d 100644 --- a/dev/specs/spec-client.md +++ b/dev/specs/spec-client.md @@ -14,9 +14,13 @@ The `client` module exposes the following public types: - Core execution methods: `exec_chat`, `exec_chat_stream`, `exec_embed`, `embed`, `embed_batch`. - Resolution/Discovery methods: `all_model_names`, `resolve_service_target`. -- **`ClientBuilder`**: Provides a fluent interface for constructing a `Client`. Used to set `ClientConfig`, default `ChatOptions`, `EmbedOptions`, and custom resolvers (`AuthResolver`, `ServiceTargetResolver`, `ModelMapper`). +- **`ClientBuilder`**: Provides a fluent interface for constructing a `Client`. Used to set `ClientConfig`, default `ChatOptions`, `EmbedOptions`, custom resolvers (`AuthResolver`, `ServiceTargetResolver`, `ModelMapper`), and per-request exec hooks (`PayloadInterceptor`, `ResponseObserver`). -- **`ClientConfig`**: Holds the resolved and default configurations used by the `Client`, including resolver functions and default options. +- **`ClientConfig`**: Holds the resolved and default configurations used by the `Client`, including resolver functions, exec hooks, and default options. + +- **`PayloadInterceptor`**: Per-request exec hook (resolver idiom: enum with sync `InterceptorFn` and async `InterceptorAsyncFn` variants, created via `from_interceptor_fn` / `from_interceptor_async_fn`). Called on each chat exec call (streaming and non-streaming) with the target `ModelIden` and the serialized provider payload (`serde_json::Value`), after `to_web_request_data` and before the HTTP request is built. Returning `Some(value)` replaces the payload sent over the wire; `None` keeps it unchanged (the payload is cloned once per request only when an interceptor is set). + +- **`ResponseObserver`**: Per-request exec hook (same idiom, `from_observer_fn` / `from_observer_async_fn`). Called on each chat exec call with the target `ModelIden`, the response `StatusCode`, and the response `HeaderMap` as soon as the HTTP response arrives and before its body/stream is consumed — including on 4xx/5xx responses. Chat exec paths only (embeddings and model listing are not hooked). - **`Headers`**: A simple map wrapper (`HashMap`) for managing HTTP headers in requests. @@ -34,7 +38,9 @@ The module is composed of several files that implement the layered client archit - `config.rs`: Defines `ClientConfig` and the core `resolve_service_target` logic, which orchestrates calls to `ModelMapper`, `AuthResolver`, and `ServiceTargetResolver` before falling back to adapter defaults. -- `client_impl.rs`: Contains the main implementation of the public API methods on `Client`, such as `exec_chat` and `exec_embed`. These methods perform service resolution and delegate to `AdapterDispatcher` for request creation and response parsing. +- `client_impl.rs`: Contains the main implementation of the public API methods on `Client`, such as `exec_chat` and `exec_embed`. These methods perform service resolution and delegate to `AdapterDispatcher` for request creation and response parsing. The chat exec paths also apply the exec hooks: the `PayloadInterceptor` runs between `to_web_request_data` and the request-builder construction (which makes the `exec_chat_stream` setup an async block), and the `ResponseObserver` is bound to the request's `ModelIden` as a crate-internal `BoundResponseObserver` that is handed to `WebClient::do_post_with_observer` (non-streaming) or threaded through `Adapter::to_chat_stream` into the web stream (streaming). + +- `exec_hooks.rs`: Defines the per-request exec hooks `PayloadInterceptor` and `ResponseObserver` (with their sync/async function traits and `Into*` conversion traits, mirroring `AuthResolver`), plus the crate-internal `BoundResponseObserver` pairing an observer with the in-flight request's `ModelIden` so the web layer can fire it without knowing about model resolution. - `headers.rs`: Implements the `Headers` utility for managing key-value HTTP header maps. @@ -57,3 +63,5 @@ The module is composed of several files that implement the layered client archit - **Builder Pattern for Configuration**: `ClientBuilder` enforces configuration before client creation, simplifying object construction and ensuring necessary dependencies are set up correctly. - **Headers Simplification**: The `Headers` struct abstracts HTTP header management, ensuring that subsequent merges or overrides result in a single, final header value, which is typical for API key authorization overrides. + +- **Exec Hooks (Per-Request Observability/Interception)**: `PayloadInterceptor` and `ResponseObserver` give downstream runtimes request auditing and payload-shaping without changing the serialized `ChatOptions` (which stays hooks-free since it is Serialize/Deserialize). On the streaming path, the HTTP send is lazy (performed on the first stream poll inside `WebStream`), so the observer is carried into the stream and fires when the send resolves — before the status check, so it also fires on failing responses (alongside the headers-carrying `Error::HttpError`). The synthetic `Event::Open` of the SSE stream is emitted before any HTTP activity and is deliberately not tied to the observer. diff --git a/dev/specs/spec-tool.md b/dev/specs/spec-tool.md index 9ddbdf41..78efdfcf 100644 --- a/dev/specs/spec-tool.md +++ b/dev/specs/spec-tool.md @@ -60,11 +60,39 @@ pub struct ToolCall { ```rust pub struct ToolResponse { pub call_id: String, + pub fn_name: Option, pub content: String, + pub parts: Option>, } ``` - `ToolResponse::new(call_id, content)`: Links the execution output back to the original call. +- `ToolResponse::from_tool_call(&tool_call, content)`: Convenience constructor that also captures `fn_name` (needed by Gemini's `functionResponse.name`). +- `with_fn_name(name)`: Builder-style method to set the function/tool name. +- `with_parts(parts)` / `append_binary(binary)`: Builder-style methods to attach binary parts (e.g., screenshots) to the tool result. + +Adapter behavior for `parts` (image parts only; non-image parts are skipped with a warning on every adapter, and a `ToolResponse` without `parts` keeps its exact legacy serialization everywhere): + +- **Anthropic**: image parts serialize natively as `image` blocks inside the `tool_result` content array, after the text block (text block omitted when the text is empty). Both sources are emitted natively — base64 as `{"type": "base64", "media_type": ..., "data": ...}` and URL as `{"type": "url", "url": ...}` — matching user-message image handling (the Anthropic Messages API supports URL image sources). Delegating providers that reuse the Anthropic serializers (minimax, the `baidu-coding-anthropic` namespace, Vertex Claude models, opencode_go minimax models) inherit this automatically; whether a given gateway accepts URL sources is provider-side. +- **Bedrock (Converse)**: image parts serialize natively as `image` blocks inside the `toolResult` content array, after the text block. Base64 only (Converse binary handling does not support URLs). +- **OpenAI Responses**: image parts serialize natively: `function_call_output.output` becomes an array of `input_text` (when text is non-empty) plus `input_image` items (`detail: "auto"`, `image_url` as data URL or plain URL). `custom_tool_call_output` stays a raw string with the same placeholder rules as Chat Completions (`"(see attached image)"` / `"(no tool output)"`); its images are rescued into a follow-up `user` message input item (`input_text` label + `input_image` items), batched across a run of consecutive Tool messages and emitted after the run. +- **OpenAI Chat Completions** (shared by all OpenAI-compatible providers: Groq, Together, Fireworks, DeepSeek, etc.): the `tool` message stays text-only. When parts are present, its content is the text, or `"(see attached image)"` when the result has images but no text, or `"(no tool output)"` when it has neither. The images then ride in a follow-up `user` message with content `[{type: "text", text: "Attached image(s) from tool result:"}, ...image_url blocks]`. Images from a run of consecutive Tool messages are batched into ONE follow-up user message, emitted before the next non-tool message. +- **Gemini** (also Vertex/Google): the `functionResponse` content stays text-only with the same placeholder rules as Chat Completions. Images from Tool-role messages ride in a follow-up `user` turn (label text part + `inline_data` for base64 / `file_data` for URLs), batched across the run of Tool messages and emitted after them, so the `functionResponse` turns can still be merged into the single user turn the Gemini FC protocol requires. For a `ToolResponse` embedded in a User-role message, the images are appended inline in that same user turn instead. +- **Ollama (native)**: the `tool` message stays text-only with the same placeholder rules. A Tool-role message carrying multiple `ToolResponse` parts emits one `role:"tool"` message per response, in part order (matching the per-response messages the other serializers emit). Base64 images ride in a follow-up `user` message using the native `images` array (one follow-up per Tool-role message, accumulating the images of all its responses); URL sources are skipped with a warning. + +Notes: + +- genai has no model-capability catalog, so the fallback is emitted whenever parts exist — attaching parts is the caller's opt-in that the target model accepts image input. No interstitial-assistant compatibility message is inserted, and Gemini is not version-gated for multimodal `functionResponse` nesting (the universal follow-up-user-turn form is used instead). + +Embedded tool responses (a `ContentPart::ToolResponse` carried inside a User- or Assistant-role message instead of a Tool-role message): + +- **Anthropic** / **Bedrock (Converse)**: a user-embedded `ToolResponse` serializes inline as a native `tool_result` / `toolResult` content block. This embedded shape is the Anthropic wire's native representation — tool results are literally user-message content blocks there. +- **Gemini** (also Vertex/Google): a user-embedded `ToolResponse` serializes inline as a `functionResponse` part, with its images appended inline in the same user turn (see above). +- **OpenAI Chat Completions**: the wire's only tool-result representation is the standalone `role:"tool"` message, so each user-embedded `ToolResponse` is extracted into a proper `role:"tool"` message (same text/placeholder rules as Tool-role messages) emitted BEFORE the user message carrying the remaining content — such a user message conventionally directly follows the assistant `tool_calls` message, and the wire requires tool messages to sit adjacent to it (extracted tool messages are also inserted before any pending tool-image flush message for the same reason). Image parts fold into that SAME user message as `image_url` blocks (no label message), mirroring Gemini's user-embedded handling. A user message left with no content after extraction is omitted. +- **OpenAI Responses**: each user-embedded `ToolResponse` is extracted into a `function_call_output` item — or `custom_tool_call_output` when the `call_id` belongs to a custom tool call — emitted before the user message item. Function outputs keep the native `input_text`/`input_image` output-array image handling; images rescued from custom outputs fold into the carrying user message item as `input_image` items (no label item). A user message item left with no content after extraction is omitted. +- **Ollama (native)**: like Chat Completions, the wire's only tool-result representation is the standalone `role:"tool"` message, so each user-embedded `ToolResponse` is extracted into a proper `role:"tool"` message (same text/placeholder rules as Tool-role messages) emitted BEFORE the user message carrying the remaining content. Its image parts ride the same labeled follow-up `user` image message as Tool-role responses (native base64 `images` array; URL sources skipped with a warning), emitted after the remaining user message. A user message left with no content after extraction is omitted. +- **Assistant-embedded** `ToolResponse` parts are rejected with a hard error by EVERY serializer (OpenAI Chat Completions, OpenAI Responses, Anthropic, Bedrock Converse, Gemini, Ollama native): `Error::MessageContentTypeNotSupported`, with a cause pointing the caller at the Tool-role message shape (e.g., `ChatMessage::from(ToolResponse)`). Rationale: no provider wire has a representation for a tool result authored by the assistant — tool results are standalone `role:"tool"` messages / output items on the OpenAI wires and user-carried `tool_result` / `toolResult` / `functionResponse` blocks on the Anthropic-style wires — so there is no faithful translation of the shape. Serializing it would mean either dropping content silently (the previous Anthropic/Bedrock/Gemini/Ollama behavior) or inventing a wire placement the API does not define; instead, the crate-wide rule is: supported shapes are translated faithfully, unsupported shapes fail loudly, nothing vanishes. +- `call_id` matching is not validated at serialization time (provider-side validation is the norm across the crate); a user-embedded response whose `call_id` matches nothing is serialized anyway. ### Integration Points diff --git a/dev/specs/spec-webc.md b/dev/specs/spec-webc.md index a4bd3232..abd3e5c6 100644 --- a/dev/specs/spec-webc.md +++ b/dev/specs/spec-webc.md @@ -19,9 +19,9 @@ The module consists of three main internal components: - `error.rs`: Defines the `Error` enum and the module-scoped `Result` type alias. It captures network/HTTP related failures and external errors like `reqwest::Error` and `value_ext::JsonValueExtError`. -- `web_client.rs`: Contains the `WebClient` struct, a thin wrapper around `reqwest::Client`. It provides methods (`do_get`, `do_post`) for non-streaming standard HTTP communication, which assumes the response body is JSON and is parsed into `serde_json::Value`. It also defines `WebResponse`, which encapsulates the HTTP status and parsed JSON body. +- `web_client.rs`: Contains the `WebClient` struct, a thin wrapper around `reqwest::Client`. It provides methods (`do_get`, `do_post`) for non-streaming standard HTTP communication, which assumes the response body is JSON and is parsed into `serde_json::Value`. `do_post_with_observer` is the observed variant used by the chat exec path: it fires the (optional) crate-internal `BoundResponseObserver` on the response head (status + headers) right after `send()` resolves, before the body is consumed and before the status check — `do_post` simply delegates with no observer. It also defines `WebResponse`, which encapsulates the HTTP status and parsed JSON body. -- `web_stream.rs`: Implements `WebStream`, a custom `futures::Stream` implementation designed for handling non-SSE streaming protocols used by some AI providers (e.g., Cohere, Gemini). It defines `StreamMode` to specify how stream chunks should be parsed (either by a fixed delimiter or specialized handling for "Pretty JSON Array" formats). +- `web_stream.rs`: Implements `WebStream`, a custom `futures::Stream` implementation designed for handling non-SSE streaming protocols used by some AI providers (e.g., Cohere, Gemini). It defines `StreamMode` to specify how stream chunks should be parsed (either by a fixed delimiter or specialized handling for "Pretty JSON Array" formats). `WebStream` (and the `EventSourceStream` wrapping it) can carry an optional `BoundResponseObserver` (set via `with_response_observer`, threaded down from `Adapter::to_chat_stream`): since the HTTP send is lazy (performed on the first poll), the observer is awaited inside the send future as soon as the `reqwest::Response` head is in hand — before the success/failure status check, so it also fires on 4xx/5xx responses. ### Key Design Considerations @@ -31,6 +31,8 @@ The module consists of three main internal components: - **Generic JSON Response Handling:** `WebResponse` abstracts successful non-streaming responses by immediately parsing the body into `serde_json::Value`. This allows adapter modules to deserialize into their specific structures subsequently. -- **Error Richness:** The `Error::ResponseFailedStatus` variant includes the `StatusCode`, full `body`, and `HeaderMap` to provide comprehensive debugging information upon API failure. +- **Error Richness:** The `Error::ResponseFailedStatus` variant includes the `StatusCode`, full `body`, and `HeaderMap` to provide comprehensive debugging information upon API failure. On the streaming path, `WebStream` mirrors this: when the lazy request send resolves to a non-success status, it surfaces a crate-level `genai::Error::HttpError` carrying the status, canonical reason, body, and response `HeaderMap` (captured while the `reqwest::Response` is still in hand), so retry-relevant headers such as `retry-after`, `retry-after-ms`, and `x-should-retry` remain visible to downstream retry layers. + +- **Response Observation:** The per-request `ResponseObserver` exec hook (defined in the `client` module) reaches this layer as a `BoundResponseObserver` (observer + `ModelIden`). It fires exactly once per request on the response head — in `do_post_with_observer` for non-streaming calls, and inside `WebStream`'s send future (or the Bedrock byte-stream equivalent) for streaming calls — always before body consumption and before the status check, so failing responses are observed too (in addition to yielding the headers-carrying error described above). When no observer is configured, the code paths are unchanged. - **Async Implementation:** All network operations rely on `tokio` and `reqwest`, ensuring non-blocking execution throughout the I/O layer. `WebStream` leverages `futures::Stream` traits for integration with standard Rust async infrastructure. diff --git a/docs/for-llm/api-reference-for-llm.md b/docs/for-llm/api-reference-for-llm.md index cd1da6ba..c9a55b3c 100644 --- a/docs/for-llm/api-reference-for-llm.md +++ b/docs/for-llm/api-reference-for-llm.md @@ -727,7 +727,7 @@ let chat_res = client.exec_chat("genai_1::some-model", chat_req, None).await?; - `ChatResponse { model_iden, body }`: Error event in stream. - `StreamParse { model_iden, serde_error }`: Stream data parse failure. - `WebStream { model_iden, cause, error }`: Web stream error. - - `HttpError { status, canonical_reason, body }`: HTTP error. + - `HttpError { status, canonical_reason, body, headers }`: HTTP error (streaming path); `headers` carries the failed response headers (e.g., `retry-after`). - `Resolver { model_iden, resolver_error }`: Resolver error wrapper. - `AdapterNotSupported { adapter_kind, feature }`: Feature not supported by adapter. - `AdapterKindMismatch { bound, requested, model }`: A client bound to one adapter received a namespaced model or `ModelIden` targeting another adapter. Since v0.6.0. diff --git a/src/adapter/adapter_types.rs b/src/adapter/adapter_types.rs index eb85db4e..3db5a539 100644 --- a/src/adapter/adapter_types.rs +++ b/src/adapter/adapter_types.rs @@ -1,5 +1,6 @@ use crate::adapter::AdapterKind; use crate::chat::{ChatOptionsSet, ChatRequest, ChatResponse, ChatStreamResponse}; +use crate::client::BoundResponseObserver; use crate::embed::{EmbedOptionsSet, EmbedRequest, EmbedResponse}; use crate::resolver::{AuthData, Endpoint}; use crate::webc::{WebClient, WebResponse}; @@ -43,10 +44,15 @@ pub trait Adapter { ) -> Result; /// To be implemented by Adapters. + /// + /// The `response_observer` (per-request exec hook, if set) must be attached to the underlying + /// web stream so it fires when the lazy HTTP send resolves — on the response head, before the + /// stream body is consumed (also on 4xx/5xx). fn to_chat_stream( model_iden: ModelIden, reqwest_builder: RequestBuilder, options_set: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result; /// To be implemented by Adapters. diff --git a/src/adapter/adapters/anthropic/adapter_impl.rs b/src/adapter/adapters/anthropic/adapter_impl.rs index 40f68044..9cbaf161 100644 --- a/src/adapter/adapters/anthropic/adapter_impl.rs +++ b/src/adapter/adapters/anthropic/adapter_impl.rs @@ -69,8 +69,9 @@ impl Adapter for AnthropicAdapter { model_iden: ModelIden, reqwest_builder: RequestBuilder, options_set: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result { - let event_source = EventSourceStream::new(reqwest_builder); + let event_source = EventSourceStream::new(reqwest_builder).with_response_observer(response_observer); let anthropic_stream = AnthropicStreamer::new(event_source, model_iden.clone(), options_set); let chat_stream = ChatStream::from_inter_stream(anthropic_stream); Ok(ChatStreamResponse { diff --git a/src/adapter/adapters/anthropic/adapter_shared.rs b/src/adapter/adapters/anthropic/adapter_shared.rs index c5692ee6..34409956 100644 --- a/src/adapter/adapters/anthropic/adapter_shared.rs +++ b/src/adapter/adapters/anthropic/adapter_shared.rs @@ -2,12 +2,12 @@ use super::AnthropicAdapter; use super::ant_model::{AnthropicMaxTokens, AnthropicModel, AnthropicModelCapabilities}; use crate::Result; use crate::adapter::adapters::anthropic::ant_reasoning::insert_anthropic_reasoning; -use crate::adapter::adapters::support::get_api_key; +use crate::adapter::adapters::support::{assistant_embedded_tool_response_err, get_api_key}; use crate::adapter::{Adapter, AdapterKind, ServiceType, WebRequestData}; use crate::chat::{ Binary, BinarySource, CacheControl, CacheCreationDetails, ChatOptionsSet, ChatRequest, ChatResponse, ChatResponseFormat, ChatRole, ContentPart, JsonSchemaDialect, MessageContent, PromptTokensDetails, ReasoningEffort, - StopReason, Tool, ToolCall, ToolChoice, ToolConfig, ToolName, Usage, sanitize_json_schema, + StopReason, Tool, ToolCall, ToolChoice, ToolConfig, ToolName, ToolResponse, Usage, sanitize_json_schema, }; use crate::resolver::{AuthData, Endpoint}; use crate::webc::{WebClient, WebResponse}; @@ -108,6 +108,7 @@ impl AnthropicAdapter { /// Takes the GenAI ChatMessages and constructs the System string and JSON Messages for Anthropic. /// - Will push the `ChatRequest.system` and system message to `AnthropicRequestParts.system` pub(in crate::adapter::adapters) fn into_anthropic_request_parts( + model_iden: &ModelIden, mut chat_req: ChatRequest, request_cache_control: Option, ) -> Result { @@ -183,11 +184,14 @@ impl AnthropicAdapter { if is_image { match &source { - BinarySource::Url(_) => { - // As of this API version, Anthropic doesn't support images by URL directly in messages. - warn!( - "Anthropic doesn't support images from URL, need to handle it gracefully" - ); + BinarySource::Url(url) => { + values.push(json!({ + "type": "image", + "source": { + "type": "url", + "url": url, + } + })); } BinarySource::Base64(content) => { values.push(json!({ @@ -227,11 +231,7 @@ impl AnthropicAdapter { // ToolCall is not valid in user content for Anthropic; skip gracefully. ContentPart::ToolCall(_tc) => {} ContentPart::ToolResponse(tool_response) => { - values.push(json!({ - "type": "tool_result", - "content": tool_response.content, - "tool_use_id": tool_response.call_id, - })); + values.push(tool_response_to_tool_result(tool_response)); } ContentPart::ThoughtSignature(_) => {} ContentPart::ReasoningContent(_) => {} @@ -275,7 +275,12 @@ impl AnthropicAdapter { } // Unsupported for assistant role in Anthropic message content ContentPart::Binary(_) => {} - ContentPart::ToolResponse(_) => {} + // No provider wire represents a tool result authored by the + // assistant; fail loudly instead of silently dropping the + // content (use a Tool-role message). + ContentPart::ToolResponse(_) => { + return Err(assistant_embedded_tool_response_err(model_iden)); + } ContentPart::ThoughtSignature(_) => {} ContentPart::ReasoningContent(_) => {} ContentPart::Custom(custom_part) => values.push(custom_part.data), @@ -304,11 +309,7 @@ impl AnthropicAdapter { for part in msg.content { match part { ContentPart::ToolResponse(tool_response) => { - values.push(json!({ - "type": "tool_result", - "content": tool_response.content, - "tool_use_id": tool_response.call_id, - })); + values.push(tool_response_to_tool_result(tool_response)); } ContentPart::Custom(custom_part) => values.push(custom_part.data), _ => {} @@ -420,7 +421,7 @@ impl AnthropicAdapter { system, messages, tools, - } = Self::into_anthropic_request_parts(chat_req, options_set.cache_control().cloned())?; + } = Self::into_anthropic_request_parts(&model, chat_req, options_set.cache_control().cloned())?; // -- Extract Model Name and Reasoning let (_, raw_model_name) = model.model_name.namespace_and_name(); @@ -809,6 +810,88 @@ fn apply_cache_control_to_parts(cache_control: Option<&CacheControl>, parts: Vec parts } +/// Serialize a `ToolResponse` into an Anthropic `tool_result` content item. +/// +/// - Without binary parts, `content` remains a plain string (legacy shape, unchanged). +/// - With parts, `content` becomes an array of a `text` block (when the text is non-empty) +/// followed by `image` blocks for image parts (native `base64` or `url` source). +/// +/// NOTE: Anthropic `tool_result` content only accepts `text` and `image` blocks, +/// so non-image parts are skipped with a warning. Image sources serialize +/// the same way as in user-message image handling above. +fn tool_response_to_tool_result(tool_response: ToolResponse) -> Value { + let ToolResponse { + call_id, + content, + parts, + .. + } = tool_response; + + let parts = parts.unwrap_or_default(); + + if parts.is_empty() { + return json!({ + "type": "tool_result", + "content": content, + "tool_use_id": call_id, + }); + } + + let mut values: Vec = Vec::new(); + if !content.is_empty() { + values.push(json!({"type": "text", "text": content})); + } + + for binary in parts { + if !binary.is_image() { + warn!( + "Anthropic tool_result only supports text and image blocks; skipping non-image part '{}'", + binary.content_type + ); + continue; + } + let Binary { + content_type, source, .. + } = binary; + match source { + BinarySource::Base64(data) => { + values.push(json!({ + "type": "image", + "source": { + "type": "base64", + "media_type": content_type, + "data": data, + } + })); + } + BinarySource::Url(url) => { + values.push(json!({ + "type": "image", + "source": { + "type": "url", + "url": url, + } + })); + } + } + } + + // If all parts were skipped and there is no text, fall back to the legacy string shape. + if values.is_empty() { + return json!({ + "type": "tool_result", + "content": content, + "tool_use_id": call_id, + }); + } + + json!({ + "type": "tool_result", + "content": values, + "tool_use_id": call_id, + }) +} + fn anthropic_tool_choice(tool_choice: Option<&ToolChoice>) -> Option { match tool_choice? { ToolChoice::Auto => Some(json!({"type": "auto"})), diff --git a/src/adapter/adapters/anthropic/adapter_shared_tests.rs b/src/adapter/adapters/anthropic/adapter_shared_tests.rs index 570280dc..0d3ae15d 100644 --- a/src/adapter/adapters/anthropic/adapter_shared_tests.rs +++ b/src/adapter/adapters/anthropic/adapter_shared_tests.rs @@ -658,6 +658,298 @@ fn test_anthropic_adapter_protected_reasoning_suffix_is_retained() -> Result<()> Ok(()) } +/// A `ToolResponse` with an image part must serialize the Anthropic `tool_result` +/// content as an array of a text block followed by a base64 `image` block. +#[test] +fn test_anthropic_tool_response_image_part_serializes_tool_result_blocks() -> Result<()> { + // -- Setup & Fixtures + let tool_call = ToolCall { + call_id: "call_1".to_string(), + fn_name: "take_screenshot".to_string(), + fn_arguments: json!({}), + thought_signatures: None, + }; + let tool_response = ToolResponse::new("call_1", "screenshot taken").with_parts([Binary::from_base64( + "image/png", + "iVBORw0KBASE64", + None, + )]); + let chat_req = ChatRequest::new(vec![ + ChatMessage::user("Take a screenshot"), + ChatMessage::from(vec![tool_call]), + ChatMessage::from(tool_response), + ]); + let target = ServiceTarget { + endpoint: AnthropicAdapter::default_endpoint(AdapterKind::Anthropic), + auth: AuthData::from_single("test-key"), + model: ModelIden::new(AdapterKind::Anthropic, "claude-haiku-4-5"), + }; + + // -- Exec + let web_req = + AnthropicAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default())?; + + // -- Check + let messages = web_req.payload["messages"].as_array().ok_or("messages must be an array")?; + assert_eq!(messages.len(), 3, "user, assistant tool_use, and tool_result messages"); + let tool_result_msg = &messages[2]; + assert_eq!(tool_result_msg["role"], json!("user")); + assert_eq!( + tool_result_msg["content"][0], + json!({ + "type": "tool_result", + "tool_use_id": "call_1", + "content": [ + {"type": "text", "text": "screenshot taken"}, + { + "type": "image", + "source": { + "type": "base64", + "media_type": "image/png", + "data": "iVBORw0KBASE64", + } + } + ] + }) + ); + + Ok(()) +} + +/// Regression guard: a text-only `ToolResponse` must keep the legacy `tool_result` +/// shape with a plain string `content` (no content array). +#[test] +fn test_anthropic_tool_response_text_only_serializes_as_before() -> Result<()> { + // -- Setup & Fixtures + let chat_req = ChatRequest::new(vec![ChatMessage::from(ToolResponse::new("call_1", "42"))]); + let target = ServiceTarget { + endpoint: AnthropicAdapter::default_endpoint(AdapterKind::Anthropic), + auth: AuthData::from_single("test-key"), + model: ModelIden::new(AdapterKind::Anthropic, "claude-haiku-4-5"), + }; + + // -- Exec + let web_req = + AnthropicAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default())?; + + // -- Check + assert_eq!( + web_req.payload["messages"][0]["content"][0], + json!({ + "type": "tool_result", + "content": "42", + "tool_use_id": "call_1", + }) + ); + + Ok(()) +} + +/// A `ToolResponse` embedded in an Assistant message has no representation on any +/// provider wire (there is no "tool result authored by the assistant"), so the +/// serializer must reject the shape with a hard error instead of silently dropping +/// the content (the previous behavior). +#[test] +fn test_anthropic_assistant_embedded_tool_response_is_rejected() -> Result<()> { + // -- Setup & Fixtures + let assistant_msg = ChatMessage::assistant(MessageContent::from_parts(vec![ + ContentPart::from_text("checking"), + ContentPart::ToolCall(ToolCall { + call_id: "call_1".to_string(), + fn_name: "get_weather".to_string(), + fn_arguments: json!({"city": "Paris"}), + thought_signatures: None, + }), + ContentPart::ToolResponse(ToolResponse::new("call_1", "sunny")), + ])); + let chat_req = ChatRequest::new(vec![ChatMessage::user("weather?"), assistant_msg]); + let target = ServiceTarget { + endpoint: AnthropicAdapter::default_endpoint(AdapterKind::Anthropic), + auth: AuthData::from_single("test-key"), + model: ModelIden::new(AdapterKind::Anthropic, "claude-haiku-4-5"), + }; + + // -- Exec + let err = AnthropicAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default()) + .expect_err("assistant-embedded tool response must fail serialization"); + + // -- Check + let crate::Error::MessageContentTypeNotSupported { cause, .. } = err else { + return Err(format!("expected MessageContentTypeNotSupported, got: {err}").into()); + }; + assert!( + cause.contains("Assistant-role message"), + "cause must name the unsupported shape: {cause}" + ); + assert!( + cause.contains("Tool-role message"), + "cause must point at the supported Tool-role shape: {cause}" + ); + + Ok(()) +} + +/// Non-image parts are not valid inside an Anthropic `tool_result`; they must be +/// skipped while the text block and image parts (including URL sources) are preserved. +/// NOTE: URL-source images were previously skipped as well; they now serialize +/// natively as `url` image sources. +#[test] +fn test_anthropic_tool_response_non_image_parts_are_skipped() -> Result<()> { + // -- Setup & Fixtures + let tool_response = ToolResponse::new("call_1", "text kept").with_parts([ + Binary::from_base64("application/pdf", "PDFDATA", Some("doc.pdf".to_string())), + Binary::from_url("image/png", "https://example.com/shot.png", None), + ]); + let chat_req = ChatRequest::new(vec![ChatMessage::from(tool_response)]); + let target = ServiceTarget { + endpoint: AnthropicAdapter::default_endpoint(AdapterKind::Anthropic), + auth: AuthData::from_single("test-key"), + model: ModelIden::new(AdapterKind::Anthropic, "claude-haiku-4-5"), + }; + + // -- Exec + let web_req = + AnthropicAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default())?; + + // -- Check + assert_eq!( + web_req.payload["messages"][0]["content"][0], + json!({ + "type": "tool_result", + "tool_use_id": "call_1", + "content": [ + {"type": "text", "text": "text kept"}, + { + "type": "image", + "source": { + "type": "url", + "url": "https://example.com/shot.png", + } + } + ], + }), + "non-image parts must be skipped, keeping the text block and the URL image" + ); + + Ok(()) +} + +/// A `ToolResponse` with a URL-source image part must serialize the image natively +/// as an Anthropic `url` image source inside the `tool_result` content array. +#[test] +fn test_anthropic_tool_response_url_image_part_serializes_url_source() -> Result<()> { + // -- Setup & Fixtures + let tool_response = ToolResponse::new("call_1", "screenshot taken").with_parts([Binary::from_url( + "image/png", + "https://example.com/shot.png", + None, + )]); + let chat_req = ChatRequest::new(vec![ChatMessage::from(tool_response)]); + let target = ServiceTarget { + endpoint: AnthropicAdapter::default_endpoint(AdapterKind::Anthropic), + auth: AuthData::from_single("test-key"), + model: ModelIden::new(AdapterKind::Anthropic, "claude-haiku-4-5"), + }; + + // -- Exec + let web_req = + AnthropicAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default())?; + + // -- Check + assert_eq!( + web_req.payload["messages"][0]["content"][0], + json!({ + "type": "tool_result", + "tool_use_id": "call_1", + "content": [ + {"type": "text", "text": "screenshot taken"}, + { + "type": "image", + "source": { + "type": "url", + "url": "https://example.com/shot.png", + } + } + ], + }) + ); + + Ok(()) +} + +/// A user message with a URL-source image must serialize natively as an Anthropic +/// `url` image source block (the Messages API supports URL image sources). +#[test] +fn test_anthropic_user_message_url_image_serializes_url_source() -> Result<()> { + // -- Setup & Fixtures + let chat_req = ChatRequest::new(vec![ChatMessage::user(vec![ + ContentPart::from_text("What is in this picture?"), + ContentPart::from_binary_url("image/png", "https://example.com/duck.png", None), + ])]); + let target = ServiceTarget { + endpoint: AnthropicAdapter::default_endpoint(AdapterKind::Anthropic), + auth: AuthData::from_single("test-key"), + model: ModelIden::new(AdapterKind::Anthropic, "claude-haiku-4-5"), + }; + + // -- Exec + let web_req = + AnthropicAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default())?; + + // -- Check + let content = web_req.payload["messages"][0]["content"] + .as_array() + .ok_or("user content must be an array")?; + assert_eq!(content[0], json!({"type": "text", "text": "What is in this picture?"})); + assert_eq!( + content[1], + json!({ + "type": "image", + "source": { + "type": "url", + "url": "https://example.com/duck.png", + } + }) + ); + + Ok(()) +} + +/// Regression guard: a user message with a base64 image must keep the exact +/// base64 `image` source shape (unchanged by the URL-source support). +#[test] +fn test_anthropic_user_message_base64_image_serializes_as_before() -> Result<()> { + // -- Setup & Fixtures + let chat_req = ChatRequest::new(vec![ChatMessage::user(vec![ + ContentPart::from_text("What is in this picture?"), + ContentPart::from_binary_base64("image/png", "iVBORw0KBASE64", None), + ])]); + let target = ServiceTarget { + endpoint: AnthropicAdapter::default_endpoint(AdapterKind::Anthropic), + auth: AuthData::from_single("test-key"), + model: ModelIden::new(AdapterKind::Anthropic, "claude-haiku-4-5"), + }; + + // -- Exec + let web_req = + AnthropicAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default())?; + + // -- Check + assert_eq!( + web_req.payload["messages"][0]["content"][1], + json!({ + "type": "image", + "source": { + "type": "base64", + "media_type": "image/png", + "data": "iVBORw0KBASE64", + } + }) + ); + + Ok(()) +} + // region: --- Support fn build_characterization_payload(model_name: &str, reasoning_effort: Option) -> Result { diff --git a/src/adapter/adapters/baidu/adapter_impl.rs b/src/adapter/adapters/baidu/adapter_impl.rs index 3d9bc2c0..edfbee18 100644 --- a/src/adapter/adapters/baidu/adapter_impl.rs +++ b/src/adapter/adapters/baidu/adapter_impl.rs @@ -302,12 +302,17 @@ impl Adapter for BaiduAdapter { model_iden: ModelIden, reqwest_builder: RequestBuilder, options_set: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result { let baidu_info = BaiduModelEndpoint::from_model(&model_iden); match baidu_info.protocol { - BaiduProtocol::OpenAI => OpenAIAdapter::to_chat_stream(model_iden, reqwest_builder, options_set), - BaiduProtocol::Anthropic => AnthropicAdapter::to_chat_stream(model_iden, reqwest_builder, options_set), + BaiduProtocol::OpenAI => { + OpenAIAdapter::to_chat_stream(model_iden, reqwest_builder, options_set, response_observer) + } + BaiduProtocol::Anthropic => { + AnthropicAdapter::to_chat_stream(model_iden, reqwest_builder, options_set, response_observer) + } } } diff --git a/src/adapter/adapters/bedrock/adapter_api.rs b/src/adapter/adapters/bedrock/adapter_api.rs index 3300f82f..49373579 100644 --- a/src/adapter/adapters/bedrock/adapter_api.rs +++ b/src/adapter/adapters/bedrock/adapter_api.rs @@ -87,8 +87,9 @@ impl Adapter for BedrockApiAdapter { model_iden: ModelIden, reqwest_builder: RequestBuilder, options_set: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result { - let stream = async_stream_bytes(reqwest_builder); + let stream = async_stream_bytes(reqwest_builder, response_observer); let bedrock_stream = BedrockStreamer::new(Box::pin(stream), model_iden.clone(), options_set); let chat_stream = ChatStream::from_inter_stream(bedrock_stream); Ok(ChatStreamResponse { diff --git a/src/adapter/adapters/bedrock/adapter_sigv4.rs b/src/adapter/adapters/bedrock/adapter_sigv4.rs index 7c313347..f4e0a539 100644 --- a/src/adapter/adapters/bedrock/adapter_sigv4.rs +++ b/src/adapter/adapters/bedrock/adapter_sigv4.rs @@ -96,8 +96,9 @@ impl Adapter for BedrockSigv4Adapter { model_iden: ModelIden, reqwest_builder: RequestBuilder, options_set: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result { - let stream = async_stream_bytes(reqwest_builder); + let stream = async_stream_bytes(reqwest_builder, response_observer); let bedrock_stream = BedrockStreamer::new(Box::pin(stream), model_iden.clone(), options_set); let chat_stream = ChatStream::from_inter_stream(bedrock_stream); Ok(ChatStreamResponse { diff --git a/src/adapter/adapters/bedrock/converse.rs b/src/adapter/adapters/bedrock/converse.rs index bb38daa9..8f760352 100644 --- a/src/adapter/adapters/bedrock/converse.rs +++ b/src/adapter/adapters/bedrock/converse.rs @@ -4,9 +4,10 @@ //! work lives here. Publisher-specific bits (reasoning budget, etc.) go under //! `additionalModelRequestFields` via [`BedrockPublisher`]. +use crate::adapter::adapters::support::assistant_embedded_tool_response_err; use crate::chat::{ Binary, BinarySource, ChatOptionsSet, ChatRequest, ChatResponse, ChatRole, ContentPart, MessageContent, - ReasoningEffort, StopReason, Tool, ToolCall, ToolName, Usage, + ReasoningEffort, StopReason, Tool, ToolCall, ToolName, ToolResponse, Usage, }; use crate::webc::WebResponse; use crate::{Error, ModelIden, Result}; @@ -60,7 +61,7 @@ pub(super) fn build_converse_payload( system, messages, tools, - } = into_converse_request_parts(chat_req)?; + } = into_converse_request_parts(model_iden, chat_req)?; let mut payload = json!({}); @@ -267,7 +268,7 @@ struct ConverseRequestParts { } /// Translate a genai `ChatRequest` into Converse's `{system, messages, toolConfig}` shape. -fn into_converse_request_parts(chat_req: ChatRequest) -> Result { +fn into_converse_request_parts(model_iden: &ModelIden, chat_req: ChatRequest) -> Result { let mut messages: Vec = Vec::new(); let mut systems: Vec = Vec::new(); @@ -289,7 +290,7 @@ fn into_converse_request_parts(chat_req: ChatRequest) -> Result { - let blocks = assistant_content_to_converse_blocks(msg.content); + let blocks = assistant_content_to_converse_blocks(model_iden, msg.content)?; if !blocks.is_empty() { messages.push(json!({ "role": "assistant", "content": blocks })); } @@ -335,12 +336,7 @@ fn user_content_to_converse_blocks(content: MessageContent) -> Vec { } } ContentPart::ToolResponse(tool_response) => { - blocks.push(json!({ - "toolResult": { - "toolUseId": tool_response.call_id, - "content": [{ "text": tool_response.content }], - } - })); + blocks.push(tool_response_to_converse_block(tool_response)); } // Not valid in user role for Converse — skip. ContentPart::ToolCall(_) => {} @@ -352,7 +348,7 @@ fn user_content_to_converse_blocks(content: MessageContent) -> Vec { blocks } -fn assistant_content_to_converse_blocks(content: MessageContent) -> Vec { +fn assistant_content_to_converse_blocks(model_iden: &ModelIden, content: MessageContent) -> Result> { let mut blocks = Vec::new(); for part in content { match part { @@ -371,32 +367,76 @@ fn assistant_content_to_converse_blocks(content: MessageContent) -> Vec { } })); } + // No provider wire represents a tool result authored by the assistant; + // fail loudly instead of silently dropping the content (use a Tool-role message). + ContentPart::ToolResponse(_) => return Err(assistant_embedded_tool_response_err(model_iden)), // Unsupported in assistant role for Converse. ContentPart::Binary(_) => {} - ContentPart::ToolResponse(_) => {} ContentPart::ThoughtSignature(_) => {} ContentPart::ReasoningContent(_) => {} ContentPart::Custom(_) => {} } } - blocks + Ok(blocks) } fn tool_content_to_converse_blocks(content: MessageContent) -> Vec { let mut blocks = Vec::new(); for part in content { if let ContentPart::ToolResponse(tool_response) = part { - blocks.push(json!({ - "toolResult": { - "toolUseId": tool_response.call_id, - "content": [{ "text": tool_response.content }], - } - })); + blocks.push(tool_response_to_converse_block(tool_response)); } } blocks } +/// Serialize a `ToolResponse` into a Converse `toolResult` block. +/// +/// Converse natively supports image blocks inside `toolResult.content`, so image +/// parts (base64 only) are emitted after the text block. Non-image parts are +/// skipped with a warning, matching the other adapters' image-only contract. +fn tool_response_to_converse_block(tool_response: ToolResponse) -> Value { + let ToolResponse { + call_id, + content, + parts, + .. + } = tool_response; + let parts = parts.unwrap_or_default(); + + let mut content_blocks: Vec = Vec::new(); + if parts.is_empty() { + content_blocks.push(json!({ "text": content })); + } else { + if !content.is_empty() { + content_blocks.push(json!({ "text": content })); + } + for binary in parts { + if !binary.is_image() { + warn!( + "ToolResponse binary parts only support images for the Bedrock Converse adapter; skipping non-image part '{}'", + binary.content_type + ); + continue; + } + if let Some(block) = binary_to_converse_block(binary) { + content_blocks.push(block); + } + } + // If all parts were skipped, fall back to the legacy text block. + if content_blocks.is_empty() { + content_blocks.push(json!({ "text": content })); + } + } + + json!({ + "toolResult": { + "toolUseId": call_id, + "content": content_blocks, + } + }) +} + fn binary_to_converse_block(binary: Binary) -> Option { let is_image = binary.is_image(); let Binary { @@ -493,3 +533,11 @@ fn tool_to_converse_tool(tool: Tool) -> Result { Ok(json!({ "toolSpec": tool_spec })) } + +// region: --- Tests + +#[cfg(test)] +#[path = "converse_tests.rs"] +mod tests; + +// endregion: --- Tests diff --git a/src/adapter/adapters/bedrock/converse_tests.rs b/src/adapter/adapters/bedrock/converse_tests.rs new file mode 100644 index 00000000..18caad47 --- /dev/null +++ b/src/adapter/adapters/bedrock/converse_tests.rs @@ -0,0 +1,107 @@ +type Result = core::result::Result>; // For tests. + +use super::*; +use crate::adapter::AdapterKind; +use crate::chat::ChatMessage; + +fn test_model_iden() -> ModelIden { + ModelIden::new(AdapterKind::BedrockApi, "anthropic.claude-haiku-4-5") +} + +/// Converse natively supports image blocks inside `toolResult.content`, so a +/// `ToolResponse` with an image part emits the text block followed by the image block. +#[test] +fn test_bedrock_tool_response_image_part_serializes_tool_result_blocks() -> Result<()> { + // -- Setup & Fixtures + let tool_response = + ToolResponse::new("call_1", "screenshot taken").with_parts([Binary::from_base64("image/png", "PNG64", None)]); + let chat_req = ChatRequest::new(vec![ChatMessage::from(tool_response)]); + + // -- Exec + let ConverseRequestParts { messages, .. } = into_converse_request_parts(&test_model_iden(), chat_req)?; + + // -- Check + assert_eq!( + messages, + vec![json!({ + "role": "user", + "content": [{ + "toolResult": { + "toolUseId": "call_1", + "content": [ + { "text": "screenshot taken" }, + { "image": { "format": "png", "source": { "bytes": "PNG64" } } }, + ], + } + }] + })] + ); + + Ok(()) +} + +/// Regression guard: a text-only `ToolResponse` keeps the legacy single text block. +#[test] +fn test_bedrock_tool_response_text_only_serializes_as_before() -> Result<()> { + // -- Setup & Fixtures + let chat_req = ChatRequest::new(vec![ChatMessage::from(ToolResponse::new("call_1", "42"))]); + + // -- Exec + let ConverseRequestParts { messages, .. } = into_converse_request_parts(&test_model_iden(), chat_req)?; + + // -- Check + assert_eq!( + messages, + vec![json!({ + "role": "user", + "content": [{ + "toolResult": { + "toolUseId": "call_1", + "content": [{ "text": "42" }], + } + }] + })] + ); + + Ok(()) +} + +/// A `ToolResponse` embedded in an Assistant message has no representation on any +/// provider wire (there is no "tool result authored by the assistant"), so the +/// serializer must reject the shape with a hard error instead of silently dropping +/// the content (the previous behavior). +#[test] +fn test_bedrock_assistant_embedded_tool_response_is_rejected() -> Result<()> { + // -- Setup & Fixtures + let assistant_msg = ChatMessage::assistant(MessageContent::from_parts(vec![ + ContentPart::from_text("checking"), + ContentPart::ToolCall(ToolCall { + call_id: "call_1".to_string(), + fn_name: "get_weather".to_string(), + fn_arguments: json!({"city": "Paris"}), + thought_signatures: None, + }), + ContentPart::ToolResponse(ToolResponse::new("call_1", "sunny")), + ])); + let chat_req = ChatRequest::new(vec![ChatMessage::user("weather?"), assistant_msg]); + + // -- Exec + let err = into_converse_request_parts(&test_model_iden(), chat_req) + .map(|_| ()) + .expect_err("assistant-embedded tool response must fail serialization"); + + // -- Check + let Error::MessageContentTypeNotSupported { cause, .. } = err else { + return Err(format!("expected MessageContentTypeNotSupported, got: {err}").into()); + }; + assert!( + cause.contains("Assistant-role message"), + "cause must name the unsupported shape: {cause}" + ); + assert!( + cause.contains("Tool-role message"), + "cause must point at the supported Tool-role shape: {cause}" + ); + + Ok(()) +} diff --git a/src/adapter/adapters/bedrock/shared.rs b/src/adapter/adapters/bedrock/shared.rs index 94fb01b9..e1b9594c 100644 --- a/src/adapter/adapters/bedrock/shared.rs +++ b/src/adapter/adapters/bedrock/shared.rs @@ -69,9 +69,10 @@ fn urlencode_path_segment(s: &str) -> String { /// error path. pub(super) fn async_stream_bytes( reqwest_builder: RequestBuilder, + response_observer: Option, ) -> impl futures::Stream> + Send { use futures::StreamExt; - async_stream_once(reqwest_builder).flat_map(|result| match result { + async_stream_once(reqwest_builder, response_observer).flat_map(|result| match result { Ok(stream) => stream.boxed(), Err(err) => futures::stream::once(async move { Err(err) }).boxed(), }) @@ -79,6 +80,7 @@ pub(super) fn async_stream_bytes( fn async_stream_once( reqwest_builder: RequestBuilder, + response_observer: Option, ) -> impl futures::Stream< Item = std::result::Result< futures::stream::BoxStream<'static, std::result::Result>, @@ -91,13 +93,22 @@ fn async_stream_once( .send() .await .map_err(|e| Box::new(e) as crate::error::BoxError)?; + // Fire the response observer exec hook (if set) as soon as the response head is in hand — + // before the status check and before the body is consumed — so it also fires on 4xx/5xx. + if let Some(observer) = response_observer { + observer.observe(resp.status(), resp.headers().clone()).await; + } let status = resp.status(); if !status.is_success() { + // Capture the headers while the response is still in hand + // (e.g., `retry-after`, `retry-after-ms`, `x-should-retry` for retry layers) + let headers = resp.headers().clone(); let body = resp.text().await.unwrap_or_default(); let err = crate::Error::HttpError { status, canonical_reason: status.canonical_reason().unwrap_or("Unknown").to_string(), body, + headers: Box::new(headers), }; return Err(Box::new(err) as crate::error::BoxError); } diff --git a/src/adapter/adapters/cohere/adapter_impl.rs b/src/adapter/adapters/cohere/adapter_impl.rs index e4a3a94a..68fcd952 100644 --- a/src/adapter/adapters/cohere/adapter_impl.rs +++ b/src/adapter/adapters/cohere/adapter_impl.rs @@ -175,8 +175,9 @@ impl Adapter for CohereAdapter { model_iden: ModelIden, reqwest_builder: RequestBuilder, options_set: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result { - let web_stream = WebStream::new_with_delimiter(reqwest_builder, "\n"); + let web_stream = WebStream::new_with_delimiter(reqwest_builder, "\n").with_response_observer(response_observer); let cohere_stream = CohereStreamer::new(web_stream, model_iden.clone(), options_set); let chat_stream = ChatStream::from_inter_stream(cohere_stream); diff --git a/src/adapter/adapters/custom/adapter_impl.rs b/src/adapter/adapters/custom/adapter_impl.rs index b1a1574d..8d8b68c8 100644 --- a/src/adapter/adapters/custom/adapter_impl.rs +++ b/src/adapter/adapters/custom/adapter_impl.rs @@ -92,8 +92,9 @@ impl Adapter for CustomAdapter { model_iden: ModelIden, reqwest_builder: RequestBuilder, options_set: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result { - OpenAIAdapter::to_chat_stream(model_iden, reqwest_builder, options_set) + OpenAIAdapter::to_chat_stream(model_iden, reqwest_builder, options_set, response_observer) } fn to_embed_request_data( diff --git a/src/adapter/adapters/fireworks/adapter_impl.rs b/src/adapter/adapters/fireworks/adapter_impl.rs index 63ed0291..728ef71b 100644 --- a/src/adapter/adapters/fireworks/adapter_impl.rs +++ b/src/adapter/adapters/fireworks/adapter_impl.rs @@ -88,8 +88,9 @@ impl Adapter for FireworksAdapter { model_iden: ModelIden, reqwest_builder: RequestBuilder, options_set: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result { - OpenAIAdapter::to_chat_stream(model_iden, reqwest_builder, options_set) + OpenAIAdapter::to_chat_stream(model_iden, reqwest_builder, options_set, response_observer) } fn to_embed_request_data( diff --git a/src/adapter/adapters/gemini/adapter_impl.rs b/src/adapter/adapters/gemini/adapter_impl.rs index 9b578d9a..d99d329a 100644 --- a/src/adapter/adapters/gemini/adapter_impl.rs +++ b/src/adapter/adapters/gemini/adapter_impl.rs @@ -1,5 +1,7 @@ use crate::adapter::adapters::gemini::GeminiStreamer; -use crate::adapter::adapters::support::get_api_key; +use crate::adapter::adapters::support::{ + TOOL_RESULT_IMAGES_LABEL, assistant_embedded_tool_response_err, get_api_key, tool_response_fallback_text, +}; use crate::adapter::{Adapter, AdapterKind, ServiceType, WebRequestData}; use crate::chat::{ Binary, BinarySource, ChatOptionsSet, ChatRequest, ChatResponse, ChatResponseFormat, ChatRole, ChatStream, @@ -11,6 +13,7 @@ use crate::webc::{EventSourceStream, WebClient, WebResponse}; use crate::{Error, Headers, ModelIden, Result, ServiceTarget}; use reqwest::RequestBuilder; use serde_json::{Value, json}; +use tracing::warn; use value_ext::JsonValueExt; pub struct GeminiAdapter; @@ -242,8 +245,9 @@ impl Adapter for GeminiAdapter { model_iden: ModelIden, reqwest_builder: RequestBuilder, options_set: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result { - let event_source = EventSourceStream::new(reqwest_builder); + let event_source = EventSourceStream::new(reqwest_builder).with_response_observer(response_observer); let gemini_stream = GeminiStreamer::new(event_source, model_iden.clone(), options_set); let chat_stream = ChatStream::from_inter_stream(gemini_stream); @@ -588,8 +592,19 @@ impl GeminiAdapter { systems.push(system); } + // Images attached to tool responses (`ToolResponse.parts`) of Tool-role messages + // ride in a follow-up "user" turn. They are batched across a run of consecutive + // Tool messages and flushed after it, so the functionResponse turns can still be + // merged into the single "user" turn that the Gemini FC protocol requires. + let mut pending_tool_images: Vec = Vec::new(); + // -- Build for msg in chat_req.messages { + if !matches!(msg.role, ChatRole::Tool) && !pending_tool_images.is_empty() { + contents.push(gemini_tool_images_user_content(std::mem::take( + &mut pending_tool_images, + ))); + } match msg.role { // For now, system goes as "user" (later, we might have adapter_config.system_to_user_impl) ChatRole::System => { @@ -631,15 +646,26 @@ impl GeminiAdapter { } ContentPart::ToolResponse(tool_response) => { let fn_name = gemini_function_response_name(&tool_response); + let ToolResponse { content, parts, .. } = tool_response; + let parts = parts.unwrap_or_default(); + let has_parts = !parts.is_empty(); + let image_values = gemini_tool_result_image_parts(parts); + let content = if has_parts { + tool_response_fallback_text(content, !image_values.is_empty()) + } else { + content + }; parts_values.push(json!({ "functionResponse": { "name": &fn_name, "response": { "name": &fn_name, - "content": tool_response.content, + "content": content, } } })); + // Already a "user" turn: append the tool-result images inline. + parts_values.extend(image_values); } ContentPart::ThoughtSignature(thought) => { parts_values.push(json!({ @@ -712,10 +738,11 @@ impl GeminiAdapter { parts_values.push(json!({"thoughtSignature": thought})); } } + // No provider wire represents a tool result authored by the + // assistant; fail loudly instead of silently dropping the + // content (use a Tool-role message). ContentPart::ToolResponse(_) => { - if let Some(thought) = pending_thought.take() { - parts_values.push(json!({"thoughtSignature": thought})); - } + return Err(assistant_embedded_tool_response_err(model_iden)); } ContentPart::ReasoningContent(_) => {} // Custom are ignored for this logic @@ -743,15 +770,28 @@ impl GeminiAdapter { } ContentPart::ToolResponse(tool_response) => { let fn_name = gemini_function_response_name(&tool_response); + let ToolResponse { content, parts, .. } = tool_response; + let parts = parts.unwrap_or_default(); + let has_parts = !parts.is_empty(); + let image_values = gemini_tool_result_image_parts(parts); + let content = if has_parts { + tool_response_fallback_text(content, !image_values.is_empty()) + } else { + content + }; parts_values.push(json!({ "functionResponse": { "name": &fn_name, "response": { "name": &fn_name, - "content": tool_response.content, + "content": content, } } })); + // Images ride in a follow-up "user" turn emitted after the run of + // Tool messages (post functionResponse-merge), since media nested + // inside functionResponse is not supported across Gemini versions. + pending_tool_images.extend(image_values); } ContentPart::ThoughtSignature(thought) => { parts_values.push(json!({ @@ -773,6 +813,11 @@ impl GeminiAdapter { } } + // Flush tool-result images from a trailing run of Tool messages. + if !pending_tool_images.is_empty() { + contents.push(gemini_tool_images_user_content(pending_tool_images)); + } + let system = if !systems.is_empty() { Some(systems.join("\n")) } else { @@ -940,6 +985,48 @@ fn take_bool(v: &mut Value, key: &str) -> bool { .unwrap_or(false) } +/// Convert tool-result binary parts into Gemini user-content part values +/// (`inline_data` for base64, `file_data` for URLs), skipping non-image parts +/// with a warning. +fn gemini_tool_result_image_parts(parts: Vec) -> Vec { + let mut values: Vec = Vec::new(); + for binary in parts { + if !binary.is_image() { + warn!( + "ToolResponse binary parts only support images for the Gemini adapter; skipping non-image part '{}'", + binary.content_type + ); + continue; + } + let Binary { + content_type, source, .. + } = binary; + match source { + BinarySource::Url(url) => values.push(json!({ + "file_data": { + "mime_type": content_type, + "file_uri": url + } + })), + BinarySource::Base64(data) => values.push(json!({ + "inline_data": { + "mime_type": content_type, + "data": data + } + })), + } + } + values +} + +/// Build the follow-up "user" turn that carries tool-result images. +fn gemini_tool_images_user_content(image_parts: Vec) -> Value { + let mut parts: Vec = Vec::with_capacity(image_parts.len() + 1); + parts.push(json!({"text": TOOL_RESULT_IMAGES_LABEL})); + parts.extend(image_parts); + json!({"role": "user", "parts": parts}) +} + fn gemini_function_response_name(tool_response: &ToolResponse) -> String { tool_response .fn_name @@ -1026,6 +1113,112 @@ mod tests { assert_eq!(function_response["response"]["name"], "get_weather"); } + /// Tool-message `ToolResponse.parts` images ride in a follow-up "user" turn, + /// batched after the run of Tool messages so the functionResponse turns still + /// merge into the single "user" turn required by the Gemini FC protocol. + #[test] + fn tool_response_image_parts_ride_in_followup_user_turn() { + // -- Setup & Fixtures + let model_iden = ModelIden::new(AdapterKind::Gemini, "gemini-2.5-flash"); + let tr1 = ToolResponse::new("call_1", "screenshot taken") + .with_fn_name("screenshot") + .with_parts([Binary::from_base64("image/png", "PNG64", None)]); + let tr2 = ToolResponse::new("call_2", "") + .with_fn_name("chart") + .with_parts([Binary::from_base64("image/jpeg", "JPEG64", None)]); + let chat_req = ChatRequest::new(vec![ChatMessage::from(tr1), ChatMessage::from(tr2)]); + + // -- Exec + let parts = GeminiAdapter::into_gemini_request_parts(&model_iden, chat_req).unwrap(); + + // -- Check + // The two functionResponse turns merge into one, followed by the image turn. + assert_eq!(parts.contents.len(), 2); + let fn_parts = parts.contents[0]["parts"].as_array().unwrap(); + assert_eq!(fn_parts.len(), 2); + assert_eq!( + fn_parts[0]["functionResponse"]["response"]["content"], + "screenshot taken" + ); + assert_eq!( + fn_parts[1]["functionResponse"]["response"]["content"], "(see attached image)", + "image-only tool response must use the placeholder text" + ); + assert_eq!( + parts.contents[1], + json!({ + "role": "user", + "parts": [ + {"text": "Attached image(s) from tool result:"}, + {"inline_data": {"mime_type": "image/png", "data": "PNG64"}}, + {"inline_data": {"mime_type": "image/jpeg", "data": "JPEG64"}}, + ] + }) + ); + } + + /// Regression guard: a text-only `ToolResponse` keeps its legacy functionResponse + /// shape with no follow-up user turn. + #[test] + fn tool_response_text_only_serializes_as_before() { + // -- Setup & Fixtures + let model_iden = ModelIden::new(AdapterKind::Gemini, "gemini-2.5-flash"); + let chat_req = ChatRequest::new(vec![ChatMessage::from( + ToolResponse::new("call_1", "42").with_fn_name("calc"), + )]); + + // -- Exec + let parts = GeminiAdapter::into_gemini_request_parts(&model_iden, chat_req).unwrap(); + + // -- Check + assert_eq!( + parts.contents, + vec![json!({ + "role": "user", + "parts": [{"functionResponse": {"name": "calc", "response": {"name": "calc", "content": "42"}}}] + })] + ); + } + + /// A `ToolResponse` embedded in an Assistant message has no representation on any + /// provider wire (there is no "tool result authored by the assistant"), so the + /// serializer must reject the shape with a hard error instead of silently dropping + /// the content (the previous behavior). + #[test] + fn assistant_embedded_tool_response_is_rejected() { + // -- Setup & Fixtures + let model_iden = ModelIden::new(AdapterKind::Gemini, "gemini-2.5-flash"); + let assistant_msg = ChatMessage::assistant(MessageContent::from_parts(vec![ + ContentPart::from_text("checking"), + ContentPart::ToolCall(ToolCall { + call_id: "call_1".to_string(), + fn_name: "get_weather".to_string(), + fn_arguments: json!({"city": "Paris"}), + thought_signatures: None, + }), + ContentPart::ToolResponse(ToolResponse::new("call_1", "sunny")), + ])); + let chat_req = ChatRequest::new(vec![ChatMessage::user("weather?"), assistant_msg]); + + // -- Exec + let err = GeminiAdapter::into_gemini_request_parts(&model_iden, chat_req) + .map(|_| ()) + .expect_err("assistant-embedded tool response must fail serialization"); + + // -- Check + let Error::MessageContentTypeNotSupported { cause, .. } = err else { + panic!("expected MessageContentTypeNotSupported, got: {err}"); + }; + assert!( + cause.contains("Assistant-role message"), + "cause must name the unsupported shape: {cause}" + ); + assert!( + cause.contains("Tool-role message"), + "cause must point at the supported Tool-role shape: {cause}" + ); + } + #[test] fn tool_choice_required_maps_to_gemini_any_mode() { let model_iden = ModelIden::new(AdapterKind::Gemini, "gemini-2.5-flash"); diff --git a/src/adapter/adapters/github_copilot/adapter_impl.rs b/src/adapter/adapters/github_copilot/adapter_impl.rs index bfda5df3..cd9f2b8a 100644 --- a/src/adapter/adapters/github_copilot/adapter_impl.rs +++ b/src/adapter/adapters/github_copilot/adapter_impl.rs @@ -75,8 +75,9 @@ impl Adapter for GithubCopilotAdapter { model_iden: ModelIden, reqwest_builder: RequestBuilder, options_set: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result { - OpenAIAdapter::to_chat_stream(model_iden, reqwest_builder, options_set) + OpenAIAdapter::to_chat_stream(model_iden, reqwest_builder, options_set, response_observer) } fn to_embed_request_data( diff --git a/src/adapter/adapters/ollama/adapter_impl.rs b/src/adapter/adapters/ollama/adapter_impl.rs index 597ff2b5..837c2272 100644 --- a/src/adapter/adapters/ollama/adapter_impl.rs +++ b/src/adapter/adapters/ollama/adapter_impl.rs @@ -63,7 +63,7 @@ impl Adapter for OllamaAdapter { let url = Self::get_service_url(&model, service_type, endpoint)?; // -- Ollama Request Parts - let OllamaRequestParts { messages, tools } = Self::into_ollama_request_parts(chat_req)?; + let OllamaRequestParts { messages, tools } = Self::into_ollama_request_parts(&model, chat_req)?; // -- Ollama Options let mut options = json!({}); @@ -188,9 +188,10 @@ impl Adapter for OllamaAdapter { model_iden: ModelIden, reqwest_builder: RequestBuilder, options_set: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result { let streamer = OllamaStreamer::new( - crate::webc::WebStream::new_with_delimiter(reqwest_builder, "\n"), + crate::webc::WebStream::new_with_delimiter(reqwest_builder, "\n").with_response_observer(response_observer), model_iden.clone(), options_set, ); diff --git a/src/adapter/adapters/ollama/adapter_shared.rs b/src/adapter/adapters/ollama/adapter_shared.rs index e1f973b6..18329495 100644 --- a/src/adapter/adapters/ollama/adapter_shared.rs +++ b/src/adapter/adapters/ollama/adapter_shared.rs @@ -3,11 +3,16 @@ use super::OllamaAdapter; use crate::Headers; use crate::adapter::AdapterKind; -use crate::chat::{Binary, BinarySource, ChatRequest, ContentPart, Tool, ToolName, Usage}; +use crate::adapter::adapters::support::{ + TOOL_RESULT_IMAGES_LABEL, assistant_embedded_tool_response_err, tool_response_fallback_text, +}; +use crate::chat::{Binary, BinarySource, ChatRequest, ChatRole, ContentPart, Tool, ToolName, ToolResponse, Usage}; use crate::resolver::Endpoint; use crate::webc::WebClient; -use crate::{Error, Result}; +use crate::{Error, ModelIden, Result}; use serde_json::{Value, json}; +use std::sync::Arc; +use tracing::warn; use value_ext::JsonValueExt; /// Support functions for other adapters that share Ollama APIs @@ -61,7 +66,10 @@ impl OllamaAdapter { } /// Takes the GenAI ChatMessages and constructs the JSON Messages for Ollama. - pub(in crate::adapter::adapters) fn into_ollama_request_parts(chat_req: ChatRequest) -> Result { + pub(in crate::adapter::adapters) fn into_ollama_request_parts( + model_iden: &ModelIden, + chat_req: ChatRequest, + ) -> Result { let mut messages = Vec::new(); // -- System @@ -81,6 +89,13 @@ impl OllamaAdapter { let mut content = String::new(); let mut images = Vec::new(); let mut tool_calls = Vec::new(); + // Images attached to tool responses (`ToolResponse.parts`); they ride in a + // follow-up "user" message since tool messages carry only text content. + let mut tool_response_images = Vec::new(); + // Whether a `ToolResponse` part of this message was emitted as a standalone + // `role:"tool"` message (see below); when nothing else remains, the carrying + // message is omitted. + let mut had_tool_responses = false; for part in msg.content { match part { @@ -102,13 +117,34 @@ impl OllamaAdapter { })); } ContentPart::ToolResponse(tr) => { - // Note: Ollama native API expects role "tool" for tool response - ollama_msg.x_insert("content", tr.content)?; + // No provider wire represents a tool result authored by the assistant; + // fail loudly instead of garbling it into assistant content + // (use a Tool-role message). + if matches!(msg.role, ChatRole::Assistant) { + return Err(assistant_embedded_tool_response_err(model_iden)); + } + // Note: Ollama native API expects role "tool" for tool response, and + // the standalone `role:"tool"` message is the wire's only tool-result + // representation, so ONE such message is emitted PER response, in part + // order — a Tool-role message can carry several (matching the + // per-response messages the OpenAI Chat Completions serializer emits). + // For a tool response embedded in a user message (the Anthropic-style + // shape where tool results ride as user-message content blocks), the + // extracted tool message is emitted BEFORE the remaining user message. + // Images of every response in the message ride the same single + // labeled follow-up user image message. + let tr_content = tool_response_content_text(tr, &mut tool_response_images); + had_tool_responses = true; + messages.push(json!({ + "role": "tool", + "content": tr_content, + })); } _ => {} } } + let leftover_is_empty = content.is_empty() && images.is_empty() && tool_calls.is_empty(); if !content.is_empty() { ollama_msg.x_insert("content", content)?; } @@ -119,7 +155,22 @@ impl OllamaAdapter { ollama_msg.x_insert("tool_calls", tool_calls)?; } - messages.push(ollama_msg); + if had_tool_responses && leftover_is_empty { + // The message carried only tool responses (a Tool-role message, or a + // user message whose embedded responses were all extracted); nothing is + // left for the carrying message to say, so it is omitted. + } else { + messages.push(ollama_msg); + } + + // Follow-up user message carrying the tool-result images. + if !tool_response_images.is_empty() { + messages.push(json!({ + "role": "user", + "content": TOOL_RESULT_IMAGES_LABEL, + "images": tool_response_images, + })); + } } // -- Tools @@ -166,3 +217,53 @@ pub(in crate::adapter::adapters) struct OllamaRequestParts { pub messages: Vec, pub tools: Option>, } + +// region: --- Support + +/// Resolve the text content of an Ollama `role:"tool"` message for a `ToolResponse`, +/// pushing its usable image parts (raw base64 only, no data URL) onto +/// `tool_response_images`, which ride in the follow-up `user` image message since +/// Ollama tool messages carry only text content. A response without `parts` keeps its +/// exact legacy text; when parts are present, the `tool_response_fallback_text` +/// placeholder rules apply, tracking only the images contributed by THIS response so +/// the "(see attached image)" placeholder is not emitted for a response whose own +/// parts were all skipped while an earlier response in the same message contributed +/// images. Shared by the Tool-role path and the user-embedded extraction path. +fn tool_response_content_text(tool_response: ToolResponse, tool_response_images: &mut Vec>) -> String { + let ToolResponse { content, parts, .. } = tool_response; + let parts = parts.unwrap_or_default(); + + if parts.is_empty() { + content + } else { + let images_count_before = tool_response_images.len(); + for binary in parts { + if binary.is_image() { + match binary.source { + // Note: Ollama native API expects raw base64 (no data URL). + BinarySource::Base64(data) => tool_response_images.push(data), + BinarySource::Url(_) => { + warn!("Ollama native API doesn't support image URLs; skipping tool-result image part"); + } + } + } else { + warn!( + "ToolResponse binary parts only support images for the Ollama adapter; skipping non-image part '{}'", + binary.content_type + ); + } + } + let has_own_images = tool_response_images.len() > images_count_before; + tool_response_fallback_text(content, has_own_images) + } +} + +// endregion: --- Support + +// region: --- Tests + +#[cfg(test)] +#[path = "adapter_shared_tests.rs"] +mod tests; + +// endregion: --- Tests diff --git a/src/adapter/adapters/ollama/adapter_shared_tests.rs b/src/adapter/adapters/ollama/adapter_shared_tests.rs new file mode 100644 index 00000000..a3f6f4c0 --- /dev/null +++ b/src/adapter/adapters/ollama/adapter_shared_tests.rs @@ -0,0 +1,347 @@ +type Result = core::result::Result>; // For tests. + +use super::*; +use crate::chat::{ChatMessage, MessageContent}; + +fn test_model_iden() -> ModelIden { + ModelIden::new(AdapterKind::Ollama, "llama3.2") +} + +/// Tool-result images cannot ride inside an Ollama `tool` message: the tool message +/// keeps its text (or the placeholder), and the images are carried by a follow-up +/// `user` message with the base64 `images` array. +#[test] +fn test_ollama_tool_response_image_parts_ride_in_followup_user_message() -> Result<()> { + // -- Setup & Fixtures + let tool_response = ToolResponse::new("call_1", "").with_parts([Binary::from_base64("image/png", "PNG64", None)]); + let chat_req = ChatRequest::new(vec![ChatMessage::from(tool_response)]); + + // -- Exec + let OllamaRequestParts { messages, .. } = OllamaAdapter::into_ollama_request_parts(&test_model_iden(), chat_req)?; + + // -- Check + assert_eq!(messages.len(), 2, "tool message + follow-up user image message"); + assert_eq!( + messages[0], + json!({"role": "tool", "content": "(see attached image)"}), + "image-only tool response must use the placeholder text" + ); + assert_eq!( + messages[1], + json!({ + "role": "user", + "content": "Attached image(s) from tool result:", + "images": ["PNG64"], + }) + ); + + Ok(()) +} + +/// The `"(see attached image)"` placeholder must track the images contributed by the +/// CURRENT tool response only: with two responses in one message where the first +/// contributes an image and the second is empty with only skipped parts, the second's +/// text must be `"(no tool output)"` (not `"(see attached image)"`), even though the +/// message-level image accumulator is non-empty from the first response. +/// +/// NOTE: this test previously pinned the old single-tool-message shape, where each +/// response's content overwrote the previous in the carrying message and only the +/// last survived. That overwrite was a bug, and the expectation here changed +/// deliberately with the fix: a Tool-role message carrying multiple `ToolResponse` +/// parts now emits one `role:"tool"` message per response, in part order. +#[test] +fn test_ollama_tool_response_placeholder_tracks_own_images_only() -> Result<()> { + // -- Setup & Fixtures + let first = ToolResponse::new("call_1", "").with_parts([Binary::from_base64("image/png", "PNG64", None)]); + // URL-based image parts are skipped by the Ollama native adapter, so this + // response contributes no usable image of its own. + let second = ToolResponse::new("call_2", "").with_parts([Binary::from_url( + "image/png", + "https://example.com/shot.png", + None, + )]); + let tool_msg = ChatMessage::tool(MessageContent::from_parts(vec![ + ContentPart::ToolResponse(first), + ContentPart::ToolResponse(second), + ])); + let chat_req = ChatRequest::new(vec![tool_msg]); + + // -- Exec + let OllamaRequestParts { messages, .. } = OllamaAdapter::into_ollama_request_parts(&test_model_iden(), chat_req)?; + + // -- Check + assert_eq!( + messages.len(), + 3, + "one tool message per response + follow-up user image message" + ); + assert_eq!( + messages[0]["content"], + json!("(see attached image)"), + "the first response contributed its own image, so it claims the placeholder" + ); + assert_eq!( + messages[1]["content"], + json!("(no tool output)"), + "a response whose own parts were all skipped must not claim an attached image" + ); + assert_eq!( + messages[2], + json!({ + "role": "user", + "content": "Attached image(s) from tool result:", + "images": ["PNG64"], + }), + "the first response's image still rides in the follow-up user message" + ); + + Ok(()) +} + +/// A Tool-role message carrying multiple `ToolResponse` parts (the +/// `ChatMessage::from(Vec)` shape) emits ONE `role:"tool"` message PER +/// response, in part order, matching the per-response messages the other serializers +/// emit (previously each response's content overwrote the previous in a single tool +/// message, so only the last response's text survived). +#[test] +fn test_ollama_tool_role_multiple_responses_emit_one_tool_message_each() -> Result<()> { + // -- Setup & Fixtures + let tool_msg = ChatMessage::from(vec![ + ToolResponse::new("call_1", "one"), + ToolResponse::new("call_2", "two"), + ]); + let chat_req = ChatRequest::new(vec![tool_msg]); + + // -- Exec + let OllamaRequestParts { messages, .. } = OllamaAdapter::into_ollama_request_parts(&test_model_iden(), chat_req)?; + + // -- Check + assert_eq!( + messages, + vec![ + json!({"role": "tool", "content": "one"}), + json!({"role": "tool", "content": "two"}), + ] + ); + + Ok(()) +} + +/// Images from MULTIPLE `ToolResponse` parts of one Tool-role message accumulate, in +/// part order, into the SINGLE labeled follow-up user image message (not one +/// follow-up per response), emitted after the per-response tool messages. +#[test] +fn test_ollama_tool_role_multiple_responses_images_accumulate_in_one_followup() -> Result<()> { + // -- Setup & Fixtures + let first = ToolResponse::new("call_1", "").with_parts([Binary::from_base64("image/png", "A64", None)]); + let second = ToolResponse::new("call_2", "done").with_parts([Binary::from_base64("image/png", "B64", None)]); + let chat_req = ChatRequest::new(vec![ChatMessage::from(vec![first, second])]); + + // -- Exec + let OllamaRequestParts { messages, .. } = OllamaAdapter::into_ollama_request_parts(&test_model_iden(), chat_req)?; + + // -- Check + assert_eq!( + messages, + vec![ + json!({"role": "tool", "content": "(see attached image)"}), + json!({"role": "tool", "content": "done"}), + json!({ + "role": "user", + "content": "Attached image(s) from tool result:", + "images": ["A64", "B64"], + }), + ] + ); + + Ok(()) +} + +/// Regression guard: a text-only `ToolResponse` keeps its legacy shape with no +/// follow-up user message — byte-identical on the wire, pinned via the serialized +/// string (`serde_json` runs with `preserve_order`, so key order is meaningful). +#[test] +fn test_ollama_tool_response_text_only_serializes_as_before() -> Result<()> { + // -- Setup & Fixtures + let chat_req = ChatRequest::new(vec![ChatMessage::from(ToolResponse::new("call_1", "42"))]); + + // -- Exec + let OllamaRequestParts { messages, .. } = OllamaAdapter::into_ollama_request_parts(&test_model_iden(), chat_req)?; + + // -- Check + assert_eq!(messages, vec![json!({"role": "tool", "content": "42"})]); + assert_eq!( + serde_json::to_string(&messages)?, + r#"[{"role":"tool","content":"42"}]"#, + "single-response wire bytes must stay identical" + ); + + Ok(()) +} + +/// A `ToolResponse` embedded in a User-role message (the Anthropic-style shape where +/// tool results ride as user content blocks) is extracted into a standalone +/// `role:"tool"` message emitted before the user message carrying the remaining +/// content (previously the response text was garbled into the user `content`, where +/// sibling text parts overwrote it). +#[test] +fn test_ollama_user_embedded_tool_response_extracted_before_user_message() -> Result<()> { + // -- Setup & Fixtures + let user_msg = ChatMessage::user(MessageContent::from_parts(vec![ + ContentPart::ToolResponse(ToolResponse::new("call_1", "sunny")), + ContentPart::from_text("thanks, now summarize"), + ])); + let chat_req = ChatRequest::new(vec![user_msg]); + + // -- Exec + let OllamaRequestParts { messages, .. } = OllamaAdapter::into_ollama_request_parts(&test_model_iden(), chat_req)?; + + // -- Check + assert_eq!( + messages, + vec![ + json!({"role": "tool", "content": "sunny"}), + json!({"role": "user", "content": "thanks, now summarize"}), + ] + ); + + Ok(()) +} + +/// Multiple `ToolResponse` parts embedded in one user message are ALL extracted, in +/// part order, before the remaining user message. +#[test] +fn test_ollama_user_embedded_tool_responses_all_extracted_in_order() -> Result<()> { + // -- Setup & Fixtures + let user_msg = ChatMessage::user(MessageContent::from_parts(vec![ + ContentPart::ToolResponse(ToolResponse::new("call_1", "one")), + ContentPart::from_text("both done"), + ContentPart::ToolResponse(ToolResponse::new("call_2", "two")), + ])); + let chat_req = ChatRequest::new(vec![user_msg]); + + // -- Exec + let OllamaRequestParts { messages, .. } = OllamaAdapter::into_ollama_request_parts(&test_model_iden(), chat_req)?; + + // -- Check + assert_eq!( + messages, + vec![ + json!({"role": "tool", "content": "one"}), + json!({"role": "tool", "content": "two"}), + json!({"role": "user", "content": "both done"}), + ] + ); + + Ok(()) +} + +/// A user message carrying ONLY an embedded `ToolResponse` has nothing left to say +/// after the extraction, so the now-empty user message is omitted. +#[test] +fn test_ollama_user_embedded_tool_response_only_omits_empty_user_message() -> Result<()> { + // -- Setup & Fixtures + let user_msg = ChatMessage::user(MessageContent::from_parts(vec![ContentPart::ToolResponse( + ToolResponse::new("call_1", "sunny"), + )])); + let chat_req = ChatRequest::new(vec![user_msg]); + + // -- Exec + let OllamaRequestParts { messages, .. } = OllamaAdapter::into_ollama_request_parts(&test_model_iden(), chat_req)?; + + // -- Check + assert_eq!(messages, vec![json!({"role": "tool", "content": "sunny"})]); + + Ok(()) +} + +/// Image parts of a user-embedded `ToolResponse` ride the same follow-up user image +/// message as Tool-role responses (label + base64 `images` array), emitted after the +/// remaining user message; the extracted tool message uses the placeholder text. +#[test] +fn test_ollama_user_embedded_tool_response_image_rides_followup_user_message() -> Result<()> { + // -- Setup & Fixtures + let tool_response = ToolResponse::new("call_1", "").with_parts([Binary::from_base64("image/png", "PNG64", None)]); + let user_msg = ChatMessage::user(MessageContent::from_parts(vec![ + ContentPart::ToolResponse(tool_response), + ContentPart::from_text("what is in the screenshot?"), + ])); + let chat_req = ChatRequest::new(vec![user_msg]); + + // -- Exec + let OllamaRequestParts { messages, .. } = OllamaAdapter::into_ollama_request_parts(&test_model_iden(), chat_req)?; + + // -- Check + assert_eq!( + messages, + vec![ + json!({"role": "tool", "content": "(see attached image)"}), + json!({"role": "user", "content": "what is in the screenshot?"}), + json!({ + "role": "user", + "content": "Attached image(s) from tool result:", + "images": ["PNG64"], + }), + ] + ); + + Ok(()) +} + +/// Regression guard: a plain user message (text and image parts, no embedded tool +/// response) keeps its legacy single-message shape. +#[test] +fn test_ollama_plain_user_message_serializes_as_before() -> Result<()> { + // -- Setup & Fixtures + let user_msg = ChatMessage::user(MessageContent::from_parts(vec![ + ContentPart::from_text("describe "), + ContentPart::from_text("this"), + ContentPart::Binary(Binary::from_base64("image/png", "IMG64", None)), + ])); + let chat_req = ChatRequest::new(vec![user_msg]); + + // -- Exec + let OllamaRequestParts { messages, .. } = OllamaAdapter::into_ollama_request_parts(&test_model_iden(), chat_req)?; + + // -- Check + assert_eq!( + messages, + vec![json!({"role": "user", "content": "describe this", "images": ["IMG64"]})] + ); + + Ok(()) +} + +/// A `ToolResponse` embedded in an Assistant message has no representation on any +/// provider wire (there is no "tool result authored by the assistant"), so the +/// serializer must reject the shape with a hard error instead of garbling it into +/// assistant content (the previous behavior). +#[test] +fn test_ollama_assistant_embedded_tool_response_is_rejected() -> Result<()> { + // -- Setup & Fixtures + let assistant_msg = ChatMessage::assistant(MessageContent::from_parts(vec![ + ContentPart::from_text("checking"), + ContentPart::ToolResponse(ToolResponse::new("call_1", "sunny")), + ])); + let chat_req = ChatRequest::new(vec![ChatMessage::user("weather?"), assistant_msg]); + + // -- Exec + let err = OllamaAdapter::into_ollama_request_parts(&test_model_iden(), chat_req) + .map(|_| ()) + .expect_err("assistant-embedded tool response must fail serialization"); + + // -- Check + let Error::MessageContentTypeNotSupported { cause, .. } = err else { + return Err(format!("expected MessageContentTypeNotSupported, got: {err}").into()); + }; + assert!( + cause.contains("Assistant-role message"), + "cause must name the unsupported shape: {cause}" + ); + assert!( + cause.contains("Tool-role message"), + "cause must point at the supported Tool-role shape: {cause}" + ); + + Ok(()) +} diff --git a/src/adapter/adapters/ollama_cloud/adapter_impl.rs b/src/adapter/adapters/ollama_cloud/adapter_impl.rs index 9ad468fc..aa61eacf 100644 --- a/src/adapter/adapters/ollama_cloud/adapter_impl.rs +++ b/src/adapter/adapters/ollama_cloud/adapter_impl.rs @@ -56,7 +56,7 @@ impl Adapter for OllamaCloudAdapter { } = target; let api_key = get_api_key(auth, &model)?; let url = OllamaAdapter::get_service_url(&model, service_type, endpoint)?; - let OllamaRequestParts { messages, tools } = OllamaAdapter::into_ollama_request_parts(chat_req)?; + let OllamaRequestParts { messages, tools } = OllamaAdapter::into_ollama_request_parts(&model, chat_req)?; let mut options = json!({}); if let Some(temperature) = chat_options.temperature() { @@ -122,8 +122,9 @@ impl Adapter for OllamaCloudAdapter { model_iden: ModelIden, reqwest_builder: RequestBuilder, options_set: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result { - OllamaAdapter::to_chat_stream(model_iden, reqwest_builder, options_set) + OllamaAdapter::to_chat_stream(model_iden, reqwest_builder, options_set, response_observer) } fn to_embed_request_data( diff --git a/src/adapter/adapters/omlx/adapter_impl.rs b/src/adapter/adapters/omlx/adapter_impl.rs index fda31e86..c42f873f 100644 --- a/src/adapter/adapters/omlx/adapter_impl.rs +++ b/src/adapter/adapters/omlx/adapter_impl.rs @@ -89,8 +89,9 @@ impl Adapter for OmlxAdapter { model_iden: ModelIden, reqwest_builder: RequestBuilder, options_set: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result { - OpenAIAdapter::to_chat_stream(model_iden, reqwest_builder, options_set) + OpenAIAdapter::to_chat_stream(model_iden, reqwest_builder, options_set, response_observer) } fn to_embed_request_data( diff --git a/src/adapter/adapters/openai/adapter_impl.rs b/src/adapter/adapters/openai/adapter_impl.rs index 4fc06e3c..e8714f6e 100644 --- a/src/adapter/adapters/openai/adapter_impl.rs +++ b/src/adapter/adapters/openai/adapter_impl.rs @@ -145,8 +145,9 @@ impl Adapter for OpenAIAdapter { model_iden: ModelIden, reqwest_builder: RequestBuilder, options_sets: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result { - let event_source = EventSourceStream::new(reqwest_builder); + let event_source = EventSourceStream::new(reqwest_builder).with_response_observer(response_observer); let openai_stream = OpenAIStreamer::new(event_source, model_iden.clone(), options_sets); let chat_stream = ChatStream::from_inter_stream(openai_stream); diff --git a/src/adapter/adapters/openai/adapter_shared.rs b/src/adapter/adapters/openai/adapter_shared.rs index b904469a..d9682122 100644 --- a/src/adapter/adapters/openai/adapter_shared.rs +++ b/src/adapter/adapters/openai/adapter_shared.rs @@ -3,10 +3,13 @@ use super::cache_policy::{OpenAiPromptCachePolicy, OpenAiProtocol, is_gpt_5_6_or_later, openai_prompt_cache_policy}; use super::schema::{OpenAiResponseFormatPlan, response_format_plan, tool_parameters_schema}; use crate::adapter::adapters::openai::OpenAIAdapter; -use crate::adapter::adapters::support::get_api_key; +use crate::adapter::adapters::support::{ + TOOL_RESULT_IMAGES_LABEL, assistant_embedded_tool_response_err, get_api_key, tool_response_fallback_text, +}; use crate::adapter::{AdapterDispatcher, AdapterKind, ServiceType, WebRequestData}; use crate::chat::{ - BinarySource, CacheControl, ChatOptionsSet, ChatRequest, ChatRole, ContentPart, ReasoningEffort, ToolChoice, Usage, + BinarySource, CacheControl, ChatOptionsSet, ChatRequest, ChatRole, ContentPart, ReasoningEffort, ToolChoice, + ToolResponse, Usage, }; use crate::resolver::{AuthData, Endpoint}; use crate::webc::WebClient; @@ -304,8 +307,23 @@ impl OpenAIAdapter { messages.push(json!({"role": "system", "content": system_msg})); } + // Images attached to tool responses (`ToolResponse.parts`) cannot ride inside a + // Chat Completions `tool` message, so they are carried by a follow-up `user` + // message. Images from a run of consecutive Tool messages are batched into one + // trailing user message, emitted before the next non-tool message. + let mut pending_tool_images: Vec = Vec::new(); + // -- Process the messages for msg in chat_req.messages { + // Index of the tool-image flush message emitted for this iteration (if any), + // so tool messages extracted from an embedded `ToolResponse` can be inserted + // before it, adjacent to the tool-message run they belong to. + let mut flushed_images_at: Option = None; + if !matches!(msg.role, ChatRole::Tool) && !pending_tool_images.is_empty() { + flushed_images_at = Some(messages.len()); + messages.push(tool_images_user_message(std::mem::take(&mut pending_tool_images))); + } + let cache_controlled = cache_policy.is_some() && msg .options @@ -338,6 +356,13 @@ impl OpenAIAdapter { messages.push(json! ({"role": "user", "content": content})); } else { let mut values: Vec = Vec::new(); + // Tool responses embedded in this user message (the Anthropic-style + // shape where tool results ride as user-message content blocks) are + // extracted into proper `role:"tool"` messages emitted BEFORE this + // user message: such a user message conventionally directly follows + // the assistant `tool_calls` message, and the Chat Completions wire + // requires tool messages to sit adjacent to it. + let mut embedded_tool_messages: Vec = Vec::new(); for part in msg.content { match part { ContentPart::Text(content) => values.push(json!({"type": "text", "text": content})), @@ -398,17 +423,39 @@ impl OpenAIAdapter { // continue would allow to gracefully skip pushing unserializable message // TODO: Probably need to warn if it is a ToolCalls type of content ContentPart::ToolCall(_) => (), - ContentPart::ToolResponse(_) => (), + ContentPart::ToolResponse(tool_response) => { + // Extracted as a `role:"tool"` message before this user message + // (see `embedded_tool_messages` above). Image parts are folded + // into this same user message as `image_url` blocks, mirroring + // the Gemini serializer's user-embedded handling. + let mut rescued_images: Vec = Vec::new(); + embedded_tool_messages + .push(tool_response_to_tool_message(tool_response, &mut rescued_images)); + values.extend(rescued_images); + } ContentPart::ThoughtSignature(_) => (), ContentPart::ReasoningContent(_) => (), // Custom are ignored for this logic ContentPart::Custom(_) => {} } } - if cache_controlled { - apply_chat_cache_breakpoint(model_iden, &mut values, "message")?; + let had_embedded_tool_responses = !embedded_tool_messages.is_empty(); + if had_embedded_tool_responses { + // Insert before the tool-image flush message emitted for this + // iteration (if any), so the extracted tool messages stay adjacent + // to the preceding tool-message run. + let insert_at = flushed_images_at.unwrap_or(messages.len()); + messages.splice(insert_at..insert_at, embedded_tool_messages); + } + if values.is_empty() && had_embedded_tool_responses { + // The user message carried only embedded tool responses; nothing is + // left for it to say, so the now-empty user message is omitted. + } else { + if cache_controlled { + apply_chat_cache_breakpoint(model_iden, &mut values, "message")?; + } + messages.push(json! ({"role": "user", "content": values})); } - messages.push(json! ({"role": "user", "content": values})); } } @@ -436,7 +483,12 @@ impl OpenAIAdapter { // TODO: Probably need towarn on this one (probably need to add binary here) ContentPart::Binary(_) => (), - ContentPart::ToolResponse(_) => (), + // No provider wire represents a tool result authored by the + // assistant; fail loudly instead of dropping the content or + // inventing a placement (use a Tool-role message). + ContentPart::ToolResponse(_) => { + return Err(assistant_embedded_tool_response_err(model_iden)); + } ContentPart::ThoughtSignature(_) => {} // Custom are ignored for this logic ContentPart::Custom(_) => {} @@ -469,11 +521,7 @@ impl OpenAIAdapter { ChatRole::Tool => { for part in msg.content { if let ContentPart::ToolResponse(tool_response) = part { - messages.push(json!({ - "role": "tool", - "content": tool_response.content, - "tool_call_id": tool_response.call_id, - })) + messages.push(tool_response_to_tool_message(tool_response, &mut pending_tool_images)); } } @@ -482,6 +530,11 @@ impl OpenAIAdapter { } } + // Flush tool-result images from a trailing run of Tool messages. + if !pending_tool_images.is_empty() { + messages.push(tool_images_user_message(pending_tool_images)); + } + // -- Process the tools let tools = chat_req.tools.map(|tools| { tools @@ -593,6 +646,63 @@ struct OpenAIRequestParts { tools: Option>, } +/// Serialize a `ToolResponse` into a Chat Completions `role:"tool"` message. +/// +/// The tool message content is text-only on this wire, so image parts are +/// rescued as `image_url` blocks appended to `rescued_images`, and the tool +/// message keeps the response text (or the `tool_response_fallback_text` +/// placeholder rules when parts are present). The caller decides where the +/// rescued images ride: the batched follow-up user message for Tool-role +/// messages, or folded into the carrying user message for user-embedded +/// responses. +fn tool_response_to_tool_message(tool_response: ToolResponse, rescued_images: &mut Vec) -> Value { + let ToolResponse { + call_id, + content, + parts, + .. + } = tool_response; + let parts = parts.unwrap_or_default(); + + if parts.is_empty() { + json!({ + "role": "tool", + "content": content, + "tool_call_id": call_id, + }) + } else { + let mut image_values: Vec = Vec::new(); + for binary in parts { + if binary.is_image() { + let image_url = binary.into_url(); + image_values.push(json!({"type": "image_url", "image_url": {"url": image_url}})); + } else { + warn!( + "ToolResponse binary parts only support images for OpenAI-compatible adapters; skipping non-image part '{}'", + binary.content_type + ); + } + } + let content = tool_response_fallback_text(content, !image_values.is_empty()); + rescued_images.extend(image_values); + json!({ + "role": "tool", + "content": content, + "tool_call_id": call_id, + }) + } +} + +/// Build the follow-up `user` message that carries tool-result images +/// (`ToolResponse.parts`), since Chat Completions `tool` message content +/// cannot include image blocks. +fn tool_images_user_message(image_values: Vec) -> Value { + let mut values: Vec = Vec::with_capacity(image_values.len() + 1); + values.push(json!({"type": "text", "text": TOOL_RESULT_IMAGES_LABEL})); + values.extend(image_values); + json!({"role": "user", "content": values}) +} + fn apply_chat_cache_breakpoint(_model_iden: &ModelIden, content: &mut [Value], _scope: &'static str) -> Result<()> { let Some(content_block) = content.iter_mut().rev().find(|value| { matches!( diff --git a/src/adapter/adapters/openai/adapter_shared_tests.rs b/src/adapter/adapters/openai/adapter_shared_tests.rs index 2d6b0953..f78d2b95 100644 --- a/src/adapter/adapters/openai/adapter_shared_tests.rs +++ b/src/adapter/adapters/openai/adapter_shared_tests.rs @@ -1,8 +1,8 @@ use super::{OpenAIAdapter, ToWebRequestDataOptions}; use crate::adapter::AdapterKind; use crate::chat::{ - CacheControl, ChatMessage, ChatOptions, ChatOptionsSet, ChatRequest, ContentPart, MessageContent, Tool, ToolCall, - ToolChoice, + Binary, CacheControl, ChatMessage, ChatOptions, ChatOptionsSet, ChatRequest, ContentPart, MessageContent, Tool, + ToolCall, ToolChoice, ToolResponse, }; use crate::resolver::{AuthData, Endpoint}; use crate::{ModelIden, ServiceTarget}; @@ -450,6 +450,338 @@ fn test_managed_body_thinking_uses_model_name_derived_effort() -> Result<()> { // endregion: --- Managed Thinking +/// Tool-result images cannot ride inside a Chat Completions `tool` message: the tool +/// message keeps its text, and the images from a run of consecutive tool messages are +/// batched into ONE follow-up `user` message, emitted before the next non-tool message. +#[test] +fn test_tool_response_image_parts_batched_into_followup_user_message() -> Result<()> { + // -- Setup & Fixtures + let tool_response_1 = ToolResponse::new("call_1", "screenshot taken").with_parts([Binary::from_base64( + "image/png", + "BASE64PNG", + None, + )]); + let tool_response_2 = + ToolResponse::new("call_2", "chart built").with_parts([Binary::from_base64("image/jpeg", "BASE64JPEG", None)]); + let chat_req = ChatRequest::new(vec![ + ChatMessage::from(tool_response_1), + ChatMessage::from(tool_response_2), + ChatMessage::user("continue"), + ]); + + // -- Exec + let web_req = OpenAIAdapter::util_to_web_request_data( + target("gpt-4o-mini"), + crate::adapter::ServiceType::Chat, + chat_req, + ChatOptionsSet::default(), + None, + )?; + + // -- Check + let messages = web_req.payload["messages"].as_array().ok_or("messages should be an array")?; + assert_eq!(messages.len(), 4, "2 tool + 1 batched image user + 1 user"); + assert_eq!( + messages[0], + json!({"role": "tool", "content": "screenshot taken", "tool_call_id": "call_1"}) + ); + assert_eq!( + messages[1], + json!({"role": "tool", "content": "chart built", "tool_call_id": "call_2"}) + ); + assert_eq!( + messages[2], + json!({ + "role": "user", + "content": [ + {"type": "text", "text": "Attached image(s) from tool result:"}, + {"type": "image_url", "image_url": {"url": "data:image/png;base64,BASE64PNG"}}, + {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,BASE64JPEG"}}, + ] + }), + "images from the run of tool messages must batch into one follow-up user message" + ); + assert_eq!(messages[3], json!({"role": "user", "content": "continue"})); + + Ok(()) +} + +/// An image-only tool response (no text) gets the "(see attached image)" placeholder +/// as the tool message content. +#[test] +fn test_tool_response_image_only_uses_placeholder_text() -> Result<()> { + // -- Setup & Fixtures + let tool_response = ToolResponse::new("call_1", "").with_parts([Binary::from_base64("image/png", "PNG64", None)]); + let chat_req = ChatRequest::new(vec![ChatMessage::from(tool_response)]); + + // -- Exec + let web_req = OpenAIAdapter::util_to_web_request_data( + target("gpt-4o-mini"), + crate::adapter::ServiceType::Chat, + chat_req, + ChatOptionsSet::default(), + None, + )?; + + // -- Check + assert_eq!( + web_req.payload["messages"][0], + json!({"role": "tool", "content": "(see attached image)", "tool_call_id": "call_1"}) + ); + assert_eq!( + web_req.payload["messages"][1]["content"][1]["image_url"]["url"], + json!("data:image/png;base64,PNG64") + ); + + Ok(()) +} + +/// Regression guard: a text-only `ToolResponse` must serialize exactly as before, +/// with no follow-up user message. +#[test] +fn test_tool_response_text_only_serializes_as_before() -> Result<()> { + // -- Setup & Fixtures + let chat_req = ChatRequest::new(vec![ChatMessage::from(ToolResponse::new("call_1", "42"))]); + + // -- Exec + let web_req = OpenAIAdapter::util_to_web_request_data( + target("gpt-4o-mini"), + crate::adapter::ServiceType::Chat, + chat_req, + ChatOptionsSet::default(), + None, + )?; + + // -- Check + let messages = web_req.payload["messages"].as_array().ok_or("messages should be an array")?; + assert_eq!(messages.len(), 1, "no follow-up user message for text-only"); + assert_eq!( + messages[0], + json!({"role": "tool", "content": "42", "tool_call_id": "call_1"}) + ); + + Ok(()) +} + +// region: --- Embedded Tool Responses + +/// A `ToolResponse` embedded in a User-role message (Anthropic-style user-carried +/// tool result) must be extracted as a proper `role:"tool"` message placed BEFORE +/// the user message carrying the remaining content, adjacent to the assistant +/// `tool_calls` message that conventionally precedes it. +#[test] +fn test_user_embedded_tool_response_extracted_before_user_message() -> Result<()> { + // -- Setup & Fixtures + let assistant_msg = ChatMessage::assistant(MessageContent::from_parts(vec![ContentPart::ToolCall(ToolCall { + call_id: "call_1".to_string(), + fn_name: "get_weather".to_string(), + fn_arguments: json!({"city": "Paris"}), + thought_signatures: None, + })])); + let user_msg = ChatMessage::user(vec![ + ContentPart::ToolResponse(ToolResponse::new("call_1", "sunny")), + ContentPart::from_text("thanks, and tomorrow?"), + ]); + let chat_req = ChatRequest::new(vec![assistant_msg, user_msg]); + + // -- Exec + let web_req = OpenAIAdapter::util_to_web_request_data( + target("gpt-4o-mini"), + crate::adapter::ServiceType::Chat, + chat_req, + ChatOptionsSet::default(), + None, + )?; + + // -- Check + let messages = web_req.payload["messages"].as_array().ok_or("messages should be an array")?; + assert_eq!(messages.len(), 3, "assistant + extracted tool + user"); + assert_eq!(messages[0]["role"], "assistant"); + assert_eq!( + messages[1], + json!({"role": "tool", "content": "sunny", "tool_call_id": "call_1"}), + "embedded tool response must become a role:\"tool\" message before the user message" + ); + assert_eq!( + messages[2], + json!({"role": "user", "content": [{"type": "text", "text": "thanks, and tomorrow?"}]}) + ); + + Ok(()) +} + +/// Image parts of a user-embedded `ToolResponse` fold into the SAME user message +/// (as `image_url` blocks), mirroring the Gemini serializer's user-embedded +/// handling, while the extracted tool message keeps the text. +#[test] +fn test_user_embedded_tool_response_image_part_folds_into_user_message() -> Result<()> { + // -- Setup & Fixtures + let tool_response = + ToolResponse::new("call_1", "screenshot taken").with_parts([Binary::from_base64("image/png", "PNG64", None)]); + let user_msg = ChatMessage::user(vec![ + ContentPart::ToolResponse(tool_response), + ContentPart::from_text("what do you see?"), + ]); + let chat_req = ChatRequest::new(vec![user_msg]); + + // -- Exec + let web_req = OpenAIAdapter::util_to_web_request_data( + target("gpt-4o-mini"), + crate::adapter::ServiceType::Chat, + chat_req, + ChatOptionsSet::default(), + None, + )?; + + // -- Check + let messages = web_req.payload["messages"].as_array().ok_or("messages should be an array")?; + assert_eq!(messages.len(), 2, "extracted tool + user (no separate image message)"); + assert_eq!( + messages[0], + json!({"role": "tool", "content": "screenshot taken", "tool_call_id": "call_1"}) + ); + assert_eq!( + messages[1], + json!({ + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": "data:image/png;base64,PNG64"}}, + {"type": "text", "text": "what do you see?"}, + ] + }), + "the rescued image must ride in the same user message, without a label message" + ); + + Ok(()) +} + +/// A user message whose content is ONLY embedded tool responses leaves nothing to +/// carry: the tool messages are extracted (multiple, in order) and the now-empty +/// user message is omitted. `call_id`s are serialized as-is (no matching validation). +#[test] +fn test_user_message_with_only_embedded_tool_responses_omits_user_message() -> Result<()> { + // -- Setup & Fixtures + let user_msg = ChatMessage::user(vec![ + ContentPart::ToolResponse(ToolResponse::new("call_1", "42")), + ContentPart::ToolResponse(ToolResponse::new("call_unmatched", "43")), + ]); + let chat_req = ChatRequest::new(vec![user_msg]); + + // -- Exec + let web_req = OpenAIAdapter::util_to_web_request_data( + target("gpt-4o-mini"), + crate::adapter::ServiceType::Chat, + chat_req, + ChatOptionsSet::default(), + None, + )?; + + // -- Check + let messages = web_req.payload["messages"].as_array().ok_or("messages should be an array")?; + assert_eq!(messages.len(), 2, "only the two extracted tool messages"); + assert_eq!( + messages[0], + json!({"role": "tool", "content": "42", "tool_call_id": "call_1"}) + ); + assert_eq!( + messages[1], + json!({"role": "tool", "content": "43", "tool_call_id": "call_unmatched"}) + ); + + Ok(()) +} + +/// When a user message with an embedded `ToolResponse` follows a Tool-role run +/// whose images are pending, the extracted tool message must land BEFORE the +/// batched tool-images user message, keeping it adjacent to the tool-message run +/// (the wire rejects a tool message that follows a user message). +#[test] +fn test_user_embedded_tool_response_stays_adjacent_to_tool_run() -> Result<()> { + // -- Setup & Fixtures + let tool_msg = ChatMessage::from( + ToolResponse::new("call_1", "screenshot taken").with_parts([Binary::from_base64("image/png", "PNG64", None)]), + ); + let user_msg = ChatMessage::user(vec![ + ContentPart::ToolResponse(ToolResponse::new("call_2", "done")), + ContentPart::from_text("go on"), + ]); + let chat_req = ChatRequest::new(vec![tool_msg, user_msg]); + + // -- Exec + let web_req = OpenAIAdapter::util_to_web_request_data( + target("gpt-4o-mini"), + crate::adapter::ServiceType::Chat, + chat_req, + ChatOptionsSet::default(), + None, + )?; + + // -- Check + let messages = web_req.payload["messages"].as_array().ok_or("messages should be an array")?; + assert_eq!(messages.len(), 4, "tool + extracted tool + image flush + user"); + assert_eq!(messages[0]["role"], "tool"); + assert_eq!(messages[0]["tool_call_id"], "call_1"); + assert_eq!( + messages[1], + json!({"role": "tool", "content": "done", "tool_call_id": "call_2"}), + "extracted tool message must come before the batched tool-images user message" + ); + assert_eq!(messages[2]["content"][0]["text"], "Attached image(s) from tool result:"); + assert_eq!( + messages[3], + json!({"role": "user", "content": [{"type": "text", "text": "go on"}]}) + ); + + Ok(()) +} + +/// A `ToolResponse` embedded in an Assistant message has no representation on any +/// provider wire (there is no "tool result authored by the assistant"), so the +/// serializer must reject the shape with a hard error instead of dropping the +/// content or inventing a placement. +#[test] +fn test_assistant_embedded_tool_response_is_rejected() -> Result<()> { + // -- Setup & Fixtures + let assistant_msg = ChatMessage::assistant(MessageContent::from_parts(vec![ + ContentPart::from_text("checking"), + ContentPart::ToolCall(ToolCall { + call_id: "call_1".to_string(), + fn_name: "get_weather".to_string(), + fn_arguments: json!({"city": "Paris"}), + thought_signatures: None, + }), + ContentPart::ToolResponse(ToolResponse::new("call_1", "sunny")), + ])); + let chat_req = ChatRequest::new(vec![ChatMessage::user("weather?"), assistant_msg]); + + // -- Exec + let err = OpenAIAdapter::util_to_web_request_data( + target("gpt-4o-mini"), + crate::adapter::ServiceType::Chat, + chat_req, + ChatOptionsSet::default(), + None, + ) + .expect_err("assistant-embedded tool response must fail serialization"); + + // -- Check + let crate::Error::MessageContentTypeNotSupported { cause, .. } = err else { + return Err(format!("expected MessageContentTypeNotSupported, got: {err}").into()); + }; + assert!( + cause.contains("Assistant-role message"), + "cause must name the unsupported shape: {cause}" + ); + assert!( + cause.contains("Tool-role message"), + "cause must point at the supported Tool-role shape: {cause}" + ); + + Ok(()) +} + +// endregion: --- Embedded Tool Responses + // region: --- Support fn test_model() -> ModelIden { diff --git a/src/adapter/adapters/openai_resp/adapter_impl.rs b/src/adapter/adapters/openai_resp/adapter_impl.rs index 99a4b219..b182e3b5 100644 --- a/src/adapter/adapters/openai_resp/adapter_impl.rs +++ b/src/adapter/adapters/openai_resp/adapter_impl.rs @@ -7,11 +7,13 @@ use crate::adapter::adapters::openai::cache_policy::{ use crate::adapter::adapters::openai::schema::{ OpenAiResponseFormatPlan, response_format_plan, tool_parameters_schema, }; -use crate::adapter::adapters::support::get_api_key; +use crate::adapter::adapters::support::{ + TOOL_RESULT_IMAGES_LABEL, assistant_embedded_tool_response_err, get_api_key, tool_response_fallback_text, +}; use crate::adapter::{Adapter, AdapterDispatcher, AdapterKind, ServiceType, WebRequestData}; use crate::chat::{ CacheControl, ChatOptionsSet, ChatRequest, ChatResponse, ChatRole, ChatStream, ChatStreamResponse, ContentPart, - MessageContent, ReasoningEffort, StopReason, Tool, ToolChoice, ToolConfig, ToolName, Usage, + MessageContent, ReasoningEffort, StopReason, Tool, ToolChoice, ToolConfig, ToolName, ToolResponse, Usage, }; use crate::resolver::{AuthData, Endpoint}; use crate::webc::{EventSourceStream, WebClient, WebResponse}; @@ -20,6 +22,7 @@ use crate::{ModelIden, ServiceTarget}; use reqwest::RequestBuilder; use serde_json::{Map, Value, json}; use std::collections::BTreeSet; +use tracing::warn; use value_ext::JsonValueExt; pub struct OpenAIRespAdapter; @@ -342,8 +345,9 @@ impl Adapter for OpenAIRespAdapter { model_iden: ModelIden, reqwest_builder: RequestBuilder, options_sets: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result { - let event_source = EventSourceStream::new(reqwest_builder); + let event_source = EventSourceStream::new(reqwest_builder).with_response_observer(response_observer); let openai_stream = OpenAIRespStreamer::new(event_source, model_iden.clone(), options_sets); let chat_stream = ChatStream::from_inter_stream(openai_stream); @@ -430,8 +434,24 @@ impl OpenAIRespAdapter { let mut unamed_file_count = 0; + // Images rescued from custom tool outputs (`custom_tool_call_output.output` is a + // raw string on the wire, so it cannot carry `input_image` blocks). They ride in a + // follow-up `user` message input item. Images from a run of consecutive Tool + // messages are batched into one trailing item, emitted before the next non-tool + // message (same batching as the Chat Completions serializer). + let mut pending_custom_tool_images: Vec = Vec::new(); + // -- Process the messages for msg in chat_req.messages { + // Index of the tool-image flush item emitted for this iteration (if any), so + // output items extracted from an embedded `ToolResponse` can be inserted + // before it, adjacent to the tool-message run they belong to. + let mut flushed_images_at: Option = None; + if !matches!(msg.role, ChatRole::Tool) && !pending_custom_tool_images.is_empty() { + flushed_images_at = Some(input_items.len()); + input_items.push(tool_images_user_item(std::mem::take(&mut pending_custom_tool_images))); + } + let cache_controlled = cache_policy.is_some() && msg .options @@ -464,6 +484,13 @@ impl OpenAIRespAdapter { input_items.push(json! ({"role": "user", "content": content})); } else { let mut values: Vec = Vec::new(); + // Tool responses embedded in this user message (the Anthropic-style + // shape where tool results ride as user-message content blocks) are + // extracted into proper output items (`function_call_output`, or + // `custom_tool_call_output` for custom tool calls) emitted BEFORE + // this user message item, translating the conventional + // assistant-tool_calls -> user-carried-results ordering. + let mut embedded_output_items: Vec = Vec::new(); for part in msg.content { match part { @@ -518,17 +545,44 @@ impl OpenAIRespAdapter { // continue would allow to gracefully skip pushing unserializable message // TODO: Probably need to warn if it is a ToolCalls type of content ContentPart::ToolCall(_) => (), - ContentPart::ToolResponse(_) => (), + ContentPart::ToolResponse(tool_response) => { + // Extracted as an output item before this user message item + // (see `embedded_output_items` above). Function outputs carry + // their images natively in the output array; images rescued + // from custom outputs are folded into this same user message + // as `input_image` items, mirroring the Gemini serializer's + // user-embedded handling. + let mut rescued_images: Vec = Vec::new(); + embedded_output_items.push(tool_response_to_output_item( + tool_response, + &custom_call_ids, + &mut rescued_images, + )); + values.extend(rescued_images); + } ContentPart::ThoughtSignature(_) => (), ContentPart::ReasoningContent(_) => (), // Custom are ignored for this logic ContentPart::Custom(_) => {} } } - if cache_controlled { - apply_resp_cache_breakpoint(model_iden, &mut values, "message")?; + let had_embedded_tool_responses = !embedded_output_items.is_empty(); + if had_embedded_tool_responses { + // Insert before the tool-image flush item emitted for this + // iteration (if any), so the extracted output items stay adjacent + // to the preceding tool-message run. + let insert_at = flushed_images_at.unwrap_or(input_items.len()); + input_items.splice(insert_at..insert_at, embedded_output_items); + } + if values.is_empty() && had_embedded_tool_responses { + // The user message carried only embedded tool responses; nothing is + // left for it to say, so the now-empty user message item is omitted. + } else { + if cache_controlled { + apply_resp_cache_breakpoint(model_iden, &mut values, "message")?; + } + input_items.push(json! ({"role": "user", "content": values})); } - input_items.push(json! ({"role": "user", "content": values})); } } @@ -614,7 +668,12 @@ impl OpenAIRespAdapter { // TODO: Probably need towarn on this one (probably need to add binary here) ContentPart::Binary(_) => {} - ContentPart::ToolResponse(_) => {} + // No provider wire represents a tool result authored by the + // assistant; fail loudly instead of dropping the content or + // inventing a placement (use a Tool-role message). + ContentPart::ToolResponse(_) => { + return Err(assistant_embedded_tool_response_err(model_iden)); + } // ThoughtSignature and ReasoningContent are emitted as // top-level `type:reasoning` items in the pre-pass above. ContentPart::ThoughtSignature(_) => {} @@ -638,16 +697,11 @@ impl OpenAIRespAdapter { ChatRole::Tool => { for part in msg.content { if let ContentPart::ToolResponse(tool_response) = part { - let response_type = if custom_call_ids.contains(&tool_response.call_id) { - "custom_tool_call_output" - } else { - "function_call_output" - }; - input_items.push(json!({ - "type": response_type, - "call_id": tool_response.call_id, - "output": tool_response.content, - })); + input_items.push(tool_response_to_output_item( + tool_response, + &custom_call_ids, + &mut pending_custom_tool_images, + )); } } @@ -656,6 +710,11 @@ impl OpenAIRespAdapter { } } + // Flush custom-tool-result images from a trailing run of Tool messages. + if !pending_custom_tool_images.is_empty() { + input_items.push(tool_images_user_item(pending_custom_tool_images)); + } + // -- Process the tools let tools = chat_req .tools @@ -731,6 +790,98 @@ struct OpenAIRespRequestParts { tools: Option>, } +/// Serialize a `ToolResponse` into a Responses API output item: +/// `custom_tool_call_output` when the `call_id` belongs to a custom tool call, +/// `function_call_output` otherwise (`call_id`s are not otherwise validated; +/// provider-side validation is the norm). +/// +/// The Responses API natively supports `output` as an array of `input_text` / +/// `input_image` items for function call outputs, so their image parts ride in +/// the output array. Custom tool outputs are raw strings (with the +/// `tool_response_fallback_text` placeholder rules), so their images are +/// rescued into `rescued_custom_images`; the caller decides where they ride +/// (the batched follow-up user message item for Tool-role messages, or folded +/// into the carrying user message for user-embedded responses). +fn tool_response_to_output_item( + tool_response: ToolResponse, + custom_call_ids: &BTreeSet, + rescued_custom_images: &mut Vec, +) -> Value { + let is_custom = custom_call_ids.contains(&tool_response.call_id); + let response_type = if is_custom { + "custom_tool_call_output" + } else { + "function_call_output" + }; + let ToolResponse { + call_id, + content, + parts, + .. + } = tool_response; + let parts = parts.unwrap_or_default(); + let has_parts = !parts.is_empty(); + + let mut image_values: Vec = Vec::new(); + for binary in parts { + if binary.is_image() { + image_values.push(json!({ + "type": "input_image", + "detail": "auto", + "image_url": binary.into_url(), + })); + } else { + warn!( + "ToolResponse binary parts only support images for the OpenAI Responses adapter; skipping non-image part '{}'", + binary.content_type + ); + } + } + + if is_custom { + // NOTE: The fallback text applies only when parts were present, so + // plain text-only responses keep their exact legacy serialization. + let output = if has_parts { + tool_response_fallback_text(content, !image_values.is_empty()) + } else { + content + }; + rescued_custom_images.extend(image_values); + json!({ + "type": response_type, + "call_id": call_id, + "output": output, + }) + } else if image_values.is_empty() { + json!({ + "type": response_type, + "call_id": call_id, + "output": content, + }) + } else { + let mut output: Vec = Vec::new(); + if !content.is_empty() { + output.push(json!({"type": "input_text", "text": content})); + } + output.extend(image_values); + json!({ + "type": response_type, + "call_id": call_id, + "output": output, + }) + } +} + +/// Build the follow-up `user` message input item that carries tool-result images +/// (`ToolResponse.parts`) rescued from custom tool outputs, since +/// `custom_tool_call_output.output` is a raw string and cannot include image blocks. +fn tool_images_user_item(image_values: Vec) -> Value { + let mut content: Vec = Vec::with_capacity(image_values.len() + 1); + content.push(json!({"type": "input_text", "text": TOOL_RESULT_IMAGES_LABEL})); + content.extend(image_values); + json!({"type": "message", "role": "user", "content": content}) +} + fn apply_resp_cache_breakpoint(_model_iden: &ModelIden, content: &mut [Value], _scope: &'static str) -> Result<()> { let Some(content_block) = content.iter_mut().rev().find(|value| { matches!( @@ -750,394 +901,7 @@ fn apply_resp_cache_breakpoint(_model_iden: &ModelIden, content: &mut [Value], _ // region: --- Tests #[cfg(test)] -mod tests { - type Result = core::result::Result>; - - use super::*; - use crate::adapter::AdapterKind; - use crate::chat::{ChatMessage, ChatOptions, JsonSpec, Tool, ToolCall, ToolChoice, ToolResponse}; - - #[test] - fn test_cache_control_without_eligible_content_does_not_fail_response_request() { - let target = ServiceTarget { - model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), - auth: AuthData::from_single("test-key"), - endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), - }; - let assistant_msg = ChatMessage::assistant(MessageContent::from_parts(vec![ContentPart::ToolCall(ToolCall { - call_id: "call_1".to_string(), - fn_name: "get_weather".to_string(), - fn_arguments: json!({}), - thought_signatures: None, - })])) - .with_options(CacheControl::Ephemeral); - let chat_req = ChatRequest::new(vec![ChatMessage::user("hello"), assistant_msg]); - - let web_req = - OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default()) - .expect("unsupported breakpoint placement should be ignored"); - - assert_eq!(web_req.payload["prompt_cache_options"]["mode"], "explicit"); - } - - #[test] - fn custom_grammar_tool_and_roundtrip_use_responses_native_items() { - let target = ServiceTarget { - model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), - auth: AuthData::from_single("test-key"), - endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), - }; - let patch = "*** Begin Patch\n*** Update File: source.c\n@@\n-old\n+new\n*** End Patch\n"; - let assistant = ChatMessage::assistant(vec![ToolCall { - call_id: "call_patch".to_string(), - fn_name: "apply_patch".to_string(), - fn_arguments: Value::String(patch.to_string()), - thought_signatures: None, - }]); - let response = ChatMessage::from(ToolResponse::new("call_patch", "Done!")); - let format = json!({ - "type": "grammar", - "syntax": "lark", - "definition": "start: PATCH", - }); - let request = ChatRequest::new(vec![ChatMessage::user("patch it"), assistant, response]).with_tools(vec![ - Tool::new("apply_patch") - .with_description("Apply a patch") - .with_custom_format(format.clone()), - ]); - - let web_req = - OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, request, ChatOptionsSet::default()) - .unwrap(); - assert_eq!( - web_req.payload["tools"][0], - json!({ - "type": "custom", - "name": "apply_patch", - "description": "Apply a patch", - "format": format, - }) - ); - let input = web_req.payload["input"].as_array().unwrap(); - assert!(input.iter().any(|item| { - item["type"] == "custom_tool_call" && item["call_id"] == "call_patch" && item["input"] == patch - })); - assert!(input.iter().any(|item| { - item["type"] == "custom_tool_call_output" && item["call_id"] == "call_patch" && item["output"] == "Done!" - })); - } - - #[test] - fn test_extra_body_merged_into_response_payload() { - let chat_options = ChatOptions::default() - .with_top_p(0.3) - .with_extra_body(json!({"top_p": 0.9, "metadata": {"source": "test"}})); - let options_set = ChatOptionsSet::default().with_chat_options(Some(&chat_options)); - let target = ServiceTarget { - model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5-mini"), - auth: AuthData::from_single("test-key"), - endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), - }; - - let web_req = OpenAIRespAdapter::to_web_request_data( - target, - ServiceType::Chat, - ChatRequest::from_user("hello"), - options_set, - ) - .expect("to_web_request_data should succeed"); - - assert_eq!(web_req.payload["top_p"], 0.9); - assert_eq!(web_req.payload["metadata"]["source"], "test"); - } - - #[test] - fn pydantic_union_schema_is_sanitized_for_responses() { - let schema = json!({ - "type": "object", - "properties": { - "animal": { - "discriminator": {"propertyName": "kind"}, - "oneOf": [{"$ref": "#/$defs/Cat"}, {"$ref": "#/$defs/Dog"}] - } - }, - "$defs": { - "Cat": { - "type": "object", - "properties": {"kind": {"const": "cat"}}, - "required": ["kind"] - }, - "Dog": { - "type": "object", - "properties": {"kind": {"const": "dog"}}, - "required": ["kind"] - } - } - }); - let options = ChatOptions::default().with_response_format(JsonSpec::new("union", schema)); - let options_set = ChatOptionsSet::default().with_chat_options(Some(&options)); - let target = ServiceTarget { - model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5-mini"), - auth: AuthData::from_single("test-key"), - endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), - }; - - let web_req = OpenAIRespAdapter::to_web_request_data( - target, - ServiceType::Chat, - ChatRequest::from_user("return an animal"), - options_set, - ) - .unwrap(); - - let animal = &web_req.payload["text"]["format"]["schema"]["properties"]["animal"]; - assert!(animal.get("oneOf").is_none()); - assert_eq!(animal["discriminator"], json!({"propertyName": "kind"})); - assert_eq!( - animal["anyOf"], - json!([{"$ref": "#/$defs/Cat"}, {"$ref": "#/$defs/Dog"}]) - ); - } - - #[test] - fn dynamic_map_schema_is_sent_to_backend_for_validation() { - let schema = json!({ - "type": "object", - "properties": { - "lookup": {"type": "object", "additionalProperties": {"type": "integer"}} - } - }); - let options = ChatOptions::default().with_response_format(JsonSpec::new("mapping", schema)); - let options_set = ChatOptionsSet::default().with_chat_options(Some(&options)); - let target = ServiceTarget { - model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5-mini"), - auth: AuthData::from_single("test-key"), - endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), - }; - - let web_req = OpenAIRespAdapter::to_web_request_data( - target, - ServiceType::Chat, - ChatRequest::from_user("return a mapping"), - options_set, - ) - .unwrap(); - - assert_eq!( - web_req.payload["text"]["format"]["schema"]["properties"]["lookup"]["additionalProperties"], - json!({"type": "integer"}) - ); - } - - #[test] - fn test_tool_choice_specific_tool_serialized_on_response_payload() { - let chat_options = ChatOptions::default().with_tool_choice(ToolChoice::tool("get_weather")); - let options_set = ChatOptionsSet::default().with_chat_options(Some(&chat_options)); - let target = ServiceTarget { - model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5-mini"), - auth: AuthData::from_single("test-key"), - endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), - }; - let chat_req = ChatRequest::from_user("weather").with_tools(vec![Tool::new("get_weather")]); - - let web_req = OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, options_set) - .expect("to_web_request_data should succeed"); - - assert_eq!( - web_req.payload["tool_choice"], - json!({ - "type": "function", - "name": "get_weather" - }) - ); - } - - /// Test that assistant message text content uses "output_text" type (not "input_text"). - /// - /// This is required by OpenAI's Responses API - assistant content is model output, - /// so it must use "output_text". Using "input_text" causes: - /// "Invalid value: 'input_text'. Supported values are: 'output_text' and 'refusal'." - #[test] - fn test_assistant_message_uses_output_text_content_type() { - let model_iden = ModelIden::new(AdapterKind::OpenAIResp, "gpt-5-codex"); - - // Create a chat request with an assistant message - let chat_req = ChatRequest::default() - .with_system("You are a helpful assistant.") - .append_message(ChatMessage::user("What's the weather?")) - .append_message(ChatMessage::assistant("The weather is sunny.")); - - // Serialize to OpenAI Responses API format - let parts = OpenAIRespAdapter::into_openai_request_parts(&model_iden, chat_req, None) - .expect("Should serialize successfully"); - - // Find the assistant message in input_items - let assistant_msg = parts - .input_items - .iter() - .find(|item| { - item.get("type").and_then(|t| t.as_str()) == Some("message") - && item.get("role").and_then(|r| r.as_str()) == Some("assistant") - }) - .expect("Should have an assistant message"); - - // Check the content uses "output_text" type - let content = assistant_msg - .get("content") - .and_then(|c| c.as_array()) - .expect("Assistant message should have content array"); - - assert!(!content.is_empty(), "Content should not be empty"); - - let first_content = &content[0]; - let content_type = first_content - .get("type") - .and_then(|t| t.as_str()) - .expect("Content should have a type"); - - assert_eq!( - content_type, "output_text", - "Assistant message content should use 'output_text' type, not 'input_text'" - ); - } - - #[test] - fn test_gpt_5_6_responses_defaults_to_explicit_cache_mode() { - let target = ServiceTarget { - model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), - auth: AuthData::from_single("test-key"), - endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), - }; - - let web_req = OpenAIRespAdapter::to_web_request_data( - target, - ServiceType::Chat, - ChatRequest::from_user("hello"), - ChatOptionsSet::default(), - ) - .expect("to_web_request_data should succeed"); - - assert_eq!(web_req.payload["prompt_cache_options"]["mode"], "explicit"); - assert!(web_req.payload["prompt_cache_options"].get("ttl").is_none()); - assert!( - web_req.payload["input"][0]["content"][0] - .get("prompt_cache_breakpoint") - .is_none() - ); - } - - #[test] - fn test_gpt_5_6_codex_responses_endpoint_omits_prompt_cache_options() -> Result<()> { - let target = ServiceTarget { - model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), - auth: AuthData::from_single("test-key"), - endpoint: Endpoint::from_static("https://chatgpt.com/backend-api/codex/"), - }; - - let web_req = OpenAIRespAdapter::to_web_request_data( - target, - ServiceType::Chat, - ChatRequest::from_user("hello"), - ChatOptionsSet::default(), - )?; - - assert!(web_req.payload.get("prompt_cache_options").is_none()); - Ok(()) - } - - #[test] - fn test_gpt_5_6_responses_cache_key_uses_api_default_cache_mode() { - let target = ServiceTarget { - model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6-mini"), - auth: AuthData::from_single("test-key"), - endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), - }; - let chat_options = ChatOptions::default().with_prompt_cache_key("stable-key"); - let options_set = ChatOptionsSet::default().with_chat_options(Some(&chat_options)); - - let web_req = OpenAIRespAdapter::to_web_request_data( - target, - ServiceType::Chat, - ChatRequest::from_user("hello"), - options_set, - ) - .expect("to_web_request_data should succeed"); - - assert!(web_req.payload.get("prompt_cache_options").is_none()); - assert!( - web_req.payload["input"][0]["content"][0] - .get("prompt_cache_breakpoint") - .is_none() - ); - } - - #[test] - fn test_gpt_5_6_responses_places_breakpoint_on_last_eligible_content_block() { - let target = ServiceTarget { - model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), - auth: AuthData::from_single("test-key"), - endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), - }; - let chat_req = ChatRequest::new(vec![ - ChatMessage::user(vec![ - ContentPart::from_text("stable text"), - ContentPart::from_binary_url("image/png", "https://example.com/image.png", None), - ContentPart::from_text("last text"), - ]) - .with_options(CacheControl::Ephemeral), - ]); - - let web_req = - OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default()) - .expect("to_web_request_data should succeed"); - - let blocks = web_req.payload["input"][0]["content"] - .as_array() - .expect("message content should be an array"); - assert!(blocks[0].get("prompt_cache_breakpoint").is_none()); - assert!(blocks[1].get("prompt_cache_breakpoint").is_none()); - assert_eq!(blocks[2]["prompt_cache_breakpoint"]["mode"], "explicit"); - } - - #[test] - fn test_gpt_5_6_responses_ignores_tool_cache_control() { - let target = ServiceTarget { - model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), - auth: AuthData::from_single("test-key"), - endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), - }; - let chat_req = ChatRequest::from_user("hello") - .append_tool(Tool::new("get_weather").with_cache_control(CacheControl::Ephemeral)); - - let web_req = - OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default()) - .expect("tool cache control should be ignored"); - - assert_eq!(web_req.payload["prompt_cache_options"]["mode"], "explicit"); - assert!(web_req.payload["tools"][0].get("prompt_cache_breakpoint").is_none()); - } - - #[test] - fn test_gpt_5_5_responses_keeps_legacy_cache_retention() { - let target = ServiceTarget { - model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.5"), - auth: AuthData::from_single("test-key"), - endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), - }; - let chat_options = ChatOptions::default().with_cache_control(CacheControl::Ephemeral24h); - let options_set = ChatOptionsSet::default().with_chat_options(Some(&chat_options)); - - let web_req = OpenAIRespAdapter::to_web_request_data( - target, - ServiceType::Chat, - ChatRequest::from_user("hello"), - options_set, - ) - .expect("to_web_request_data should succeed"); - - assert_eq!(web_req.payload["prompt_cache_retention"], "24h"); - assert!(web_req.payload.get("prompt_cache_options").is_none()); - } -} +#[path = "adapter_impl_tests.rs"] +mod tests; // endregion: --- Tests diff --git a/src/adapter/adapters/openai_resp/adapter_impl_tests.rs b/src/adapter/adapters/openai_resp/adapter_impl_tests.rs new file mode 100644 index 00000000..0c929d79 --- /dev/null +++ b/src/adapter/adapters/openai_resp/adapter_impl_tests.rs @@ -0,0 +1,823 @@ +type Result = core::result::Result>; + +use super::*; +use crate::adapter::AdapterKind; +use crate::chat::{Binary, ChatMessage, ChatOptions, JsonSpec, Tool, ToolCall, ToolChoice}; + +#[test] +fn test_cache_control_without_eligible_content_does_not_fail_response_request() { + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + let assistant_msg = ChatMessage::assistant(MessageContent::from_parts(vec![ContentPart::ToolCall(ToolCall { + call_id: "call_1".to_string(), + fn_name: "get_weather".to_string(), + fn_arguments: json!({}), + thought_signatures: None, + })])) + .with_options(CacheControl::Ephemeral); + let chat_req = ChatRequest::new(vec![ChatMessage::user("hello"), assistant_msg]); + + let web_req = + OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default()) + .expect("unsupported breakpoint placement should be ignored"); + + assert_eq!(web_req.payload["prompt_cache_options"]["mode"], "explicit"); +} + +#[test] +fn custom_grammar_tool_and_roundtrip_use_responses_native_items() { + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + let patch = "*** Begin Patch\n*** Update File: source.c\n@@\n-old\n+new\n*** End Patch\n"; + let assistant = ChatMessage::assistant(vec![ToolCall { + call_id: "call_patch".to_string(), + fn_name: "apply_patch".to_string(), + fn_arguments: Value::String(patch.to_string()), + thought_signatures: None, + }]); + let response = ChatMessage::from(ToolResponse::new("call_patch", "Done!")); + let format = json!({ + "type": "grammar", + "syntax": "lark", + "definition": "start: PATCH", + }); + let request = ChatRequest::new(vec![ChatMessage::user("patch it"), assistant, response]).with_tools(vec![ + Tool::new("apply_patch") + .with_description("Apply a patch") + .with_custom_format(format.clone()), + ]); + + let web_req = + OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, request, ChatOptionsSet::default()).unwrap(); + assert_eq!( + web_req.payload["tools"][0], + json!({ + "type": "custom", + "name": "apply_patch", + "description": "Apply a patch", + "format": format, + }) + ); + let input = web_req.payload["input"].as_array().unwrap(); + assert!(input.iter().any(|item| { + item["type"] == "custom_tool_call" && item["call_id"] == "call_patch" && item["input"] == patch + })); + assert!(input.iter().any(|item| { + item["type"] == "custom_tool_call_output" && item["call_id"] == "call_patch" && item["output"] == "Done!" + })); +} + +/// A `ToolResponse` with an image part must serialize natively as a +/// `function_call_output` whose `output` is an array of `input_text` / `input_image` +/// items (the Responses API supports image function-call outputs natively). +#[test] +fn test_tool_response_image_part_serializes_native_output_array() { + // -- Setup & Fixtures + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + let tool_response = + ToolResponse::new("call_1", "screenshot taken").with_parts([Binary::from_base64("image/png", "PNG64", None)]); + let chat_req = ChatRequest::new(vec![ChatMessage::from(tool_response)]); + + // -- Exec + let web_req = + OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default()) + .expect("to_web_request_data should succeed"); + + // -- Check + let input = web_req.payload["input"].as_array().expect("input array"); + let item = input + .iter() + .find(|item| item["type"] == "function_call_output") + .expect("function_call_output item must be present"); + assert_eq!(item["call_id"], "call_1"); + assert_eq!( + item["output"], + json!([ + {"type": "input_text", "text": "screenshot taken"}, + {"type": "input_image", "detail": "auto", "image_url": "data:image/png;base64,PNG64"}, + ]) + ); +} + +/// Regression guard: a text-only `ToolResponse` must keep `output` as a plain string. +#[test] +fn test_tool_response_text_only_output_stays_string() { + // -- Setup & Fixtures + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + let chat_req = ChatRequest::new(vec![ChatMessage::from(ToolResponse::new("call_1", "42"))]); + + // -- Exec + let web_req = + OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default()) + .expect("to_web_request_data should succeed"); + + // -- Check + let input = web_req.payload["input"].as_array().expect("input array"); + let item = input + .iter() + .find(|item| item["type"] == "function_call_output") + .expect("function_call_output item must be present"); + assert_eq!( + item["output"], + json!("42"), + "text-only output must remain a plain string" + ); +} + +/// A `ToolResponse` whose `call_id` belongs to a CUSTOM tool serializes as a +/// `custom_tool_call_output` with a raw string `output` (placeholder text when the +/// result is image-only), and its image parts are rescued into a follow-up `user` +/// message input item right after the output item. +#[test] +fn test_custom_tool_response_image_part_rides_in_followup_user_item() { + // -- Setup & Fixtures + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + let assistant = ChatMessage::assistant(vec![ToolCall { + call_id: "call_patch".to_string(), + fn_name: "apply_patch".to_string(), + fn_arguments: Value::String("some patch".to_string()), + thought_signatures: None, + }]); + let response = ChatMessage::from(ToolResponse::new("call_patch", "").with_parts([Binary::from_base64( + "image/png", + "PNG64", + None, + )])); + let request = ChatRequest::new(vec![ChatMessage::user("patch it"), assistant, response]).with_tools(vec![ + Tool::new("apply_patch").with_custom_format(json!({"type": "text"})), + ]); + + // -- Exec + let web_req = OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, request, ChatOptionsSet::default()) + .expect("to_web_request_data should succeed"); + + // -- Check + let input = web_req.payload["input"].as_array().expect("input array"); + let output_idx = input + .iter() + .position(|item| item["type"] == "custom_tool_call_output") + .expect("custom_tool_call_output item must be present"); + assert_eq!(input[output_idx]["call_id"], "call_patch"); + assert_eq!( + input[output_idx]["output"], + json!("(see attached image)"), + "custom output must stay a raw string with the image placeholder" + ); + let followup = input + .get(output_idx + 1) + .expect("follow-up user message item must come right after the custom output"); + assert_eq!( + *followup, + json!({ + "type": "message", + "role": "user", + "content": [ + {"type": "input_text", "text": TOOL_RESULT_IMAGES_LABEL}, + {"type": "input_image", "detail": "auto", "image_url": "data:image/png;base64,PNG64"}, + ] + }) + ); +} + +/// Regression guard: a text-only custom tool output keeps its raw-string `output` +/// and does NOT get a follow-up user message item. +#[test] +fn test_custom_tool_response_text_only_has_no_followup_item() { + // -- Setup & Fixtures + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + let assistant = ChatMessage::assistant(vec![ToolCall { + call_id: "call_patch".to_string(), + fn_name: "apply_patch".to_string(), + fn_arguments: Value::String("some patch".to_string()), + thought_signatures: None, + }]); + let response = ChatMessage::from(ToolResponse::new("call_patch", "Done!")); + let request = ChatRequest::new(vec![ChatMessage::user("patch it"), assistant, response]).with_tools(vec![ + Tool::new("apply_patch").with_custom_format(json!({"type": "text"})), + ]); + + // -- Exec + let web_req = OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, request, ChatOptionsSet::default()) + .expect("to_web_request_data should succeed"); + + // -- Check + let input = web_req.payload["input"].as_array().expect("input array"); + let item = input + .iter() + .find(|item| item["type"] == "custom_tool_call_output") + .expect("custom_tool_call_output item must be present"); + assert_eq!( + item["output"], + json!("Done!"), + "text-only output must remain the raw string" + ); + assert!( + !input.iter().any(|item| item["content"][0]["text"] == TOOL_RESULT_IMAGES_LABEL), + "no follow-up tool-images user item must be emitted for a text-only custom output" + ); +} + +#[test] +fn test_extra_body_merged_into_response_payload() { + let chat_options = ChatOptions::default() + .with_top_p(0.3) + .with_extra_body(json!({"top_p": 0.9, "metadata": {"source": "test"}})); + let options_set = ChatOptionsSet::default().with_chat_options(Some(&chat_options)); + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5-mini"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + + let web_req = + OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, ChatRequest::from_user("hello"), options_set) + .expect("to_web_request_data should succeed"); + + assert_eq!(web_req.payload["top_p"], 0.9); + assert_eq!(web_req.payload["metadata"]["source"], "test"); +} + +#[test] +fn pydantic_union_schema_is_sanitized_for_responses() { + let schema = json!({ + "type": "object", + "properties": { + "animal": { + "discriminator": {"propertyName": "kind"}, + "oneOf": [{"$ref": "#/$defs/Cat"}, {"$ref": "#/$defs/Dog"}] + } + }, + "$defs": { + "Cat": { + "type": "object", + "properties": {"kind": {"const": "cat"}}, + "required": ["kind"] + }, + "Dog": { + "type": "object", + "properties": {"kind": {"const": "dog"}}, + "required": ["kind"] + } + } + }); + let options = ChatOptions::default().with_response_format(JsonSpec::new("union", schema)); + let options_set = ChatOptionsSet::default().with_chat_options(Some(&options)); + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5-mini"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + + let web_req = OpenAIRespAdapter::to_web_request_data( + target, + ServiceType::Chat, + ChatRequest::from_user("return an animal"), + options_set, + ) + .unwrap(); + + let animal = &web_req.payload["text"]["format"]["schema"]["properties"]["animal"]; + assert!(animal.get("oneOf").is_none()); + assert_eq!(animal["discriminator"], json!({"propertyName": "kind"})); + assert_eq!( + animal["anyOf"], + json!([{"$ref": "#/$defs/Cat"}, {"$ref": "#/$defs/Dog"}]) + ); +} + +#[test] +fn dynamic_map_schema_is_sent_to_backend_for_validation() { + let schema = json!({ + "type": "object", + "properties": { + "lookup": {"type": "object", "additionalProperties": {"type": "integer"}} + } + }); + let options = ChatOptions::default().with_response_format(JsonSpec::new("mapping", schema)); + let options_set = ChatOptionsSet::default().with_chat_options(Some(&options)); + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5-mini"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + + let web_req = OpenAIRespAdapter::to_web_request_data( + target, + ServiceType::Chat, + ChatRequest::from_user("return a mapping"), + options_set, + ) + .unwrap(); + + assert_eq!( + web_req.payload["text"]["format"]["schema"]["properties"]["lookup"]["additionalProperties"], + json!({"type": "integer"}) + ); +} + +#[test] +fn test_tool_choice_specific_tool_serialized_on_response_payload() { + let chat_options = ChatOptions::default().with_tool_choice(ToolChoice::tool("get_weather")); + let options_set = ChatOptionsSet::default().with_chat_options(Some(&chat_options)); + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5-mini"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + let chat_req = ChatRequest::from_user("weather").with_tools(vec![Tool::new("get_weather")]); + + let web_req = OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, options_set) + .expect("to_web_request_data should succeed"); + + assert_eq!( + web_req.payload["tool_choice"], + json!({ + "type": "function", + "name": "get_weather" + }) + ); +} + +/// Test that assistant message text content uses "output_text" type (not "input_text"). +/// +/// This is required by OpenAI's Responses API - assistant content is model output, +/// so it must use "output_text". Using "input_text" causes: +/// "Invalid value: 'input_text'. Supported values are: 'output_text' and 'refusal'." +#[test] +fn test_assistant_message_uses_output_text_content_type() { + let model_iden = ModelIden::new(AdapterKind::OpenAIResp, "gpt-5-codex"); + + // Create a chat request with an assistant message + let chat_req = ChatRequest::default() + .with_system("You are a helpful assistant.") + .append_message(ChatMessage::user("What's the weather?")) + .append_message(ChatMessage::assistant("The weather is sunny.")); + + // Serialize to OpenAI Responses API format + let parts = OpenAIRespAdapter::into_openai_request_parts(&model_iden, chat_req, None) + .expect("Should serialize successfully"); + + // Find the assistant message in input_items + let assistant_msg = parts + .input_items + .iter() + .find(|item| { + item.get("type").and_then(|t| t.as_str()) == Some("message") + && item.get("role").and_then(|r| r.as_str()) == Some("assistant") + }) + .expect("Should have an assistant message"); + + // Check the content uses "output_text" type + let content = assistant_msg + .get("content") + .and_then(|c| c.as_array()) + .expect("Assistant message should have content array"); + + assert!(!content.is_empty(), "Content should not be empty"); + + let first_content = &content[0]; + let content_type = first_content + .get("type") + .and_then(|t| t.as_str()) + .expect("Content should have a type"); + + assert_eq!( + content_type, "output_text", + "Assistant message content should use 'output_text' type, not 'input_text'" + ); +} + +#[test] +fn test_gpt_5_6_responses_defaults_to_explicit_cache_mode() { + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + + let web_req = OpenAIRespAdapter::to_web_request_data( + target, + ServiceType::Chat, + ChatRequest::from_user("hello"), + ChatOptionsSet::default(), + ) + .expect("to_web_request_data should succeed"); + + assert_eq!(web_req.payload["prompt_cache_options"]["mode"], "explicit"); + assert!(web_req.payload["prompt_cache_options"].get("ttl").is_none()); + assert!( + web_req.payload["input"][0]["content"][0] + .get("prompt_cache_breakpoint") + .is_none() + ); +} + +#[test] +fn test_gpt_5_6_codex_responses_endpoint_omits_prompt_cache_options() -> Result<()> { + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), + auth: AuthData::from_single("test-key"), + endpoint: Endpoint::from_static("https://chatgpt.com/backend-api/codex/"), + }; + + let web_req = OpenAIRespAdapter::to_web_request_data( + target, + ServiceType::Chat, + ChatRequest::from_user("hello"), + ChatOptionsSet::default(), + )?; + + assert!(web_req.payload.get("prompt_cache_options").is_none()); + Ok(()) +} + +#[test] +fn test_gpt_5_6_responses_cache_key_uses_api_default_cache_mode() { + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6-mini"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + let chat_options = ChatOptions::default().with_prompt_cache_key("stable-key"); + let options_set = ChatOptionsSet::default().with_chat_options(Some(&chat_options)); + + let web_req = + OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, ChatRequest::from_user("hello"), options_set) + .expect("to_web_request_data should succeed"); + + assert!(web_req.payload.get("prompt_cache_options").is_none()); + assert!( + web_req.payload["input"][0]["content"][0] + .get("prompt_cache_breakpoint") + .is_none() + ); +} + +#[test] +fn test_gpt_5_6_responses_places_breakpoint_on_last_eligible_content_block() { + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + let chat_req = ChatRequest::new(vec![ + ChatMessage::user(vec![ + ContentPart::from_text("stable text"), + ContentPart::from_binary_url("image/png", "https://example.com/image.png", None), + ContentPart::from_text("last text"), + ]) + .with_options(CacheControl::Ephemeral), + ]); + + let web_req = + OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default()) + .expect("to_web_request_data should succeed"); + + let blocks = web_req.payload["input"][0]["content"] + .as_array() + .expect("message content should be an array"); + assert!(blocks[0].get("prompt_cache_breakpoint").is_none()); + assert!(blocks[1].get("prompt_cache_breakpoint").is_none()); + assert_eq!(blocks[2]["prompt_cache_breakpoint"]["mode"], "explicit"); +} + +#[test] +fn test_gpt_5_6_responses_ignores_tool_cache_control() { + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + let chat_req = ChatRequest::from_user("hello") + .append_tool(Tool::new("get_weather").with_cache_control(CacheControl::Ephemeral)); + + let web_req = + OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default()) + .expect("tool cache control should be ignored"); + + assert_eq!(web_req.payload["prompt_cache_options"]["mode"], "explicit"); + assert!(web_req.payload["tools"][0].get("prompt_cache_breakpoint").is_none()); +} + +#[test] +fn test_gpt_5_5_responses_keeps_legacy_cache_retention() { + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.5"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + let chat_options = ChatOptions::default().with_cache_control(CacheControl::Ephemeral24h); + let options_set = ChatOptionsSet::default().with_chat_options(Some(&chat_options)); + + let web_req = + OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, ChatRequest::from_user("hello"), options_set) + .expect("to_web_request_data should succeed"); + + assert_eq!(web_req.payload["prompt_cache_retention"], "24h"); + assert!(web_req.payload.get("prompt_cache_options").is_none()); +} + +// region: --- Embedded Tool Responses + +/// A `ToolResponse` embedded in a User-role message (Anthropic-style user-carried +/// tool result) must be extracted as a `function_call_output` item placed BEFORE +/// the user message item carrying the remaining content. `call_id`s are serialized +/// as-is (no matching validation; here the call_id matches no `function_call` item). +#[test] +fn test_user_embedded_tool_response_extracted_before_user_item() { + // -- Setup & Fixtures + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + let user_msg = ChatMessage::user(vec![ + ContentPart::ToolResponse(ToolResponse::new("call_1", "sunny")), + ContentPart::from_text("thanks, and tomorrow?"), + ]); + let chat_req = ChatRequest::new(vec![user_msg]); + + // -- Exec + let web_req = + OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default()) + .expect("to_web_request_data should succeed"); + + // -- Check + let input = web_req.payload["input"].as_array().expect("input array"); + assert_eq!(input.len(), 2, "extracted output item + user message item"); + assert_eq!( + input[0], + json!({"type": "function_call_output", "call_id": "call_1", "output": "sunny"}), + "embedded tool response must become a function_call_output item before the user message item" + ); + assert_eq!( + input[1], + json!({"role": "user", "content": [{"type": "input_text", "text": "thanks, and tomorrow?"}]}) + ); +} + +/// Image parts of a user-embedded `ToolResponse` for a FUNCTION tool ride natively +/// in the `function_call_output` `output` array (`input_text` + `input_image`); +/// nothing is folded into the carrying user message item. +#[test] +fn test_user_embedded_tool_response_image_part_uses_native_output_array() { + // -- Setup & Fixtures + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + let tool_response = + ToolResponse::new("call_1", "screenshot taken").with_parts([Binary::from_base64("image/png", "PNG64", None)]); + let user_msg = ChatMessage::user(vec![ + ContentPart::ToolResponse(tool_response), + ContentPart::from_text("what do you see?"), + ]); + let chat_req = ChatRequest::new(vec![user_msg]); + + // -- Exec + let web_req = + OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default()) + .expect("to_web_request_data should succeed"); + + // -- Check + let input = web_req.payload["input"].as_array().expect("input array"); + assert_eq!(input.len(), 2, "extracted output item + user message item"); + assert_eq!( + input[0], + json!({ + "type": "function_call_output", + "call_id": "call_1", + "output": [ + {"type": "input_text", "text": "screenshot taken"}, + {"type": "input_image", "detail": "auto", "image_url": "data:image/png;base64,PNG64"}, + ] + }) + ); + assert_eq!( + input[1], + json!({"role": "user", "content": [{"type": "input_text", "text": "what do you see?"}]}), + "function-output images ride natively; the user message item must not carry them" + ); +} + +/// Image parts of a user-embedded `ToolResponse` for a CUSTOM tool (raw-string +/// output wire) fold into the SAME user message item as `input_image` items (no +/// label item), mirroring the Gemini serializer's user-embedded handling. +#[test] +fn test_user_embedded_custom_tool_response_images_fold_into_user_item() { + // -- Setup & Fixtures + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + let assistant = ChatMessage::assistant(vec![ToolCall { + call_id: "call_patch".to_string(), + fn_name: "apply_patch".to_string(), + fn_arguments: Value::String("some patch".to_string()), + thought_signatures: None, + }]); + let tool_response = + ToolResponse::new("call_patch", "").with_parts([Binary::from_base64("image/png", "PNG64", None)]); + let user_msg = ChatMessage::user(vec![ + ContentPart::ToolResponse(tool_response), + ContentPart::from_text("continue"), + ]); + let chat_req = ChatRequest::new(vec![ChatMessage::user("patch it"), assistant, user_msg]).with_tools(vec![ + Tool::new("apply_patch").with_custom_format(json!({"type": "text"})), + ]); + + // -- Exec + let web_req = + OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default()) + .expect("to_web_request_data should succeed"); + + // -- Check + let input = web_req.payload["input"].as_array().expect("input array"); + let output_idx = input + .iter() + .position(|item| item["type"] == "custom_tool_call_output") + .expect("custom_tool_call_output item must be present"); + assert_eq!( + input[output_idx], + json!({"type": "custom_tool_call_output", "call_id": "call_patch", "output": "(see attached image)"}) + ); + assert_eq!( + input[output_idx + 1], + json!({ + "role": "user", + "content": [ + {"type": "input_image", "detail": "auto", "image_url": "data:image/png;base64,PNG64"}, + {"type": "input_text", "text": "continue"}, + ] + }), + "the rescued image must fold into the same user message item" + ); + assert!( + !input.iter().any(|item| item["content"][0]["text"] == TOOL_RESULT_IMAGES_LABEL), + "no separate labeled tool-images user item must be emitted" + ); +} + +/// A user message whose content is ONLY embedded tool responses leaves nothing to +/// carry: the output items are extracted (multiple, in order) and the now-empty +/// user message item is omitted. +#[test] +fn test_user_item_with_only_embedded_tool_responses_is_omitted() { + // -- Setup & Fixtures + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + let user_msg = ChatMessage::user(vec![ + ContentPart::ToolResponse(ToolResponse::new("call_1", "42")), + ContentPart::ToolResponse(ToolResponse::new("call_2", "43")), + ]); + let chat_req = ChatRequest::new(vec![user_msg]); + + // -- Exec + let web_req = + OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default()) + .expect("to_web_request_data should succeed"); + + // -- Check + let input = web_req.payload["input"].as_array().expect("input array"); + assert_eq!(input.len(), 2, "only the two extracted output items"); + assert_eq!( + input[0], + json!({"type": "function_call_output", "call_id": "call_1", "output": "42"}) + ); + assert_eq!( + input[1], + json!({"type": "function_call_output", "call_id": "call_2", "output": "43"}) + ); + assert!( + !input.iter().any(|item| item["role"] == "user"), + "no empty user message item must be emitted" + ); +} + +/// When a user message with an embedded `ToolResponse` follows a Tool-role run +/// whose custom-output images are pending, the extracted output item must land +/// BEFORE the batched tool-images user item, keeping it adjacent to the tool run +/// (mirroring the Chat Completions adjacency behavior). +#[test] +fn test_user_embedded_tool_response_stays_adjacent_to_tool_run() { + // -- Setup & Fixtures + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + let assistant = ChatMessage::assistant(vec![ToolCall { + call_id: "call_patch".to_string(), + fn_name: "apply_patch".to_string(), + fn_arguments: Value::String("some patch".to_string()), + thought_signatures: None, + }]); + let tool_msg = ChatMessage::from(ToolResponse::new("call_patch", "").with_parts([Binary::from_base64( + "image/png", + "PNG64", + None, + )])); + let user_msg = ChatMessage::user(vec![ + ContentPart::ToolResponse(ToolResponse::new("call_2", "done")), + ContentPart::from_text("go on"), + ]); + let chat_req = ChatRequest::new(vec![assistant, tool_msg, user_msg]).with_tools(vec![ + Tool::new("apply_patch").with_custom_format(json!({"type": "text"})), + ]); + + // -- Exec + let web_req = + OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default()) + .expect("to_web_request_data should succeed"); + + // -- Check + let input = web_req.payload["input"].as_array().expect("input array"); + assert_eq!( + input.len(), + 5, + "call + custom output + extracted output + image flush + user" + ); + assert_eq!(input[0]["type"], "custom_tool_call"); + assert_eq!(input[1]["type"], "custom_tool_call_output"); + assert_eq!( + input[2], + json!({"type": "function_call_output", "call_id": "call_2", "output": "done"}), + "extracted output item must come before the batched tool-images user item" + ); + assert_eq!(input[3]["content"][0]["text"], TOOL_RESULT_IMAGES_LABEL); + assert_eq!( + input[4], + json!({"role": "user", "content": [{"type": "input_text", "text": "go on"}]}) + ); +} + +/// A `ToolResponse` embedded in an Assistant message has no representation on any +/// provider wire (there is no "tool result authored by the assistant"), so the +/// serializer must reject the shape with a hard error instead of dropping the +/// content or inventing a placement. +#[test] +fn test_assistant_embedded_tool_response_is_rejected() { + // -- Setup & Fixtures + let target = ServiceTarget { + model: ModelIden::new(AdapterKind::OpenAIResp, "gpt-5.6"), + auth: AuthData::from_single("test-key"), + endpoint: OpenAIRespAdapter::default_endpoint(AdapterKind::OpenAIResp), + }; + let assistant_msg = ChatMessage::assistant(MessageContent::from_parts(vec![ + ContentPart::from_text("checking"), + ContentPart::ToolCall(ToolCall { + call_id: "call_1".to_string(), + fn_name: "get_weather".to_string(), + fn_arguments: json!({"city": "Paris"}), + thought_signatures: None, + }), + ContentPart::ToolResponse(ToolResponse::new("call_1", "sunny")), + ])); + let chat_req = ChatRequest::new(vec![ChatMessage::user("weather?"), assistant_msg]); + + // -- Exec + let err = OpenAIRespAdapter::to_web_request_data(target, ServiceType::Chat, chat_req, ChatOptionsSet::default()) + .expect_err("assistant-embedded tool response must fail serialization"); + + // -- Check + let Error::MessageContentTypeNotSupported { cause, .. } = err else { + panic!("expected MessageContentTypeNotSupported, got: {err}"); + }; + assert!( + cause.contains("Assistant-role message"), + "cause must name the unsupported shape: {cause}" + ); + assert!( + cause.contains("Tool-role message"), + "cause must point at the supported Tool-role shape: {cause}" + ); +} + +// endregion: --- Embedded Tool Responses diff --git a/src/adapter/adapters/opencode_go/adapter_impl.rs b/src/adapter/adapters/opencode_go/adapter_impl.rs index cc680b82..b69ab0b2 100644 --- a/src/adapter/adapters/opencode_go/adapter_impl.rs +++ b/src/adapter/adapters/opencode_go/adapter_impl.rs @@ -100,7 +100,11 @@ impl Adapter for OpenCodeGoAdapter { system, messages, tools, - } = AnthropicAdapter::into_anthropic_request_parts(chat_req, options_set.cache_control().cloned())?; + } = AnthropicAdapter::into_anthropic_request_parts( + &model, + chat_req, + options_set.cache_control().cloned(), + )?; let stream = matches!(service_type, ServiceType::ChatStream); let mut payload = json!({ @@ -165,14 +169,17 @@ impl Adapter for OpenCodeGoAdapter { model_iden: ModelIden, reqwest_builder: RequestBuilder, options_set: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result { let (_, model_name) = model_iden.model_name.namespace_and_name(); let model_kind = OpenCodeGoModelKind::from_model_name(model_name); match model_kind { - OpenCodeGoModelKind::OpenAI => OpenAIAdapter::to_chat_stream(model_iden, reqwest_builder, options_set), + OpenCodeGoModelKind::OpenAI => { + OpenAIAdapter::to_chat_stream(model_iden, reqwest_builder, options_set, response_observer) + } OpenCodeGoModelKind::Anthropic => { - AnthropicAdapter::to_chat_stream(model_iden, reqwest_builder, options_set) + AnthropicAdapter::to_chat_stream(model_iden, reqwest_builder, options_set, response_observer) } } } diff --git a/src/adapter/adapters/support.rs b/src/adapter/adapters/support.rs index eec55a76..a230ca46 100644 --- a/src/adapter/adapters/support.rs +++ b/src/adapter/adapters/support.rs @@ -13,6 +13,50 @@ pub fn get_api_key(auth: AuthData, model: &ModelIden) -> Result { }) } +/// Build the error for a `ContentPart::ToolResponse` embedded in an Assistant-role message. +/// +/// No provider wire has a representation for a tool result authored by the assistant +/// (tool results are standalone `role:"tool"` messages / output items on the OpenAI +/// wires, and user-carried `tool_result` / `toolResult` / `functionResponse` blocks on +/// the Anthropic-style wires), so every serializer rejects the shape with this same +/// error instead of silently dropping the content or inventing a placement the wire +/// does not define. The supported shape is a Tool-role message. +pub fn assistant_embedded_tool_response_err(model_iden: &ModelIden) -> Error { + Error::MessageContentTypeNotSupported { + model_iden: model_iden.clone(), + cause: "ContentPart::ToolResponse is not supported in an Assistant-role message — no provider wire represents a tool result authored by the assistant. Send the tool response as a Tool-role message instead (e.g., `ChatMessage::from(ToolResponse)`)", + } +} + +// region: --- Tool Response Binary Parts + +/// Leading text of the follow-up `user` message that carries tool-result images on +/// wire formats that cannot express images inside the tool-result item itself +/// (e.g., OpenAI Chat Completions `tool` messages, Gemini `functionResponse`, Ollama). +pub const TOOL_RESULT_IMAGES_LABEL: &str = "Attached image(s) from tool result:"; + +/// Resolve the text content of a tool-result message when the `ToolResponse` +/// carries binary parts. +/// +/// - Non-empty text content is kept as-is. +/// - Empty text with image parts becomes the `"(see attached image)"` placeholder, +/// pointing the model at the follow-up user message that carries the images. +/// - Empty text without any usable image part becomes `"(no tool output)"`. +/// +/// NOTE: Only called when `ToolResponse.parts` is present, so plain text-only +/// responses keep their exact legacy serialization. +pub fn tool_response_fallback_text(content: String, has_images: bool) -> String { + if !content.is_empty() { + content + } else if has_images { + "(see attached image)".to_string() + } else { + "(no tool output)".to_string() + } +} + +// endregion: --- Tool Response Binary Parts + // region: --- StreamerChatOptions #[derive(Debug)] diff --git a/src/adapter/adapters/vertex/adapter_impl.rs b/src/adapter/adapters/vertex/adapter_impl.rs index ccc858a8..ed854abb 100644 --- a/src/adapter/adapters/vertex/adapter_impl.rs +++ b/src/adapter/adapters/vertex/adapter_impl.rs @@ -178,13 +178,18 @@ impl Adapter for VertexAdapter { model_iden: ModelIden, reqwest_builder: RequestBuilder, options_set: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result { let (_, model_name) = model_iden.model_name.namespace_and_name(); let publisher = VertexPublisher::from_model_name(model_name)?; match publisher { - VertexPublisher::Google => GeminiAdapter::to_chat_stream(model_iden, reqwest_builder, options_set), - VertexPublisher::Anthropic => AnthropicAdapter::to_chat_stream(model_iden, reqwest_builder, options_set), + VertexPublisher::Google => { + GeminiAdapter::to_chat_stream(model_iden, reqwest_builder, options_set, response_observer) + } + VertexPublisher::Anthropic => { + AnthropicAdapter::to_chat_stream(model_iden, reqwest_builder, options_set, response_observer) + } } } @@ -251,7 +256,7 @@ impl VertexAdapter { system, messages, tools, - } = AnthropicAdapter::into_anthropic_request_parts(chat_req, options_set.cache_control().cloned())?; + } = AnthropicAdapter::into_anthropic_request_parts(&model, chat_req, options_set.cache_control().cloned())?; // Vertex Anthropic: model is in URL, not body; anthropic_version goes in body let stream = matches!(service_type, ServiceType::ChatStream); diff --git a/src/adapter/adapters/zai/adapter_impl.rs b/src/adapter/adapters/zai/adapter_impl.rs index c7262440..a88ccca4 100644 --- a/src/adapter/adapters/zai/adapter_impl.rs +++ b/src/adapter/adapters/zai/adapter_impl.rs @@ -105,8 +105,9 @@ impl Adapter for ZaiAdapter { model_iden: ModelIden, reqwest_builder: RequestBuilder, options_set: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result { - OpenAIAdapter::to_chat_stream(model_iden, reqwest_builder, options_set) + OpenAIAdapter::to_chat_stream(model_iden, reqwest_builder, options_set, response_observer) } fn to_embed_request_data( diff --git a/src/adapter/dispatcher.rs b/src/adapter/dispatcher.rs index 9a38794b..587a9ca0 100644 --- a/src/adapter/dispatcher.rs +++ b/src/adapter/dispatcher.rs @@ -2,6 +2,7 @@ use super::macros::dispatch_adapter; use crate::ModelIden; use crate::adapter::{Adapter, AdapterKind, ServiceType, WebRequestData}; use crate::chat::{ChatOptionsSet, ChatRequest, ChatResponse, ChatStreamResponse}; +use crate::client::BoundResponseObserver; use crate::embed::{EmbedOptionsSet, EmbedRequest, EmbedResponse}; use crate::resolver::{AuthData, Endpoint}; use crate::webc::{WebClient, WebResponse}; @@ -63,11 +64,12 @@ impl AdapterDispatcher { model_iden: ModelIden, reqwest_builder: RequestBuilder, options_set: ChatOptionsSet<'_, '_>, + response_observer: Option, ) -> Result { let adapter_kind = model_iden.adapter_kind; dispatch_adapter!( adapter_kind, - A::to_chat_stream(model_iden, reqwest_builder, options_set) + A::to_chat_stream(model_iden, reqwest_builder, options_set, response_observer) ) } diff --git a/src/adapter/macros/adapter_impl_macros.rs b/src/adapter/macros/adapter_impl_macros.rs index f73d3400..dcb38355 100644 --- a/src/adapter/macros/adapter_impl_macros.rs +++ b/src/adapter/macros/adapter_impl_macros.rs @@ -146,8 +146,14 @@ macro_rules! impl_pass_through_adapter { model_iden: $crate::ModelIden, reqwest_builder: ::reqwest::RequestBuilder, options_set: $crate::chat::ChatOptionsSet<'_, '_>, + response_observer: Option<$crate::client::BoundResponseObserver>, ) -> $crate::Result<$crate::chat::ChatStreamResponse> { - <$delegate as $crate::adapter::Adapter>::to_chat_stream(model_iden, reqwest_builder, options_set) + <$delegate as $crate::adapter::Adapter>::to_chat_stream( + model_iden, + reqwest_builder, + options_set, + response_observer, + ) } }; diff --git a/src/chat/tool/tool_response.rs b/src/chat/tool/tool_response.rs index f4f964c1..37c1f44b 100644 --- a/src/chat/tool/tool_response.rs +++ b/src/chat/tool/tool_response.rs @@ -1,4 +1,5 @@ use super::ToolCall; +use crate::chat::Binary; use serde::{Deserialize, Serialize}; /// Response produced by a tool invocation, paired with the originating tool call ID. @@ -15,6 +16,16 @@ pub struct ToolResponse { /// Tool output payload as a string. Providers may use JSON-serialized content. // For now, just a string (would probably be serialized JSON) pub content: String, + /// Optional binary attachments produced by the tool (e.g., screenshots, file reads). + /// + /// Image parts serialize natively where the wire supports them (Anthropic + /// `tool_result`, Bedrock Converse `toolResult`, OpenAI Responses + /// `function_call_output`) and ride in a follow-up user message elsewhere + /// (OpenAI Chat Completions-compatible providers, Gemini, Ollama). + /// Non-image parts are currently skipped with a warning. + /// Text-only responses (no `parts`) keep their exact legacy serialization. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub parts: Option>, } /// Constructor @@ -25,6 +36,7 @@ impl ToolResponse { call_id: tool_call_id.into(), fn_name: None, content: content.into(), + parts: None, } } @@ -34,6 +46,7 @@ impl ToolResponse { call_id: tool_call.call_id.clone(), fn_name: Some(tool_call.fn_name.clone()), content: content.into(), + parts: None, } } @@ -42,6 +55,22 @@ impl ToolResponse { self.fn_name = Some(fn_name.into()); self } + + /// Set the binary attachments of this response. Returns self for chaining. + pub fn with_parts(mut self, parts: I) -> Self + where + I: IntoIterator, + I::Item: Into, + { + self.parts = Some(parts.into_iter().map(Into::into).collect()); + self + } + + /// Append a single binary attachment to this response. Returns self for chaining. + pub fn append_binary(mut self, binary: impl Into) -> Self { + self.parts.get_or_insert_with(Vec::new).push(binary.into()); + self + } } /// Computed accessors @@ -49,9 +78,16 @@ impl ToolResponse { /// Returns an approximate in-memory size of this `ToolResponse`, in bytes, /// computed as the sum of the UTF-8 lengths of: /// - `call_id` + /// - `fn_name` (if any) /// - `content` + /// - plus the `Binary::size()` of each part (if any) pub fn size(&self) -> usize { - self.call_id.len() + self.fn_name.as_ref().map(|name| name.len()).unwrap_or(0) + self.content.len() + let parts_size: usize = self + .parts + .as_ref() + .map(|parts| parts.iter().map(Binary::size).sum()) + .unwrap_or_default(); + self.call_id.len() + self.fn_name.as_ref().map(|name| name.len()).unwrap_or(0) + self.content.len() + parts_size } } @@ -69,4 +105,16 @@ impl ToolResponse { fn content(&self) -> &str { &self.content } + + fn parts(&self) -> Option<&[Binary]> { + self.parts.as_deref() + } } + +// region: --- Tests + +#[cfg(test)] +#[path = "tool_response_tests.rs"] +mod tests; + +// endregion: --- Tests diff --git a/src/chat/tool/tool_response_tests.rs b/src/chat/tool/tool_response_tests.rs new file mode 100644 index 00000000..b8a2a3e2 --- /dev/null +++ b/src/chat/tool/tool_response_tests.rs @@ -0,0 +1,60 @@ +type Result = core::result::Result>; // For tests. + +use super::*; +use serde_json::json; + +/// A text-only ToolResponse must serialize exactly as before the `parts` addition +/// (no `parts` key), so persisted chat histories keep the same JSON shape. +#[test] +fn test_tool_response_text_only_serde_unchanged() -> Result<()> { + // -- Setup & Fixtures + let tool_response = ToolResponse::new("call_1", "42"); + + // -- Exec + let value = serde_json::to_value(&tool_response)?; + + // -- Check + assert_eq!(value, json!({"call_id": "call_1", "content": "42"})); + + Ok(()) +} + +/// Legacy JSON (without `parts`) must still deserialize. +#[test] +fn test_tool_response_deserialize_legacy_json() -> Result<()> { + // -- Setup & Fixtures + let legacy_json = json!({"call_id": "call_1", "content": "42"}); + + // -- Exec + let tool_response: ToolResponse = serde_json::from_value(legacy_json)?; + + // -- Check + assert_eq!(tool_response.call_id, "call_1"); + assert_eq!(tool_response.content, "42"); + assert!(tool_response.parts.is_none()); + + Ok(()) +} + +/// `with_parts` and `append_binary` builders populate `parts`, and `parts` +/// round-trips through serde. +#[test] +fn test_tool_response_with_parts_serde_roundtrip() -> Result<()> { + // -- Setup & Fixtures + let tool_response = ToolResponse::new("call_1", "screenshot taken") + .with_parts([Binary::from_base64("image/png", "AAA=", None)]) + .append_binary(Binary::from_base64("image/jpeg", "BBB=", Some("shot.jpg".to_string()))); + + // -- Exec + let value = serde_json::to_value(&tool_response)?; + let back: ToolResponse = serde_json::from_value(value)?; + + // -- Check + let parts = back.parts.ok_or("should have parts")?; + assert_eq!(parts.len(), 2); + assert_eq!(parts[0].content_type, "image/png"); + assert_eq!(parts[1].content_type, "image/jpeg"); + assert_eq!(parts[1].name.as_deref(), Some("shot.jpg")); + + Ok(()) +} diff --git a/src/client/builder.rs b/src/client/builder.rs index 306ae9af..9f67f160 100644 --- a/src/client/builder.rs +++ b/src/client/builder.rs @@ -1,5 +1,6 @@ use crate::adapter::AdapterKind; use crate::chat::ChatOptions; +use crate::client::{IntoPayloadInterceptorFn, IntoResponseObserverFn, PayloadInterceptor, ResponseObserver}; use crate::resolver::{ AuthResolver, IntoAuthResolverFn, IntoModelMapperFn, IntoServiceTargetResolverFn, ModelMapper, ServiceTargetResolver, @@ -96,6 +97,44 @@ impl ClientBuilder { self } + /// Set `PayloadInterceptor` on `ClientConfig` (creates it if absent). + /// + /// Per-request exec hook: called on each chat exec call (streaming and non-streaming) with + /// the target `ModelIden` and the serialized provider payload, before the HTTP request is + /// built. `Some` replaces the payload; `None` keeps it unchanged. + pub fn with_payload_interceptor(mut self, payload_interceptor: PayloadInterceptor) -> Self { + let client_config = self.config.get_or_insert_with(ClientConfig::default); + client_config.payload_interceptor = Some(payload_interceptor); + self + } + + /// Set `PayloadInterceptor` from a sync interceptor function (creates `ClientConfig` if absent). + pub fn with_payload_interceptor_fn(mut self, payload_interceptor_fn: impl IntoPayloadInterceptorFn) -> Self { + let client_config = self.config.get_or_insert_with(ClientConfig::default); + let payload_interceptor = PayloadInterceptor::from_interceptor_fn(payload_interceptor_fn); + client_config.payload_interceptor = Some(payload_interceptor); + self + } + + /// Set `ResponseObserver` on `ClientConfig` (creates it if absent). + /// + /// Per-request exec hook: called on each chat exec call (streaming and non-streaming) with + /// the target `ModelIden`, the response `StatusCode`, and the response `HeaderMap`, as soon + /// as the HTTP response arrives and before its body/stream is consumed (also on 4xx/5xx). + pub fn with_response_observer(mut self, response_observer: ResponseObserver) -> Self { + let client_config = self.config.get_or_insert_with(ClientConfig::default); + client_config.response_observer = Some(response_observer); + self + } + + /// Set `ResponseObserver` from a sync observer function (creates `ClientConfig` if absent). + pub fn with_response_observer_fn(mut self, response_observer_fn: impl IntoResponseObserverFn) -> Self { + let client_config = self.config.get_or_insert_with(ClientConfig::default); + let response_observer = ResponseObserver::from_observer_fn(response_observer_fn); + client_config.response_observer = Some(response_observer); + self + } + /// Bind the Client to a single [`AdapterKind`] (creates `ClientConfig` if absent). /// /// See [`ClientConfig::with_adapter_kind`] for semantics. Short version: diff --git a/src/client/client_impl.rs b/src/client/client_impl.rs index a4946e09..c999f4fe 100644 --- a/src/client/client_impl.rs +++ b/src/client/client_impl.rs @@ -1,6 +1,6 @@ use crate::adapter::{AdapterDispatcher, AdapterKind, ServiceType, WebRequestData}; use crate::chat::{ChatOptions, ChatOptionsSet, ChatRequest, ChatResponse, ChatStreamResponse}; -use crate::client::ModelSpec; +use crate::client::{BoundResponseObserver, ModelSpec}; use crate::embed::{EmbedOptions, EmbedOptionsSet, EmbedRequest, EmbedResponse}; use crate::resolver::{AuthData, ProviderConfig}; use crate::{Client, Error, ModelIden, Result, ServiceTarget}; @@ -127,9 +127,22 @@ impl Client { headers = override_headers; }; + // -- Apply the payload interceptor exec hook (if set), which can replace the payload. + let payload = match self.config().payload_interceptor() { + Some(interceptor) => interceptor.intercept(model.clone(), payload.clone()).await.unwrap_or(payload), + None => payload, + }; + + // -- Bind the response observer exec hook (if set) so it fires on the response head, + // before the body is consumed (also on 4xx/5xx). + let response_observer = self + .config() + .response_observer() + .map(|observer| BoundResponseObserver::new(observer.clone(), model.clone())); + let web_res = self .web_client() - .do_post(&url, &headers, &payload) + .do_post_with_observer(&url, &headers, &payload, response_observer.as_ref()) .await .map_err(|webc_error| Error::WebModelCall { model_iden: model.clone(), @@ -190,8 +203,9 @@ impl Client { #[cfg(feature = "otel")] let otel_span = crate::otel::span::chat_request_span(&model, &target.endpoint, &options_set, &chat_req, true); - // Stream setup is synchronous; wrap it so setup errors are recorded on the span too. - let result = (move || { + // Stream setup is async (payload interceptor hook); wrap it so setup errors are recorded on the span too. + // Note: The HTTP send itself remains lazy (performed on the first stream poll). + let result = async { let WebRequestData { mut url, mut headers, @@ -214,6 +228,12 @@ impl Client { headers = override_headers; }; + // -- Apply the payload interceptor exec hook (if set), which can replace the payload. + let payload = match self.config().payload_interceptor() { + Some(interceptor) => interceptor.intercept(model.clone(), payload.clone()).await.unwrap_or(payload), + None => payload, + }; + let reqwest_builder = self.web_client() .new_req_builder(&url, &headers, &payload) @@ -222,10 +242,19 @@ impl Client { webc_error, })?; - let res = AdapterDispatcher::to_chat_stream(model, reqwest_builder, options_set)?; + // -- Bind the response observer exec hook (if set) so it fires when the lazy send + // resolves inside the stream — on the response head, before the status check and + // before the stream body is consumed (also on 4xx/5xx). + let response_observer = self + .config() + .response_observer() + .map(|observer| BoundResponseObserver::new(observer.clone(), model.clone())); + + let res = AdapterDispatcher::to_chat_stream(model, reqwest_builder, options_set, response_observer)?; Ok(res) - })(); + } + .await; match result { Ok(res) => { @@ -319,3 +348,11 @@ impl Client { result } } + +// region: --- Tests + +#[cfg(test)] +#[path = "client_impl_tests.rs"] +mod tests; + +// endregion: --- Tests diff --git a/src/client/client_impl_tests.rs b/src/client/client_impl_tests.rs new file mode 100644 index 00000000..d979a326 --- /dev/null +++ b/src/client/client_impl_tests.rs @@ -0,0 +1,418 @@ +//! Offline tests for the per-request exec hooks (`PayloadInterceptor` / `ResponseObserver`) +//! on the chat exec paths (`exec_chat` and `exec_chat_stream`), using a local one-shot +//! HTTP server (no network, no provider keys). + +use crate::adapter::AdapterKind; +use crate::chat::{ChatRequest, ChatStreamEvent}; +use crate::resolver::{AuthData, Endpoint}; +use crate::{Client, Error, ModelIden, ResponseObserver, ServiceTarget}; +use futures::StreamExt; +use reqwest::StatusCode; +use reqwest::header::HeaderMap; +use serde_json::{Value, json}; +use std::future::Future; +use std::pin::Pin; +use std::sync::{Arc, Mutex}; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio::net::TcpListener; + +type Result = core::result::Result>; + +#[tokio::test] +async fn test_client_exec_chat_stream_payload_interceptor_replaces_wire_payload() -> Result<()> { + // -- Setup & Fixtures + let (url, body_rx) = support_spawn_capture_server(support_sse_ok_response()).await?; + let seen: Arc>> = Arc::new(Mutex::new(None)); + let seen_clone = seen.clone(); + let client = Client::builder() + .with_payload_interceptor_fn(move |model_iden: ModelIden, mut payload: Value| -> Option { + *seen_clone.lock().unwrap() = Some((model_iden, payload.clone())); + payload["x_intercepted"] = json!(true); + Some(payload) + }) + .build(); + + // -- Exec + let chat_res = client + .exec_chat_stream( + support_target(&url), + ChatRequest::from_user("Why is the sky red?"), + None, + ) + .await?; + let content = support_collect_content(chat_res).await?; + + // -- Check + assert_eq!(content, "Hello"); + // The interceptor saw the target ModelIden and the original serialized payload. + let (seen_model, seen_payload) = seen.lock().unwrap().take().ok_or("Interceptor should have been called")?; + assert_eq!(seen_model.adapter_kind, AdapterKind::OpenAI); + assert_eq!(&*seen_model.model_name, "gpt-test"); + assert_eq!(seen_payload.get("model").and_then(|v| v.as_str()), Some("gpt-test")); + assert_eq!(seen_payload.get("x_intercepted"), None); + // The replacement payload is what actually went over the wire. + let wire_body = body_rx.await?; + let wire_json: Value = serde_json::from_str(&wire_body)?; + assert_eq!(wire_json.get("x_intercepted"), Some(&json!(true))); + assert_eq!(wire_json.get("model").and_then(|v| v.as_str()), Some("gpt-test")); + + Ok(()) +} + +#[tokio::test] +async fn test_client_exec_chat_stream_response_observer_on_success() -> Result<()> { + // -- Setup & Fixtures + let (url, _body_rx) = support_spawn_capture_server(support_sse_ok_response()).await?; + let (observed, order) = support_new_observer_state(); + let client = Client::builder() + .with_response_observer(support_async_observer(observed.clone(), order.clone())) + .build(); + + // -- Exec + let mut chat_res = client + .exec_chat_stream( + support_target(&url), + ChatRequest::from_user("Why is the sky red?"), + None, + ) + .await?; + // The HTTP send is lazy — nothing observed before the stream is polled. + assert!( + observed.lock().unwrap().is_none(), + "Observer should not fire before first poll" + ); + let mut content = String::new(); + while let Some(event) = chat_res.stream.next().await { + if let ChatStreamEvent::Chunk(chunk) = event? { + if content.is_empty() { + order.lock().unwrap().push("first-chunk".to_string()); + } + content.push_str(&chunk.content); + } + } + + // -- Check + assert_eq!(content, "Hello"); + let (model_iden, status, headers) = observed.lock().unwrap().take().ok_or("Observer should have fired")?; + assert_eq!(&*model_iden.model_name, "gpt-test"); + assert_eq!(status, StatusCode::OK); + assert_eq!( + headers.get("x-obs-test").and_then(|v| v.to_str().ok()), + Some("obs-value") + ); + // The observer fired on the response head, before the stream body was consumed. + assert_eq!( + *order.lock().unwrap(), + vec!["observer".to_string(), "first-chunk".to_string()] + ); + + Ok(()) +} + +#[tokio::test] +async fn test_client_exec_chat_stream_response_observer_on_http_error() -> Result<()> { + // -- Setup & Fixtures + let error_body = r#"{"error":{"message":"rate limited"}}"#; + let raw_response = format!( + "HTTP/1.1 429 Too Many Requests\r\n\ + content-type: application/json\r\n\ + retry-after: 2\r\n\ + content-length: {}\r\n\ + connection: close\r\n\ + \r\n\ + {error_body}", + error_body.len() + ); + let (url, _body_rx) = support_spawn_capture_server(raw_response).await?; + let (observed, _order) = support_new_observer_state(); + let client = Client::builder() + .with_response_observer(support_async_observer(observed.clone(), _order.clone())) + .build(); + + // -- Exec + let mut chat_res = client + .exec_chat_stream( + support_target(&url), + ChatRequest::from_user("Why is the sky red?"), + None, + ) + .await?; + let mut stream_err: Option = None; + while let Some(event) = chat_res.stream.next().await { + if let Err(err) = event { + stream_err = Some(err); + break; + } + } + + // -- Check + // The observer fired on the failing response head. + let (_model_iden, status, headers) = observed.lock().unwrap().take().ok_or("Observer should have fired")?; + assert_eq!(status, StatusCode::TOO_MANY_REQUESTS); + assert_eq!(headers.get("retry-after").and_then(|v| v.to_str().ok()), Some("2")); + // AND the returned HttpError still carries the response headers (block-1 behavior). + let stream_err = stream_err.ok_or("Stream should have yielded an error")?; + let Error::WebStream { error, .. } = stream_err else { + return Err(format!("Should be Error::WebStream, but was: {stream_err}").into()); + }; + let http_err = error + .downcast::() + .map_err(|err| format!("Error should downcast to genai Error, but was: {err}"))?; + match *http_err { + Error::HttpError { + status, body, headers, .. + } => { + assert_eq!(status.as_u16(), 429); + assert_eq!(body, error_body); + assert_eq!(headers.get("retry-after").and_then(|v| v.to_str().ok()), Some("2")); + } + other => return Err(format!("Should be Error::HttpError, but was: {other}").into()), + } + + Ok(()) +} + +#[tokio::test] +async fn test_client_exec_chat_stream_no_hooks_regression() -> Result<()> { + // -- Setup & Fixtures + let (url_baseline, body_rx_baseline) = support_spawn_capture_server(support_sse_ok_response()).await?; + let (url_noop, body_rx_noop) = support_spawn_capture_server(support_sse_ok_response()).await?; + let client_baseline = Client::builder().build(); + // A `None`-returning interceptor must keep the payload unchanged (byte-identical wire body). + let client_noop = Client::builder() + .with_payload_interceptor_fn(|_model_iden: ModelIden, _payload: Value| -> Option { None }) + .build(); + + // -- Exec + let chat_req = ChatRequest::from_user("Why is the sky red?"); + let res_baseline = client_baseline + .exec_chat_stream(support_target(&url_baseline), chat_req.clone(), None) + .await?; + let content_baseline = support_collect_content(res_baseline).await?; + let res_noop = client_noop.exec_chat_stream(support_target(&url_noop), chat_req, None).await?; + let content_noop = support_collect_content(res_noop).await?; + + // -- Check + assert_eq!(content_baseline, "Hello"); + assert_eq!(content_noop, "Hello"); + let body_baseline = body_rx_baseline.await?; + let body_noop = body_rx_noop.await?; + assert_eq!(body_baseline, body_noop, "Wire payload must be byte-identical"); + + Ok(()) +} + +#[tokio::test] +async fn test_client_exec_chat_payload_interceptor_and_observer() -> Result<()> { + // -- Setup & Fixtures + let (url, body_rx) = support_spawn_capture_server(support_json_ok_response()).await?; + let (observed, _order) = support_new_observer_state(); + let client = Client::builder() + .with_payload_interceptor_fn(|_model_iden: ModelIden, mut payload: Value| -> Option { + payload["x_intercepted"] = json!(true); + Some(payload) + }) + .with_response_observer(support_async_observer(observed.clone(), _order.clone())) + .build(); + + // -- Exec + let chat_res = client + .exec_chat( + support_target(&url), + ChatRequest::from_user("Why is the sky red?"), + None, + ) + .await?; + + // -- Check + assert_eq!(chat_res.first_text(), Some("Hello")); + let wire_body = body_rx.await?; + let wire_json: Value = serde_json::from_str(&wire_body)?; + assert_eq!(wire_json.get("x_intercepted"), Some(&json!(true))); + let (model_iden, status, headers) = observed.lock().unwrap().take().ok_or("Observer should have fired")?; + assert_eq!(&*model_iden.model_name, "gpt-test"); + assert_eq!(status, StatusCode::OK); + assert_eq!( + headers.get("x-obs-test").and_then(|v| v.to_str().ok()), + Some("obs-value") + ); + + Ok(()) +} + +#[tokio::test] +async fn test_client_exec_chat_response_observer_on_http_error() -> Result<()> { + // -- Setup & Fixtures + let error_body = r#"{"error":{"message":"boom"}}"#; + let raw_response = format!( + "HTTP/1.1 500 Internal Server Error\r\n\ + content-type: application/json\r\n\ + x-obs-test: obs-value\r\n\ + content-length: {}\r\n\ + connection: close\r\n\ + \r\n\ + {error_body}", + error_body.len() + ); + let (url, _body_rx) = support_spawn_capture_server(raw_response).await?; + let (observed, _order) = support_new_observer_state(); + let client = Client::builder() + .with_response_observer(support_async_observer(observed.clone(), _order.clone())) + .build(); + + // -- Exec + let res = client + .exec_chat( + support_target(&url), + ChatRequest::from_user("Why is the sky red?"), + None, + ) + .await; + + // -- Check + // The observer fired on the failing response head, before the error body was consumed. + let (_model_iden, status, headers) = observed.lock().unwrap().take().ok_or("Observer should have fired")?; + assert_eq!(status, StatusCode::INTERNAL_SERVER_ERROR); + assert_eq!( + headers.get("x-obs-test").and_then(|v| v.to_str().ok()), + Some("obs-value") + ); + // And the call still returns the regular web error. + let err = res.err().ok_or("exec_chat should have failed")?; + let Error::WebModelCall { webc_error, .. } = err else { + return Err(format!("Should be Error::WebModelCall, but was: {err}").into()); + }; + match webc_error { + crate::webc::Error::ResponseFailedStatus { status, body, .. } => { + assert_eq!(status.as_u16(), 500); + assert_eq!(body, error_body); + } + other => return Err(format!("Should be ResponseFailedStatus, but was: {other}").into()), + } + + Ok(()) +} + +// region: --- Support + +/// Builds a fully-resolved ServiceTarget pointing at the local test server (OpenAI adapter). +fn support_target(url: &str) -> ServiceTarget { + ServiceTarget { + endpoint: Endpoint::from_owned(url.to_string()), + auth: AuthData::from_single("test-key"), + model: ModelIden::new(AdapterKind::OpenAI, "gpt-test"), + } +} + +/// Raw SSE success response with one content chunk (OpenAI chat completions shape). +fn support_sse_ok_response() -> String { + let chunk = r#"{"id":"chatcmpl-1","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}"#; + format!( + "HTTP/1.1 200 OK\r\n\ + content-type: text/event-stream\r\n\ + x-obs-test: obs-value\r\n\ + connection: close\r\n\ + \r\n\ + data: {chunk}\n\ndata: [DONE]\n\n" + ) +} + +/// Raw JSON success response (OpenAI chat completions shape) for the non-streaming path. +fn support_json_ok_response() -> String { + let body = r#"{"id":"chatcmpl-1","model":"gpt-test","choices":[{"index":0,"message":{"role":"assistant","content":"Hello"},"finish_reason":"stop"}],"usage":{"prompt_tokens":1,"completion_tokens":1,"total_tokens":2}}"#; + format!( + "HTTP/1.1 200 OK\r\n\ + content-type: application/json\r\n\ + x-obs-test: obs-value\r\n\ + content-length: {}\r\n\ + connection: close\r\n\ + \r\n\ + {body}", + body.len() + ) +} + +type ObservedState = Arc>>; +type OrderState = Arc>>; + +fn support_new_observer_state() -> (ObservedState, OrderState) { + (Arc::new(Mutex::new(None)), Arc::new(Mutex::new(Vec::new()))) +} + +/// Builds an async ResponseObserver that records the observed (model, status, headers) and +/// appends "observer" to the order log (to assert it fired before body consumption). +fn support_async_observer(observed: ObservedState, order: OrderState) -> ResponseObserver { + ResponseObserver::from_observer_async_fn( + move |model_iden: ModelIden, + status: StatusCode, + headers: HeaderMap| + -> Pin + Send>> { + let observed = observed.clone(); + let order = order.clone(); + Box::pin(async move { + *observed.lock().unwrap() = Some((model_iden, status, headers)); + order.lock().unwrap().push("observer".to_string()); + }) + }, + ) +} + +/// Consumes the chat stream and concatenates the text chunks. +async fn support_collect_content(mut chat_res: crate::chat::ChatStreamResponse) -> Result { + let mut content = String::new(); + while let Some(event) = chat_res.stream.next().await { + if let ChatStreamEvent::Chunk(chunk) = event? { + content.push_str(&chunk.content); + } + } + Ok(content) +} + +/// Spawns a one-shot HTTP server that reads the full request (headers + content-length body), +/// sends the captured request body through the returned channel, then answers with the given +/// raw HTTP response. +async fn support_spawn_capture_server( + raw_response: String, +) -> Result<(String, tokio::sync::oneshot::Receiver)> { + let listener = TcpListener::bind("127.0.0.1:0").await?; + let addr = listener.local_addr()?; + let (body_tx, body_rx) = tokio::sync::oneshot::channel::(); + tokio::spawn(async move { + if let Ok((mut socket, _)) = listener.accept().await { + // -- Read the full request: headers, then content-length body bytes. + let mut buf: Vec = Vec::new(); + let mut chunk = [0u8; 4096]; + let body = loop { + let Ok(n) = socket.read(&mut chunk).await else { + break String::new(); + }; + if n == 0 { + break String::new(); + } + buf.extend_from_slice(&chunk[..n]); + if let Some(header_end) = support_find_subslice(&buf, b"\r\n\r\n") { + let headers_txt = String::from_utf8_lossy(&buf[..header_end]).to_lowercase(); + let content_length: usize = headers_txt + .lines() + .find_map(|line| line.strip_prefix("content-length:")) + .and_then(|v| v.trim().parse().ok()) + .unwrap_or(0); + let body_start = header_end + 4; + if buf.len() >= body_start + content_length { + break String::from_utf8_lossy(&buf[body_start..body_start + content_length]).to_string(); + } + } + }; + let _ = body_tx.send(body); + let _ = socket.write_all(raw_response.as_bytes()).await; + let _ = socket.shutdown().await; + } + }); + Ok((format!("http://{addr}/"), body_rx)) +} + +fn support_find_subslice(haystack: &[u8], needle: &[u8]) -> Option { + haystack.windows(needle.len()).position(|window| window == needle) +} + +// endregion: --- Support diff --git a/src/client/config.rs b/src/client/config.rs index 285e5322..2e50aa1a 100644 --- a/src/client/config.rs +++ b/src/client/config.rs @@ -1,6 +1,6 @@ use crate::adapter::{AdapterDispatcher, AdapterKind}; use crate::chat::ChatOptions; -use crate::client::{ModelSpec, ServiceTarget}; +use crate::client::{ModelSpec, PayloadInterceptor, ResponseObserver, ServiceTarget}; use crate::embed::EmbedOptions; use crate::resolver::{AuthData, AuthResolver, Endpoint, ModelMapper, ServiceTargetResolver}; use crate::{Error, ModelIden, Result, WebConfig}; @@ -15,6 +15,8 @@ pub struct ClientConfig { pub(super) chat_options: Option, pub(super) embed_options: Option, pub(super) adapter_kind: Option, + pub(super) payload_interceptor: Option, + pub(super) response_observer: Option, } /// Chainable setters related to the ClientConfig. @@ -61,6 +63,26 @@ impl ClientConfig { self } + /// Sets the PayloadInterceptor (per-request exec hook). + /// + /// Called on each chat exec call (streaming and non-streaming) with the target `ModelIden` + /// and the serialized provider payload, before the HTTP request is built. `Some` replaces + /// the payload; `None` keeps it unchanged. + pub fn with_payload_interceptor(mut self, payload_interceptor: PayloadInterceptor) -> Self { + self.payload_interceptor = Some(payload_interceptor); + self + } + + /// Sets the ResponseObserver (per-request exec hook). + /// + /// Called on each chat exec call (streaming and non-streaming) with the target `ModelIden`, + /// the response `StatusCode`, and the response `HeaderMap`, as soon as the HTTP response + /// arrives and before its body/stream is consumed (also on 4xx/5xx responses). + pub fn with_response_observer(mut self, response_observer: ResponseObserver) -> Self { + self.response_observer = Some(response_observer); + self + } + /// Binds this Client to a single [`AdapterKind`]. /// /// A Client that has configured an [`AuthResolver`] or [`ServiceTargetResolver`] @@ -125,6 +147,16 @@ impl ClientConfig { pub fn adapter_kind(&self) -> Option { self.adapter_kind } + + /// Returns the PayloadInterceptor, if set. + pub fn payload_interceptor(&self) -> Option<&PayloadInterceptor> { + self.payload_interceptor.as_ref() + } + + /// Returns the ResponseObserver, if set. + pub fn response_observer(&self) -> Option<&ResponseObserver> { + self.response_observer.as_ref() + } } /// Resolvers diff --git a/src/client/exec_hooks.rs b/src/client/exec_hooks.rs new file mode 100644 index 00000000..bcfbc509 --- /dev/null +++ b/src/client/exec_hooks.rs @@ -0,0 +1,376 @@ +//! Per-request exec hooks that library users can set on the Client to observe or intercept +//! the chat execution web calls (`exec_chat` and `exec_chat_stream`). +//! +//! - [`PayloadInterceptor`] receives the target [`ModelIden`] and the serialized provider payload +//! (`serde_json::Value`) right before the HTTP request is built, and can replace the payload. +//! +//! - [`ResponseObserver`] receives the target [`ModelIden`], the response `StatusCode`, and the +//! response `HeaderMap` as soon as the HTTP response arrives, before the body/stream is consumed +//! (including on 4xx/5xx responses). +//! +//! Both follow the resolver idiom (see `AuthResolver`): dedicated types with sync and async +//! function variants, installed via the `ClientBuilder` and stored in the `ClientConfig`. + +use crate::ModelIden; +use reqwest::StatusCode; +use reqwest::header::HeaderMap; +use serde_json::Value; +use std::pin::Pin; +use std::sync::Arc; + +// region: --- PayloadInterceptor + +/// Holder for the payload interceptor function. +/// +/// The interceptor is called once per chat exec call (streaming and non-streaming), after the +/// adapter serialized the provider payload and before the HTTP request is built. Returning +/// `Some(value)` replaces the payload sent over the wire; returning `None` keeps it unchanged. +/// +/// Note: When an interceptor is set, the payload is cloned once per request to hand it to the +/// interceptor by value (no clone occurs when no interceptor is configured). +#[derive(Debug, Clone)] +pub enum PayloadInterceptor { + /// The `PayloadInterceptorFn` trait object (sync). + InterceptorFn(Arc>), + /// The `PayloadInterceptorAsyncFn` trait object (async). + InterceptorAsyncFn(Arc>), +} + +impl PayloadInterceptor { + /// Create a new `PayloadInterceptor` from a sync interceptor function. + pub fn from_interceptor_fn(interceptor_fn: impl IntoPayloadInterceptorFn) -> Self { + PayloadInterceptor::InterceptorFn(interceptor_fn.into_interceptor_fn()) + } + + /// Create a new `PayloadInterceptor` from an async interceptor function. + pub fn from_interceptor_async_fn(interceptor_fn: impl IntoPayloadInterceptorAsyncFn) -> Self { + PayloadInterceptor::InterceptorAsyncFn(interceptor_fn.into_async_interceptor_fn()) + } +} + +impl PayloadInterceptor { + pub(crate) async fn intercept(&self, model_iden: ModelIden, payload: Value) -> Option { + match self { + PayloadInterceptor::InterceptorFn(interceptor_fn) => interceptor_fn.clone().exec_fn(model_iden, payload), + PayloadInterceptor::InterceptorAsyncFn(interceptor_fn) => interceptor_fn.exec_fn(model_iden, payload).await, + } + } +} + +// endregion: --- PayloadInterceptor + +// region: --- PayloadInterceptorFn + +/// The `PayloadInterceptorFn` trait object (sync variant). +pub trait PayloadInterceptorFn: Send + Sync { + /// Execute the interceptor. `Some` replaces the payload; `None` keeps it unchanged. + fn exec_fn(&self, model_iden: ModelIden, payload: Value) -> Option; + + /// Clone the trait object. + fn clone_box(&self) -> Box; +} + +/// `PayloadInterceptorFn` blanket implementation for any function matching the signature. +impl PayloadInterceptorFn for F +where + F: FnOnce(ModelIden, Value) -> Option + Send + Sync + Clone + 'static, +{ + fn exec_fn(&self, model_iden: ModelIden, payload: Value) -> Option { + (self.clone())(model_iden, payload) + } + + fn clone_box(&self) -> Box { + Box::new(self.clone()) + } +} + +impl Clone for Box { + fn clone(&self) -> Self { + self.clone_box() + } +} + +impl std::fmt::Debug for dyn PayloadInterceptorFn { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "PayloadInterceptorFn") + } +} + +/// Custom and convenient trait used in the `PayloadInterceptor::from_interceptor_fn` argument. +pub trait IntoPayloadInterceptorFn { + /// Convert the argument into a `PayloadInterceptorFn` trait object. + fn into_interceptor_fn(self) -> Arc>; +} + +impl IntoPayloadInterceptorFn for Arc> { + fn into_interceptor_fn(self) -> Arc> { + self + } +} + +// Implement `IntoPayloadInterceptorFn` for closures. +impl IntoPayloadInterceptorFn for F +where + F: FnOnce(ModelIden, Value) -> Option + Send + Sync + Clone + 'static, +{ + fn into_interceptor_fn(self) -> Arc> { + Arc::new(Box::new(self)) + } +} + +// endregion: --- PayloadInterceptorFn + +// region: --- PayloadInterceptorAsyncFn + +/// The `PayloadInterceptorAsyncFn` trait object (async variant). +pub trait PayloadInterceptorAsyncFn: Send + Sync { + /// Execute the interceptor. `Some` replaces the payload; `None` keeps it unchanged. + fn exec_fn(&self, model_iden: ModelIden, payload: Value) -> Pin> + Send>>; + + /// Clone the trait object. + fn clone_box(&self) -> Box; +} + +impl PayloadInterceptorAsyncFn for F +where + F: Fn(ModelIden, Value) -> Pin> + Send>> + Send + Sync + Clone + 'static, +{ + fn exec_fn(&self, model_iden: ModelIden, payload: Value) -> Pin> + Send>> { + self(model_iden, payload) + } + + fn clone_box(&self) -> Box { + Box::new(self.clone()) + } +} + +impl Clone for Box { + fn clone(&self) -> Self { + self.clone_box() + } +} + +impl std::fmt::Debug for dyn PayloadInterceptorAsyncFn { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "PayloadInterceptorAsyncFn") + } +} + +/// Custom and convenient trait used in the `PayloadInterceptor::from_interceptor_async_fn` argument. +pub trait IntoPayloadInterceptorAsyncFn { + /// Convert the argument into a `PayloadInterceptorAsyncFn` trait object. + fn into_async_interceptor_fn(self) -> Arc>; +} + +impl IntoPayloadInterceptorAsyncFn for Arc> { + fn into_async_interceptor_fn(self) -> Arc> { + self + } +} + +impl IntoPayloadInterceptorAsyncFn for F +where + F: Fn(ModelIden, Value) -> Pin> + Send>> + Send + Sync + Clone + 'static, +{ + fn into_async_interceptor_fn(self) -> Arc> { + Arc::new(Box::new(self)) + } +} + +// endregion: --- PayloadInterceptorAsyncFn + +// region: --- ResponseObserver + +/// Holder for the response observer function. +/// +/// The observer is called once per chat exec call (streaming and non-streaming) with the target +/// [`ModelIden`], the response `StatusCode`, and the response `HeaderMap`, as soon as the HTTP +/// response arrives and before its body/stream is consumed. It also fires on 4xx/5xx responses. +/// +/// Note: On the streaming path, the HTTP request is sent lazily on the first stream poll, so the +/// observer fires during stream consumption (not at `exec_chat_stream` return time). +#[derive(Debug, Clone)] +pub enum ResponseObserver { + /// The `ResponseObserverFn` trait object (sync). + ObserverFn(Arc>), + /// The `ResponseObserverAsyncFn` trait object (async). + ObserverAsyncFn(Arc>), +} + +impl ResponseObserver { + /// Create a new `ResponseObserver` from a sync observer function. + pub fn from_observer_fn(observer_fn: impl IntoResponseObserverFn) -> Self { + ResponseObserver::ObserverFn(observer_fn.into_observer_fn()) + } + + /// Create a new `ResponseObserver` from an async observer function. + pub fn from_observer_async_fn(observer_fn: impl IntoResponseObserverAsyncFn) -> Self { + ResponseObserver::ObserverAsyncFn(observer_fn.into_async_observer_fn()) + } +} + +impl ResponseObserver { + pub(crate) async fn observe(&self, model_iden: ModelIden, status: StatusCode, headers: HeaderMap) { + match self { + ResponseObserver::ObserverFn(observer_fn) => observer_fn.clone().exec_fn(model_iden, status, headers), + ResponseObserver::ObserverAsyncFn(observer_fn) => observer_fn.exec_fn(model_iden, status, headers).await, + } + } +} + +// endregion: --- ResponseObserver + +// region: --- ResponseObserverFn + +/// The `ResponseObserverFn` trait object (sync variant). +pub trait ResponseObserverFn: Send + Sync { + /// Execute the observer with the response status and headers. + fn exec_fn(&self, model_iden: ModelIden, status: StatusCode, headers: HeaderMap); + + /// Clone the trait object. + fn clone_box(&self) -> Box; +} + +/// `ResponseObserverFn` blanket implementation for any function matching the signature. +impl ResponseObserverFn for F +where + F: FnOnce(ModelIden, StatusCode, HeaderMap) + Send + Sync + Clone + 'static, +{ + fn exec_fn(&self, model_iden: ModelIden, status: StatusCode, headers: HeaderMap) { + (self.clone())(model_iden, status, headers) + } + + fn clone_box(&self) -> Box { + Box::new(self.clone()) + } +} + +impl Clone for Box { + fn clone(&self) -> Self { + self.clone_box() + } +} + +impl std::fmt::Debug for dyn ResponseObserverFn { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "ResponseObserverFn") + } +} + +/// Custom and convenient trait used in the `ResponseObserver::from_observer_fn` argument. +pub trait IntoResponseObserverFn { + /// Convert the argument into a `ResponseObserverFn` trait object. + fn into_observer_fn(self) -> Arc>; +} + +impl IntoResponseObserverFn for Arc> { + fn into_observer_fn(self) -> Arc> { + self + } +} + +// Implement `IntoResponseObserverFn` for closures. +impl IntoResponseObserverFn for F +where + F: FnOnce(ModelIden, StatusCode, HeaderMap) + Send + Sync + Clone + 'static, +{ + fn into_observer_fn(self) -> Arc> { + Arc::new(Box::new(self)) + } +} + +// endregion: --- ResponseObserverFn + +// region: --- ResponseObserverAsyncFn + +/// The `ResponseObserverAsyncFn` trait object (async variant). +pub trait ResponseObserverAsyncFn: Send + Sync { + /// Execute the observer with the response status and headers. + fn exec_fn( + &self, + model_iden: ModelIden, + status: StatusCode, + headers: HeaderMap, + ) -> Pin + Send>>; + + /// Clone the trait object. + fn clone_box(&self) -> Box; +} + +impl ResponseObserverAsyncFn for F +where + F: Fn(ModelIden, StatusCode, HeaderMap) -> Pin + Send>> + Send + Sync + Clone + 'static, +{ + fn exec_fn( + &self, + model_iden: ModelIden, + status: StatusCode, + headers: HeaderMap, + ) -> Pin + Send>> { + self(model_iden, status, headers) + } + + fn clone_box(&self) -> Box { + Box::new(self.clone()) + } +} + +impl Clone for Box { + fn clone(&self) -> Self { + self.clone_box() + } +} + +impl std::fmt::Debug for dyn ResponseObserverAsyncFn { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "ResponseObserverAsyncFn") + } +} + +/// Custom and convenient trait used in the `ResponseObserver::from_observer_async_fn` argument. +pub trait IntoResponseObserverAsyncFn { + /// Convert the argument into a `ResponseObserverAsyncFn` trait object. + fn into_async_observer_fn(self) -> Arc>; +} + +impl IntoResponseObserverAsyncFn for Arc> { + fn into_async_observer_fn(self) -> Arc> { + self + } +} + +impl IntoResponseObserverAsyncFn for F +where + F: Fn(ModelIden, StatusCode, HeaderMap) -> Pin + Send>> + Send + Sync + Clone + 'static, +{ + fn into_async_observer_fn(self) -> Arc> { + Arc::new(Box::new(self)) + } +} + +// endregion: --- ResponseObserverAsyncFn + +// region: --- BoundResponseObserver + +/// Crate plumbing: a [`ResponseObserver`] bound to the [`ModelIden`] of the in-flight request. +/// +/// Carried into the web layer (`WebStream` and friends) so the observer can fire when the +/// `reqwest::Response` first materializes — before any status check or body consumption — +/// without the web layer having to know about model resolution. +#[derive(Debug, Clone)] +pub(crate) struct BoundResponseObserver { + model_iden: ModelIden, + observer: ResponseObserver, +} + +impl BoundResponseObserver { + pub(crate) fn new(observer: ResponseObserver, model_iden: ModelIden) -> Self { + Self { model_iden, observer } + } + + pub(crate) async fn observe(&self, status: StatusCode, headers: HeaderMap) { + self.observer.observe(self.model_iden.clone(), status, headers).await; + } +} + +// endregion: --- BoundResponseObserver diff --git a/src/client/mod.rs b/src/client/mod.rs index 3fbb6d75..7bbfc898 100644 --- a/src/client/mod.rs +++ b/src/client/mod.rs @@ -9,6 +9,7 @@ mod builder; mod client_impl; mod client_types; mod config; +mod exec_hooks; mod headers; mod model_spec; mod service_target; @@ -17,6 +18,7 @@ mod web_config; pub use builder::*; pub use client_types::*; pub use config::*; +pub use exec_hooks::*; pub use headers::*; pub use model_spec::*; pub use service_target::*; diff --git a/src/error.rs b/src/error.rs index c36b0e6e..8d5983b4 100644 --- a/src/error.rs +++ b/src/error.rs @@ -3,6 +3,7 @@ use crate::chat::ChatRole; use crate::{ModelIden, resolver, webc}; use derive_more::{Display, From}; use reqwest::StatusCode; +use reqwest::header::HeaderMap; use value_ext::JsonValueExtError; /// Type alias for boxed errors that are Send + Sync @@ -123,6 +124,9 @@ Cause:\n{cause} status: StatusCode, canonical_reason: String, body: String, + /// Response headers of the failed HTTP call (e.g., `retry-after`, `retry-after-ms`, `x-should-retry`), + /// so that downstream retry layers can honor provider-requested retry delays. + headers: Box, }, // -- Modules diff --git a/src/otel/error.rs b/src/otel/error.rs index b7c14d94..987a3913 100644 --- a/src/otel/error.rs +++ b/src/otel/error.rs @@ -49,6 +49,7 @@ pub fn error_type(error: &Error) -> String { // -- Adapter support Error::AdapterNotSupported { .. } => "adapter_not_supported".to_string(), Error::AdapterKindMismatch { .. } => "adapter_kind_mismatch".to_string(), + Error::CacheBreakpointNoEligibleContent { .. } => "cache_breakpoint_no_eligible_content".to_string(), // -- Internals / externals Error::Internal(_) => "internal".to_string(), @@ -97,6 +98,7 @@ mod tests { status: StatusCode::TOO_MANY_REQUESTS, canonical_reason: "Too Many Requests".to_string(), body: String::new(), + headers: Box::new(HeaderMap::new()), }; assert_eq!(error_type(&error), "429"); } diff --git a/src/webc/event_source_stream.rs b/src/webc/event_source_stream.rs index 0bdf1308..6198d71a 100644 --- a/src/webc/event_source_stream.rs +++ b/src/webc/event_source_stream.rs @@ -32,6 +32,15 @@ impl EventSourceStream { opened: false, } } + + /// Sets the (optional) response observer exec hook on the inner `WebStream`. + /// + /// Note: The observer fires when the lazy HTTP send resolves (on the response head), not on + /// the synthetic `Event::Open`, which is emitted before any HTTP activity. + pub fn with_response_observer(mut self, response_observer: Option) -> Self { + self.inner = self.inner.with_response_observer(response_observer); + self + } } impl Stream for EventSourceStream { diff --git a/src/webc/web_client.rs b/src/webc/web_client.rs index cce44855..107ab20b 100644 --- a/src/webc/web_client.rs +++ b/src/webc/web_client.rs @@ -1,4 +1,5 @@ use crate::Headers; +use crate::client::BoundResponseObserver; use crate::webc::{Error, Result}; use reqwest::header::HeaderMap; use reqwest::{Method, RequestBuilder, StatusCode}; @@ -55,10 +56,27 @@ impl WebClient { } pub async fn do_post(&self, url: &str, headers: &Headers, content: &Value) -> Result { + self.do_post_with_observer(url, headers, content, None).await + } + + /// Same as `do_post`, but fires the (optional) bound response observer on the response head + /// (status + headers) as soon as the response arrives, before the body is consumed — + /// including on non-success statuses. + pub async fn do_post_with_observer( + &self, + url: &str, + headers: &Headers, + content: &Value, + response_observer: Option<&BoundResponseObserver>, + ) -> Result { let reqwest_builder = self.new_req_builder(url, headers, content)?; let reqwest_res = reqwest_builder.send().await?; + if let Some(observer) = response_observer { + observer.observe(reqwest_res.status(), reqwest_res.headers().clone()).await; + } + let response = WebResponse::from_reqwest_response(reqwest_res).await?; Ok(response) diff --git a/src/webc/web_stream.rs b/src/webc/web_stream.rs index fbf24651..790d85c5 100644 --- a/src/webc/web_stream.rs +++ b/src/webc/web_stream.rs @@ -6,6 +6,7 @@ use std::collections::VecDeque; use std::pin::Pin; use std::task::{Context, Poll}; +use crate::client::BoundResponseObserver; use crate::error::{BoxError, Error as GenaiError}; /// WebStream is a simple web stream implementation that splits the stream messages by a given delimiter. @@ -28,6 +29,9 @@ pub struct WebStream { // When a multi-byte character is split across TCP/HTTP chunk boundaries, // the trailing bytes are carried over to be prepended to the next chunk. utf8_carry: Vec, + // Optional response observer exec hook, fired once when the lazy send resolves — + // on the response head, before the status check and before the body is consumed. + response_observer: Option, } pub enum StreamMode { @@ -49,6 +53,7 @@ impl WebStream { partial_message: None, remaining_messages: None, utf8_carry: Vec::new(), + response_observer: None, } } @@ -61,8 +66,15 @@ impl WebStream { partial_message: None, remaining_messages: None, utf8_carry: Vec::new(), + response_observer: None, } } + + /// Sets the (optional) response observer exec hook to fire on the response head. + pub fn with_response_observer(mut self, response_observer: Option) -> Self { + self.response_observer = response_observer; + self + } } impl Stream for WebStream { @@ -90,6 +102,9 @@ impl Stream for WebStream { // For error responses, we need to read the body to get the error message // Store a future that reads the body and returns an error let error_future = async move { + // Capture the headers while the response is still in hand + // (e.g., `retry-after`, `retry-after-ms`, `x-should-retry` for retry layers) + let headers = response.headers().clone(); let body = response .text() .await @@ -98,6 +113,7 @@ impl Stream for WebStream { status, canonical_reason: status.canonical_reason().unwrap_or("Unknown").to_string(), body, + headers: Box::new(headers), })) }; this.response_future = Some(Box::pin(error_future)); @@ -192,7 +208,17 @@ impl Stream for WebStream { } if let Some(reqwest_builder) = this.reqwest_builder.take() { - let fut = async move { reqwest_builder.send().await.map_err(|e| Box::new(e) as BoxError) }; + let response_observer = this.response_observer.take(); + let fut = async move { + let response = reqwest_builder.send().await.map_err(|e| Box::new(e) as BoxError)?; + // Fire the response observer as soon as the response head is in hand — + // before the status check above and before the body is consumed — + // so it also fires on 4xx/5xx responses. + if let Some(observer) = response_observer { + observer.observe(response.status(), response.headers().clone()).await; + } + Ok(response) + }; this.response_future = Some(Box::pin(fut)); continue; } @@ -264,3 +290,11 @@ fn process_buff_string_delimited( candidate_message, }) } + +// region: --- Tests + +#[cfg(test)] +#[path = "web_stream_tests.rs"] +mod tests; + +// endregion: --- Tests diff --git a/src/webc/web_stream_tests.rs b/src/webc/web_stream_tests.rs new file mode 100644 index 00000000..1a652e53 --- /dev/null +++ b/src/webc/web_stream_tests.rs @@ -0,0 +1,81 @@ +use super::*; +use crate::error::Error as GenaiError; +use futures::StreamExt; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio::net::TcpListener; + +type Result = core::result::Result>; + +#[tokio::test] +async fn test_web_stream_http_error_captures_headers() -> Result<()> { + // -- Setup & Fixtures + let body = r#"{"error":{"message":"rate limited"}}"#; + let raw_response = format!( + "HTTP/1.1 429 Too Many Requests\r\n\ + content-type: application/json\r\n\ + retry-after: 2\r\n\ + retry-after-ms: 1500\r\n\ + x-should-retry: true\r\n\ + content-length: {}\r\n\ + connection: close\r\n\ + \r\n\ + {body}", + body.len() + ); + let url = support_spawn_one_shot_http_server(raw_response).await?; + let reqwest_builder = reqwest::Client::new().post(&url).json(&serde_json::json!({"stream": true})); + let mut web_stream = WebStream::new_with_sse(reqwest_builder); + + // -- Exec + let first_item = web_stream.next().await.ok_or("Should have a first stream item")?; + + // -- Check + let err = first_item.err().ok_or("First stream item should be an error")?; + let err = err + .downcast::() + .map_err(|err| format!("Error should downcast to genai Error, but was: {err}"))?; + match *err { + GenaiError::HttpError { + status, + canonical_reason, + body: err_body, + headers, + } => { + assert_eq!(status.as_u16(), 429); + assert_eq!(canonical_reason, "Too Many Requests"); + assert_eq!(err_body, body); + assert_eq!(headers.get("retry-after").and_then(|v| v.to_str().ok()), Some("2")); + assert_eq!( + headers.get("retry-after-ms").and_then(|v| v.to_str().ok()), + Some("1500") + ); + assert_eq!( + headers.get("x-should-retry").and_then(|v| v.to_str().ok()), + Some("true") + ); + } + other => return Err(format!("Should be an Error::HttpError, but was: {other}").into()), + } + + Ok(()) +} + +// region: --- Support + +/// Spawns a one-shot HTTP server that answers the first request with the given raw HTTP response. +async fn support_spawn_one_shot_http_server(raw_response: String) -> Result { + let listener = TcpListener::bind("127.0.0.1:0").await?; + let addr = listener.local_addr()?; + tokio::spawn(async move { + if let Ok((mut socket, _)) = listener.accept().await { + // Best effort: read the (small) request bytes before responding. + let mut buf = [0u8; 4096]; + let _ = socket.read(&mut buf).await; + let _ = socket.write_all(raw_response.as_bytes()).await; + let _ = socket.shutdown().await; + } + }); + Ok(format!("http://{addr}/")) +} + +// endregion: --- Support