Exec hooks, streaming error headers, tool-result binary parts, Anthropic URL images, embedded tool-response fixes

CHANGELOG.md

@@ -2,12 +2,18 @@
 
 ## v0.7.0-beta.x (see [genai versions](https://crates.io/crates/genai/versions))
 
+- `!` API CHANGE - `Error::HttpError` adds a `headers: Box<HeaderMap>` field carrying the response headers of failed streaming HTTP calls (e.g., `retry-after`, `retry-after-ms`, `x-should-retry`), matching the non-streaming `webc::Error::ResponseFailedStatus`, so downstream retry layers can honor provider-requested retry delays. Exhaustive matchers/constructors of `Error::HttpError` must add the `headers` field (or match with `..`).
 - `!` API CHANGE - `Tool` adds the public `custom_format: Option<Value>` field for provider-native freeform custom-tool formats. Downstream `Tool` struct literals must add `custom_format: None`, or preferably migrate to `Tool::new(...)` and builder methods. `Tool::with_custom_format(...)` is the new builder API.
+- `!` API CHANGE - `ToolResponse` adds the public `parts: Option<Vec<Binary>>` field for binary tool-result attachments (e.g., screenshots produced by agentic tools). Downstream `ToolResponse` struct literals must add `parts: None`, or preferably migrate to `ToolResponse::new(...)` and the new `ToolResponse::with_parts(...)` / `ToolResponse::append_binary(...)` builders. Image parts serialize natively where the wire supports them (Anthropic `tool_result`, Bedrock Converse `toolResult`, OpenAI Responses `function_call_output`) and ride in a follow-up user message elsewhere (OpenAI Chat Completions-compatible providers, Gemini, Ollama). Non-image parts are skipped with a warning, and text-only tool responses keep their exact previous serialization on every adapter.
+- `!` BEHAVIOR CHANGE - A `ContentPart::ToolResponse` embedded in an Assistant-role message now fails serialization with the existing `Error::MessageContentTypeNotSupported` (no new error variant) on every adapter: OpenAI Chat Completions and Responses, Anthropic, Bedrock Converse, Gemini, and Ollama native — including the delegating providers that reuse those serializers (Vertex Claude models, opencode_go, minimax, github_copilot, ollama_cloud, ...). Previously Anthropic, Bedrock Converse, and Gemini silently dropped the embedded response, Ollama garbled it into the assistant `content`, and the OpenAI serializers silently dropped it. No provider wire represents a tool result authored by the assistant, so the shape has no faithful translation; the error's cause points at the supported Tool-role message shape (e.g., `ChatMessage::from(ToolResponse)`). The crate-wide rule is now: supported shapes are translated faithfully, unsupported shapes fail loudly, nothing vanishes.
+- `+` `Client` adds per-request exec hooks, following the resolver idiom (dedicated types with sync/async function variants, installed via the `ClientBuilder`, stored in the `ClientConfig`). `ClientBuilder::with_payload_interceptor(...)` / `with_payload_interceptor_fn(...)` set a `PayloadInterceptor` called on each chat exec call (streaming and non-streaming) with the target `ModelIden` and the serialized provider payload (`serde_json::Value`) before the HTTP request is built — returning `Some(value)` replaces the payload, `None` keeps it unchanged. `ClientBuilder::with_response_observer(...)` / `with_response_observer_fn(...)` set a `ResponseObserver` called with the `ModelIden`, response `StatusCode`, and `HeaderMap` as soon as the HTTP response arrives and before its body/stream is consumed — including on 4xx/5xx responses (on the streaming path, the send is lazy, so the observer fires during the first stream poll, before the status check and the `Error::HttpError` construction). Chat exec paths only (`exec_chat` / `exec_chat_stream`); embeddings and model listing are not hooked.
 - `+` New Providers:
   - AtlasCloud - default env: `ATLASCLOUD_API_KEY`, Adapter: OpenAI, endpoint: `https://api.atlascloud.ai/v1/` (activated on the `atlascloud::` namespace) (PR #259)
   - Qwen Cloud - default env: `QWEN_CLOUD_API_KEY`, Adapter: OpenAI, endpoint: `https://dashscope-intl.aliyuncs.com/compatible-mode/v1/` (activated on the `qwen_cloud::` namespace)
   - Kimi - default env: `KIMI_API_KEY`, Adapter: OpenAI, endpoint: `https://api.moonshot.ai/v1/` (activated on the `kimi::` namespace or `kimi` model prefix, moonshot.ai)
 - Anthropic:
+  - `+` Serialize `ToolResponse.parts` image attachments as base64 `image` blocks inside the `tool_result` content array (after the text block). Text-only tool responses keep the legacy plain-string `content`. Non-image parts are skipped with a warning, since Anthropic `tool_result` content only accepts text and image blocks.
+  - `^` Support URL image sources in user messages and tool results: `BinarySource::Url` images now serialize natively as `{"type": "image", "source": {"type": "url", "url": ...}}` blocks instead of being silently omitted with a warning (the Anthropic Messages API natively supports URL image sources). Base64 image serialization is unchanged. Delegating providers that reuse the Anthropic serializers (minimax, the `baidu-coding-anthropic` namespace, Vertex Claude models, opencode_go minimax models) inherit this automatically; whether a given gateway accepts URL sources is provider-side.
   - `+` Expose streaming SSE ping messages as provider-neutral `ChatStreamEvent::Heartbeat` events, allowing callers to distinguish a live long-running stream from a stall. (PR #271)
   - `+` Add prompt caching on tools via `Tool::with_cache_control`, and make request-level `ChatOptions::with_cache_control` automatically apply a cache breakpoint to the static (tools+system) prefix, which was previously ignored. `Ephemeral24h` is documented as clamped to Anthropic's max `1h` TTL.
   - `+` Support the `extra_body` `ChatOptions` field, merging extra request body fields. ([#255](https://github.com/jeremychone/rust-genai/pull/255))
@@ -21,6 +27,9 @@
     - Fable and Mythos omit `thinking`, because it is always on and cannot be explicitly disabled.
     - The Anthropic `-zero` model suffix is canonical, while `-none` remains a backward-compatible alias. Both map to `Zero` and are stripped.
 - OpenAI:
+  - `-` Chat Completions and Responses: a `ToolResponse` embedded in a User-role message is now serialized instead of silently dropped (text and all). This user-embedded shape (the Anthropic-style form where tool results ride as user content blocks) is extracted into proper `role:"tool"` messages / `function_call_output` items (`custom_tool_call_output` for custom tool calls) emitted before the carrying user message, with images folded into that same user message (`image_url` / `input_image` blocks, no label); a user message left empty by the extraction is omitted. Text/placeholder and custom-output rules match Tool-role serialization, and `call_id`s are serialized as-is (provider-side validation, as elsewhere). The Ollama native serializer (shared by `ollama_cloud`) gets the same extraction — it previously garbled the shape (the response text was inserted as the user `content`, where sibling text parts overwrote it, and image parts were lost): the user-embedded `ToolResponse` now becomes a `role:"tool"` message before the carrying user message, its images ride the existing labeled follow-up user image message (native base64 `images` array), and the same empty-user-message omission applies; in the same stroke, the Ollama Tool-role path now emits one `role:"tool"` message per `ToolResponse` when a Tool-role message carries several, in part order (previously each response's text overwrote the previous, keeping only the last), with their images still batched into that single labeled follow-up user image message. A `ToolResponse` embedded in an Assistant-role message fails loudly instead — see the crate-wide BEHAVIOR CHANGE entry above.
+  - `+` Chat Completions: `ToolResponse.parts` images ride in a follow-up `user` message (`image_url` blocks), batched across a run of consecutive tool messages; the `tool` message keeps its text, or the `"(see attached image)"` placeholder when the result is image-only. Applies to all OpenAI-compatible providers sharing this serializer.
+  - `+` Responses: `ToolResponse.parts` images serialize natively as `input_image` items in the `function_call_output` `output` array (after the `input_text` item). Custom tool-call outputs stay raw strings (with the `"(see attached image)"` / `"(no tool output)"` placeholder rules); their images ride in a follow-up `user` message input item, batched across a run of consecutive tool messages.
   - `+` Support OpenAI Responses freeform custom tools with grammar-constrained raw-string input. Custom tools serialize as `type: "custom"`, custom tool-call input streams incrementally, and round-trips as `custom_tool_call` / `custom_tool_call_output` items. (PR #266)
   - `^` Capture `cache_write_tokens` from prompt-cache usage and normalize it to `Usage.prompt_tokens_details.cache_creation_tokens` for Chat Completions and Responses API payloads.
   - `^` GPT-5.6 and later now use cache opt-in only. Here is how to opt in: (see [PR #260](https://github.com/jeremychone/rust-genai/pull/260))
@@ -34,14 +43,19 @@
   - `+` Sanitize JSON Schema for structured responses and strict tools. (PR #263)
   - `!` Apply the `ReasoningEffort::None` to `ReasoningEffort::Zero` rename mechanically, while preserving provider-specific keyword mappings.
 - Gemini:
+  - `+` `ToolResponse.parts` images ride in a follow-up `user` turn (`inline_data` / `file_data`) emitted after the merged `functionResponse` turn; the `functionResponse` keeps its text, or the `"(see attached image)"` placeholder when the result is image-only. Also applies to Vertex (Google publisher).
   - `^` Forward JSON Schema raw via `responseJsonSchema` and `parametersJsonSchema`. (PR #257)
   - `!` Map `ReasoningEffort::Zero` to a budget of `0`, which might be rejected by the provider on some models.
   - `-` Protect known model names such as `deepseek-r1-zero` from reasoning suffix stripping by using a whitelist in `from_model_name()`.
 - Bedrock:
+  - `+` `ToolResponse.parts` images serialize natively as `image` blocks inside the Converse `toolResult` content array (after the text block).
   - `!` Apply the `ReasoningEffort::None` to `ReasoningEffort::Zero` rename mechanically, with behavior unchanged.
+- Ollama:
+  - `+` `ToolResponse.parts` images (base64 only) ride in a follow-up `user` message via the native `images` array; the `tool` message keeps its text, or the `"(see attached image)"` placeholder when the result is image-only.
 - Cross-provider adapters:
   - `^` Move messages after tools in JSON payloads for better prompt cache utilization. (PR #262)
 - OpenTelemetry:
+  - `-` Fix `otel` feature compilation, by covering the `CacheBreakpointNoEligibleContent` error variant in the `error.type` derivation (broken since v0.7.0-beta.18).
   - `+` Add optional OpenTelemetry GenAI semantic-convention instrumentation behind the new `otel` feature, off by default, using a pure `tracing` bridge with no extra dependencies.
     - Auto-instruments `exec_chat`, `exec_chat_stream`, and `exec_embed` as `gen_ai.*` spans, including operation, provider, request params, server address/port, usage tokens, finish reasons, response id/model, streaming time-to-first-chunk, and `error.type`. Prompt and response content capture is opt-in via `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT`. Adds opt-in `genai::otel` helpers for agent, workflow, and tool spans, plus the evaluation-result event. Export by wiring `tracing-opentelemetry` in the application. See `docs/otel.md` and `examples/c12-otel.rs`.
 

dev/specs/spec-chat.md

@@ -34,7 +34,7 @@ The module exports the following key data structures:
 - **Tooling:**
   - `Tool`: Metadata and schema defining a function the model can call.
   - `ToolCall`: The model's invocation request for a specific tool.
-  - `ToolResponse`: The output returned from executing a tool, matched by call ID.
+  - `ToolResponse`: The output returned from executing a tool, matched by call ID. Carries text `content` plus optional binary `parts` (e.g., screenshots); see `spec-tool.md` for the per-adapter image serialization matrix and the mapping of user-embedded tool responses (the Anthropic-style shape, serialized natively or extracted to standalone tool messages/items per wire). A `ToolResponse` embedded in an Assistant-role message is rejected with a hard error on every adapter — no provider wire supports assistant-authored tool results; use a Tool-role message.
 
 - **Metadata:**
   - `Usage`, `PromptTokensDetails`, `CompletionTokensDetails`: Normalized token usage statistics.

dev/specs/spec-client.md

@@ -14,9 +14,13 @@ The `client` module exposes the following public types:
   - Core execution methods: `exec_chat`, `exec_chat_stream`, `exec_embed`, `embed`, `embed_batch`.
   - Resolution/Discovery methods: `all_model_names`, `resolve_service_target`.
 
-- **`ClientBuilder`**: Provides a fluent interface for constructing a `Client`. Used to set `ClientConfig`, default `ChatOptions`, `EmbedOptions`, and custom resolvers (`AuthResolver`, `ServiceTargetResolver`, `ModelMapper`).
+- **`ClientBuilder`**: Provides a fluent interface for constructing a `Client`. Used to set `ClientConfig`, default `ChatOptions`, `EmbedOptions`, custom resolvers (`AuthResolver`, `ServiceTargetResolver`, `ModelMapper`), and per-request exec hooks (`PayloadInterceptor`, `ResponseObserver`).
 
-- **`ClientConfig`**: Holds the resolved and default configurations used by the `Client`, including resolver functions and default options.
+- **`ClientConfig`**: Holds the resolved and default configurations used by the `Client`, including resolver functions, exec hooks, and default options.
+
+- **`PayloadInterceptor`**: Per-request exec hook (resolver idiom: enum with sync `InterceptorFn` and async `InterceptorAsyncFn` variants, created via `from_interceptor_fn` / `from_interceptor_async_fn`). Called on each chat exec call (streaming and non-streaming) with the target `ModelIden` and the serialized provider payload (`serde_json::Value`), after `to_web_request_data` and before the HTTP request is built. Returning `Some(value)` replaces the payload sent over the wire; `None` keeps it unchanged (the payload is cloned once per request only when an interceptor is set).
+
+- **`ResponseObserver`**: Per-request exec hook (same idiom, `from_observer_fn` / `from_observer_async_fn`). Called on each chat exec call with the target `ModelIden`, the response `StatusCode`, and the response `HeaderMap` as soon as the HTTP response arrives and before its body/stream is consumed — including on 4xx/5xx responses. Chat exec paths only (embeddings and model listing are not hooked).
 
 - **`Headers`**: A simple map wrapper (`HashMap<String, String>`) for managing HTTP headers in requests.
 
@@ -34,7 +38,9 @@ The module is composed of several files that implement the layered client archit
 
 - `config.rs`: Defines `ClientConfig` and the core `resolve_service_target` logic, which orchestrates calls to `ModelMapper`, `AuthResolver`, and `ServiceTargetResolver` before falling back to adapter defaults.
 
-- `client_impl.rs`: Contains the main implementation of the public API methods on `Client`, such as `exec_chat` and `exec_embed`. These methods perform service resolution and delegate to `AdapterDispatcher` for request creation and response parsing.
+- `client_impl.rs`: Contains the main implementation of the public API methods on `Client`, such as `exec_chat` and `exec_embed`. These methods perform service resolution and delegate to `AdapterDispatcher` for request creation and response parsing. The chat exec paths also apply the exec hooks: the `PayloadInterceptor` runs between `to_web_request_data` and the request-builder construction (which makes the `exec_chat_stream` setup an async block), and the `ResponseObserver` is bound to the request's `ModelIden` as a crate-internal `BoundResponseObserver` that is handed to `WebClient::do_post_with_observer` (non-streaming) or threaded through `Adapter::to_chat_stream` into the web stream (streaming).
+
+- `exec_hooks.rs`: Defines the per-request exec hooks `PayloadInterceptor` and `ResponseObserver` (with their sync/async function traits and `Into*` conversion traits, mirroring `AuthResolver`), plus the crate-internal `BoundResponseObserver` pairing an observer with the in-flight request's `ModelIden` so the web layer can fire it without knowing about model resolution.
 
 - `headers.rs`: Implements the `Headers` utility for managing key-value HTTP header maps.
 
@@ -57,3 +63,5 @@ The module is composed of several files that implement the layered client archit
 - **Builder Pattern for Configuration**: `ClientBuilder` enforces configuration before client creation, simplifying object construction and ensuring necessary dependencies are set up correctly.
 
 - **Headers Simplification**: The `Headers` struct abstracts HTTP header management, ensuring that subsequent merges or overrides result in a single, final header value, which is typical for API key authorization overrides.
+
+- **Exec Hooks (Per-Request Observability/Interception)**: `PayloadInterceptor` and `ResponseObserver` give downstream runtimes request auditing and payload-shaping without changing the serialized `ChatOptions` (which stays hooks-free since it is Serialize/Deserialize). On the streaming path, the HTTP send is lazy (performed on the first stream poll inside `WebStream`), so the observer is carried into the stream and fires when the send resolves — before the status check, so it also fires on failing responses (alongside the headers-carrying `Error::HttpError`). The synthetic `Event::Open` of the SSE stream is emitted before any HTTP activity and is deliberately not tied to the observer.