[API Proposal]: Add extensible request routing for IChatClient

[joshuajyue]

Background and motivation

Applications may need to select from multiple IChatClient instances for each request. Selection can depend on message content, request options, availability, cost, latency, or other application-defined policy.

Two related but distinct behaviors are useful:

• Routing selects one client and delegates the request to it.
• Failover may select another client after an invocation fails before streaming output reaches the caller.

RoutingChatClient provides the minimal one-shot abstraction alongside IChatClient in Microsoft.Extensions.AI.Abstractions. Microsoft.Extensions.AI adds SemanticRoutingChatClient for embedding-based one-shot selection, FailoverChatClient for retry and terminal attempt tracking, and OrderedFailoverChatClient for ordered list-based failover.

API proposal

namespace Microsoft.Extensions.AI;

// Microsoft.Extensions.AI.Abstractions
[Experimental("MEAI001")]
public abstract class RoutingChatClient : IChatClient
{
    public static RoutingChatClient Create(
        Func<RoutingContext, CancellationToken, ValueTask<IChatClient>> clientSelector);

    protected abstract ValueTask<IChatClient> SelectClientAsync(
        RoutingContext context,
        CancellationToken cancellationToken);

    public virtual Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default);

    public virtual IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default);

    public virtual object? GetService(Type serviceType, object? serviceKey = null);

    public void Dispose();

    protected virtual void Dispose(bool disposing);
}

[Experimental("MEAI001")]
public class RoutingContext
{
    public RoutingContext(
        IEnumerable<ChatMessage> messages,
        ChatOptions? chatOptions);

    public IEnumerable<ChatMessage> Messages { get; set; }

    public ChatOptions? ChatOptions { get; set; }
}

// Microsoft.Extensions.AI
[Experimental("MEAI001")]
public abstract class FailoverChatClient : RoutingChatClient
{
    public int? MaximumAttemptsPerRequest { get; set; }

    protected abstract ValueTask<IChatClient?> SelectNextClientAsync(
        RoutingContext context,
        FailoverChatClientAttempt previousAttempt,
        CancellationToken cancellationToken);

    protected virtual ValueTask OnRoutingCompletedAsync(
        RoutingContext context,
        FailoverChatClientAttempt? terminalAttempt,
        CancellationToken cancellationToken);
}

[Experimental("MEAI001")]
public sealed class FailoverChatClientAttempt
{
    public IChatClient Client { get; }

    public TimeSpan Duration { get; }

    public TimeSpan? TimeToFirstUpdate { get; }

    public Exception? Exception { get; }

    public bool ResponseCompleted { get; }

    public bool OutputCommitted { get; }
}

[Experimental("MEAI001")]
public sealed class OrderedFailoverChatClient : FailoverChatClient
{
    public OrderedFailoverChatClient(
        IReadOnlyList<IChatClient> clients,
        bool leaveOpen = false);
}

[Experimental("MEAI001")]
public sealed class SemanticRoutingChatClient : RoutingChatClient
{
    public SemanticRoutingChatClient(
        IEmbeddingGenerator<string, Embedding<float>> embeddingGenerator,
        IReadOnlyDictionary<IChatClient, IReadOnlyList<string>> clientProfiles,
        IChatClient defaultClient,
        float scoreThreshold = 0.3f,
        bool leaveOpen = false);
}

One-shot routing

For each request or streaming enumeration, RoutingChatClient:

1. Creates one RoutingContext.
2. Calls SelectClientAsync once.
3. Invokes the selected client once.
4. Propagates its response or failure unchanged.

SelectClientAsync must return an initial client. A policy that cannot select one throws its own meaningful exception.

The callback factory covers simple policies without requiring a derived class:

using RoutingChatClient router = RoutingChatClient.Create(
    (context, cancellationToken) =>
        new(IsComplex(context.Messages) ? capable : fast));

The callback is invoked once per request or enumeration. Selected clients remain caller-owned.

Semantic routing

SemanticRoutingChatClient is a sealed one-shot router. It lazily embeds and caches app-provided example utterances, embeds the last user message for each request, and selects the client with the highest cosine similarity. The required default client is used when no user message is available or the best score is below scoreThreshold.

using var semantic = new SemanticRoutingChatClient(
    embeddingGenerator,
    new Dictionary<IChatClient, IReadOnlyList<string>>
    {
        [codeClient] = ["debug this code", "implement an API"],
        [writingClient] = ["rewrite this paragraph", "draft an announcement"],
    },
    defaultClient: generalClient);

The router owns its configured clients and embedding generator by default; leaveOpen: true leaves them caller-owned. It performs no failover. Applications can compose fallback explicitly:

using var client = new OrderedFailoverChatClient(
    [semantic, defaultFallback]);

Failover lifecycle

FailoverChatClient uses inherited SelectClientAsync for initial selection. After an eligible pre-output failure, it calls SelectNextClientAsync with the failed attempt.

• Returning another client continues failover.
• Returning null stops failover and rethrows the previous attempt's exception.
• Cancellation never causes reselection.
• Streaming failures can cause reselection only before any update is exposed.
• MaximumAttemptsPerRequest optionally bounds client invocations.
• OnRoutingCompletedAsync runs exactly once after the request context is created.
• Its terminal attempt is non-null whenever a client was invoked and null only when initial selection terminated before invocation.
• Only the final invoked attempt reaches the completion hook; nonterminal failures go to SelectNextClientAsync.

sealed class ApplicationFailoverClient(
    IChatClient fast,
    IChatClient capable) : FailoverChatClient
{
    protected override ValueTask<IChatClient> SelectClientAsync(
        RoutingContext context,
        CancellationToken cancellationToken) =>
        new(IsComplex(context.Messages) ? capable : fast);

    protected override ValueTask<IChatClient?> SelectNextClientAsync(
        RoutingContext context,
        FailoverChatClientAttempt previousAttempt,
        CancellationToken cancellationToken) =>
        ReferenceEquals(previousAttempt.Client, fast)
            ? new(capable)
            : new((IChatClient?)null);

    protected override ValueTask OnRoutingCompletedAsync(
        RoutingContext context,
        FailoverChatClientAttempt? terminalAttempt,
        CancellationToken cancellationToken)
    {
        RemoveRequestState(context);
        RecordFinalOutcome(terminalAttempt);
        return default;
    }
}

Outcome	ResponseCompleted	Exception
Successful	true	null
Failed	false	Non-null
Canceled	false	Typically an OperationCanceledException
Streaming stopped early	false	null

OutputCommitted independently indicates whether streaming output reached the caller. Once output is committed, a later failure is propagated rather than causing failover. If invocation and enumerator disposal both throw, the disposal exception is reported. Completion-hook exceptions propagate and replace any response or exception already produced by the request.

The same RoutingContext instance is supplied to initial selection, every next-client selection, and terminal completion. This allows a failover policy to key request-local state by context identity and remove it deterministically in the completion hook.

Optional non-failover tracking

A three-level prototype that adds terminal tracking without failover is preserved separately in this gist. It is intentionally not part of this proposal while we evaluate whether non-failover tracking warrants another public base class.

Related work: reasoning history

Chat history may contain provider- or model-specific TextReasoningContent, including encrypted ProtectedData and replayable RawRepresentation. Routing such history to an incompatible destination may cause request failures. Routing clients cannot safely infer compatibility because distinct clients may represent compatible configurations.

Applications may apply ReasoningChatReducer when their routing policy identifies an incompatible transition. It can remove all reasoning content or preserve visible reasoning text while removing provider-specific protected and raw state.

Risks

• A failover selector that repeatedly returns failing clients can loop indefinitely unless it returns null, throws, or uses MaximumAttemptsPerRequest.
• A routing client may serve concurrent requests; shared policy state must be thread-safe.
• A streaming response cannot safely move to another client after output reaches the caller.
• Derived implementations that retain clients must define and implement their ownership and disposal behavior.
• The messages sequence may be enumerated during selection and invocation; callers should supply a repeatable sequence or materialize it when required.
• Replacing RoutingContext.Messages or RoutingContext.ChatOptions changes the values supplied to the selected client and subsequent failover selections.