Skip to content

feat: streaming responses for /turn (SSE) — surface tokens + tool calls live #167

Description

@pdettori

Summary

POST /turn runs a full agent turn but returns a single JSON blob after the
turn completes
— no incremental output. Any interactive / desktop-harness-style
UX (watching tokens arrive, watching the agent call tools) is impossible today.
This issue tracks adding a streaming response mode to /turn so callers can
observe assistant-text deltas and tool-call events live, while keeping the
existing one-shot JSON contract intact.

Current behavior

  • handleTurn reads the body, calls runTurn, and emits one JSON response
    { sessionId, response, ... }packages/knative-server/src/server.ts:66
    (route wired at :338, response written at :91).
  • runTurn runs await session.prompt(prompt) to completion and only then
    extracts the final assistant message text —
    harness/src/run-turn.ts:361 and following (function at :296).
  • Net: the client blocks for the whole turn and sees nothing until end_turn.

The agent itself is already a real tool-using agent in this path — runTurn
registers k8sSandboxExtension (harness/src/run-turn.ts:323), so tool calls
run in sandbox-0. Streaming would surface those tool calls as they happen.

Proposed approach

  • Content-negotiate on /turn: when the request sends
    Accept: text/event-stream, respond with SSE; otherwise behave exactly as
    today. (Keeps a single public noun; a separate POST /turn/stream route is an
    acceptable alternative if negotiation is awkward.)
  • Relay the Pi agent session's turn events as SSE frames:
    • assistant-text deltas,
    • tool-use start / result events,
    • a terminal frame carrying { sessionId, stopReason } so the client can
      resume on the same session.
  • Thread a callback / async-iterator out of runTurn (today it only returns the
    final TurnResult) so the HTTP layer can forward events without runTurn
    knowing about HTTP.
  • Back-compat is a hard requirement: the non-streaming JSON path must be
    byte-for-byte unchanged, and session persistence / resume must be identical.

Scope / acceptance criteria

  • Accept: text/event-stream (or /turn/stream) yields incremental SSE
    frames: text deltas + tool-call events + a terminal {sessionId,stopReason}.
  • Default /turn (no streaming Accept) is unchanged.
  • Session is persisted and resumable identically to the sync path.
  • Client disconnect aborts the in-flight turn (wire to the session's abort).
  • A curl -N example + a smoke assertion demonstrating deltas.

Open questions / spike

  • Exact Pi event surface: which events does the session from
    createAgentSession emit for text deltas and tool_use? A short spike against
    pi-fork should confirm before design lock.
  • SSE vs chunked JSON-lines — SSE is the interoperable default; confirm no proxy
    (Kourier/Knative activator) buffers it in a way that defeats streaming.
  • Knative timeoutSeconds still bounds a streamed turn (same as sync today) —
    document the ceiling; long-running work belongs in the async path instead.

Out of scope

  • Backgrounded / async free-form prompts — tracked separately (companion issue:
    async free-form prompt dispatch). Async is "fire and poll"; this issue is
    "watch live". The two are orthogonal.
  • Auth / credential changes.

References

  • packages/knative-server/src/server.ts:66 (handleTurn), :91, :338
  • harness/src/run-turn.ts:296 (runTurn), :323 (extensions), :361 (session.prompt)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions