Skip to content

fix(agent-sessions): ingest detection and attribute decoding - #1127

Merged
JeremyFunk merged 10 commits into
mainfrom
fix/agent-sessions-ingest-detection
Sep 29, 2026
Merged

JeremyFunk merged 10 commits into
mainfrom
fix/agent-sessions-ingest-detection

Conversation

@JeremyFunk

@JeremyFunk JeremyFunk commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Merge order: #1121 must merge before this PR. This PR reclassifies OpenInference spans as langchain, llamaindex and openai_agents_sdk. The detail page decodes those vendor ids with the OpenInference integration only once #1121 registers it under them. Merged alone, the OpenAI Agents TS spans lose their detail-page decoding (see #1).

The ingest gateway's AI detection (apps/ingest/src/ai_session.rs) missed several frameworks, and the OTLP value encoder (apps/ingest/src/telemetry.rs) made some attribute values unreadable. Every fix below is checked against the real OTLP captures wherever one exercises it. Replaying all 113 captures through stamp_trace_request, before and after, changes vendor stamps only in the captures listed here.

Capture Before (spans) After (spans)
docs_langchain_a / _b / _srv unknown:genai 25/37/12 + unknown:openinference 21/9/5 langchain 46/46/17
docs_microsoft-agent-framework_net unknown:genai 10 microsoft_agent_framework 10
docs_openai-agents_ts unknown:openinference 13 openai_agents_sdk 13
docs_strands_ts unknown:genai 8 strands 8
eve_slack / eve_slack_no_messages unknown:genai 30 / 23 vercel_ai_sdk (all)
vercel_ai_sdk_user / _legacy 0 of 28 / 18 spans with a session 17 / 17 with a session
openrouter 3 connection-test spans stamped unstamped

(docs_llamaindex_a/b already read llamaindex because the guide stamps a marker attribute; without the marker they take the same path as LangChain.)

#1 Unfingerprinted scopes (21cf6adcb, review fix 5f8fa1043)

  • Symptom: OpenInference LangChain / LlamaIndex, the .NET Agent Framework, Strands TS under a custom service.name and OpenInference's TS OpenAI Agents instrumentor all showed as "Unidentified" (table above).
  • Cause: scope_facts (ai_session.rs:319) did not know openinference.instrumentation.langchain / .llama_index (they only set the foreign-OpenInference flag, :347), Experimental.Microsoft.Agents.AI or @arizeai/openinference-instrumentation-openai-agents. The Strands TS SDK names its tracer and gen_ai.provider.name (or gen_ai.system) after service.name (@strands-agents/sdk telemetry/tracer.js _getCommonAttributes), so a custom service name hides the strands-agents marker detect_strands (:1183) keys on.
  • Fix:
    • The four scopes map to langchain, llamaindex, microsoft_agent_framework and openai_agents_sdk.
    • Strands is also detected when the scope and the provider both equal service.name and the span carries the Strands-only gen_ai.event.start_time.
    • langchain and llamaindex also read OpenInference's session.id and the gen_ai.conversation.id dual-write as session keys. The session ids in the captures are unchanged.
  • Decision, official OTel GenAI instrumentations (opentelemetry.instrumentation.genai.*): kept generic (unknown:genai).
    • They instrument a provider SDK, not a framework, and emit the spec dialect the default integration already decodes.
    • In the documented setup the app's own invoke_agent span carries the session, and that span decides the session's vendor, so a vendor id on the chat spans would not name the session anyway.
    • A new id would also need label, icon and color entries in the web app.
  • Test: instrumentor_scopes_name_their_framework, strands_typescript_is_detected_under_a_custom_service_name.
  • Depends on fix(agent-sessions): transcript and tool payload decoding #1121 for detail-page decoding: the OpenInference integration is registered for unknown:openinference today. fix(agent-sessions): transcript and tool payload decoding #1121 registers it under langchain, llamaindex and openai_agents_sdk. Until fix(agent-sessions): transcript and tool payload decoding #1121 lands, the reclassified spans decode through the default GenAI integration. The Python LangChain/LlamaIndex spans dual-write gen_ai.*, so they are unaffected. The OpenAI Agents TS spans (OpenInference keys only) would lose their detail-page decoding, hence the merge-order gate at the top.
  • Follow-up, not in this PR: other OpenInference scopes of known frameworks (haystack, google_adk, pydantic_ai, and the JS @arizeai/openinference-instrumentation-langchain) are not fingerprinted either. No capture exercises them, so they are left for a follow-up.

#2 Vercel AI SDK v7 spans without ai.* keys, and runtimeContext session ids (26ede77a5)

  • Symptom: in eve_slack, 30 of 122 AI SDK spans (chat / invoke_agent / execute_tool of the v7 tracer) landed in unknown:genai. vercel_ai_sdk_user sent ai.settings.context.sessionId, yet none of its 28 spans got a session (now 17 do). The guide therefore required usage: true, runtimeContext: true and an enrichSpan hook.
  • Cause: detect_vercel_ai_sdk (ai_session.rs:1208) required an ai.* key even inside the SDK's own scope. @ai-sdk/otel writes those keys only for opted-in supplemental attributes. The session keys (:956) did not include ai.settings.context.*, where the SDK writes runtime context (@ai-sdk/otel getRuntimeContextAttributes).
  • Fix:
    • Inside the ai / gen_ai scopes, a gen_ai.operation.name is enough.
    • ai.settings.context.sessionId and ai.settings.context.conversationId are read, ranked after gen_ai.conversation.id.
  • Test: vercel_v7_spans_without_ai_keys_and_runtime_context_sessions.
  • Not done, "sibling evidence": detection stays per span, and the SDK scope covers the same spans. A cross-span pass would need a second walk of the request on the hot path.

#4 Spring AI chat spans of non-OpenAI starters (4ef77a3bb)

  • Symptom: a Spring AI chat call without tools fell to unknown:genai unless gen_ai.system=openai.
  • Cause: detect_spring_ai (ai_session.rs:1205) accepted a bare chat span in the org.springframework.boot scope only for OpenAI. A model starter's chat observation carries gen_ai.* only, with its own provider as gen_ai.system. spring.ai.* keys appear only with tools (docs_spring-ai_a: the chat span without tools has none).
  • Fix: any gen_ai.operation.name in the Boot scope is Spring AI. The scope's HTTP spans have no operation name.
  • Test: spring_ai_chat_spans_of_any_model_starter.
  • Not replayed with a non-OpenAI starter: every capture calls through the OpenAI starter.

#6 agent_step read as Vercel AI SDK evidence (9c37e53e1)

  • Symptom: any span with gen_ai.operation.name=agent_step became vercel_ai_sdk. The LangChain and LlamaIndex guide processors had to use invoke_workflow for their step spans instead.
  • Cause: detect_vercel_ai_sdk accepted the op on its own (ai_session.rs:1210).
  • Fix: the clause is gone. The SDK's own step spans are still detected through its scope (Uiv2 #2).
  • Test: agent_step_outside_the_vercel_scope_is_not_vercel.

#16 OTLP bytes attributes stored as hex (d09f70b25, review fix f3e1be3d7)

  • Symptom: LangChain traced through LangSmith's OTel export (docs_langchain_ls, 50 spans) had gen_ai.prompt / gen_ai.completion stored as hex, so the content was unreadable.
  • Cause: any_value_string (telemetry.rs:4019) hex-encoded every bytesValue. LangSmith 0.14 sends those keys as UTF-8 JSON bytes.
  • Fix:
    • Bytes that are valid UTF-8 and contain no control characters other than tab, LF and CR are stored as text.
    • Anything else keeps the hex form. So binary that happens to be valid UTF-8 stays hex, and all-zero bytes stay "".
    • The local-mode encoder (apps/cli/src/server/otlp/encode.ts, a port of the Rust one) does the same. Like Rust, it keeps a leading BOM.
  • Test: bytes_attributes_decode_as_text_when_valid_utf8 (Rust), "decodes bytes attributes as text…" (encode.test.ts). Both cover the JSON text, invalid UTF-8, control-byte binary, all-zero and BOM cases.
  • Not resolved here:
    • The content is now stored as LangSmith's raw JSON wrappers ({"messages": …} / {"generations": …}).
    • The transcript renders those wrappers as raw JSON until the read side unwraps them (read-side decoding, a sibling batch).
    • So LangSmith turn labels and tool results are not fixed by this PR.
  • Applies to new rows only. Spans already stored keep their hex.

#18 Structured attribute values (8995b9352)

  • Evaluated:
    • Across all 113 captures, no span sends a nested array or map. The only structured span values are flat string arrays (gen_ai.response.finish_reasons, agno.tools, ...), and only ADK log bodies are maps.
    • Still, the GenAI semconv describes gen_ai.input.messages in structured form. Such a value was stored as an array of escaped JSON strings, which no reader decodes.
  • Cause: any_value_string (telemetry.rs:4020-4027) turned every element and map value into a string first, so each nesting level was re-encoded.
  • Fix: arrays and maps render through any_value_json, which keeps nested arrays and maps as JSON. Scalars keep their string form, so every flat array or map is stored exactly as before. The CLI port matches, up to map key order: Rust sorts keys, TS keeps insertion order. That difference already existed for flat maps.
  • Test: structured_attributes_stay_json_when_nested (Rust, including the unchanged flat cases), "keeps nested arrays and maps as JSON" (encode.test.ts).

#22 OpenRouter connection-test span stamped as AI (d08eaf7da)

  • Symptom: each "Test Connection" click in OpenRouter's Broadcast settings created a junk session (trace:e6d594d2…, vendor openrouter, 3 spans, 0 LLM calls).
  • Cause: detect_openrouter (ai_session.rs:1100) matched on the scope alone.
  • Fix: an OpenRouter span needs a gen_ai.operation.name. All 3,965 generation, attempt and moderation spans in the capture carry one; the connection-test span carries no attributes.
  • Test: openrouter_connection_test_span_is_not_ai.

Notes

  • Forward-only: all changes are ingest-side. Spans already stored keep their old stamps and values; no migration.
  • No contract change: no new vendor ids, so the web label/icon/color maps and ai-vendors.ts are untouched.
  • Verification:
    • cargo test --lib ai_session, cargo test --lib telemetry, bun test src/server/otlp/encode.test.ts (apps/cli).
    • The capture replay above: a throwaway example binary, not committed.
  • Overlap:
  • Review fix: 2d4ae4202 types the CLI port's anyValueJson as a named JSON type instead of unknown (effect-lint).
    • A sibling branch adds a span-event hook to stamp_trace_request. This PR does not touch that function.

Summary by CodeRabbit

  • Improvements
    • AI telemetry now recognizes spans from more agent and framework integrations, with improved session ID detection and handling of framework-specific operations.
    • OTLP byte attributes are displayed as text when they contain valid, readable UTF-8, and as hexadecimal otherwise. Nested arrays and maps retain their JSON structure instead of being flattened or escaped.

…pt scopes of known frameworks

Symptom: sessions from OpenInference's LangChain and LlamaIndex
instrumentors, the .NET Agent Framework, Strands TypeScript under a custom
service.name and OpenInference's TypeScript OpenAI Agents instrumentor were
listed as "Unidentified" (unknown:genai / unknown:openinference per span).

Cause: scope_facts matched none of these scopes. OpenInference's langchain
and llama_index scopes only set the generic foreign-OpenInference flag;
`Experimental.Microsoft.Agents.AI` and
`@arizeai/openinference-instrumentation-openai-agents` matched nothing; the
Strands TS SDK names both its tracer and `gen_ai.provider.name` (or
`gen_ai.system`) after the service, so a custom service.name hid its
"strands-agents" marker.

Fix: map the scopes to langchain, llamaindex, microsoft_agent_framework and
openai_agents_sdk; detect Strands when the scope and the provider both equal
service.name. langchain and llamaindex also read OpenInference's `session.id`
and the `gen_ai.conversation.id` dual-write as session keys.

Seen in: docs_langchain_a, docs_llamaindex_a,
docs_microsoft-agent-framework_net, docs_strands_ts, docs_openai-agents_ts.
…and read runtimeContext session ids

Symptom: with the AI SDK v7 OpenTelemetry integration, chat, invoke_agent
and execute_tool spans fell to unknown:genai unless the app enabled the
`usage` supplemental attributes, and an id passed as
`runtimeContext: { sessionId }` did not group the session, so the guide had
to require `usage: true`, `runtimeContext: true` and an `enrichSpan` hook.

Cause: detect_vercel_ai_sdk required an `ai.*` key even inside the SDK's own
scope; v7 writes those keys only for opted-in supplemental attributes. The
session keys did not include `ai.settings.context.*`, where the SDK writes
runtime context.

Fix: inside the `ai`/`gen_ai` scopes a `gen_ai.operation.name` is enough.
`ai.settings.context.sessionId` and `ai.settings.context.conversationId` are
read after `gen_ai.conversation.id`.

Seen in: docs_vercel-ai-sdk_a / _b (ai 7.0.54, @ai-sdk/otel tracer "gen_ai").
Symptom: Spring AI chat calls through a non-OpenAI starter (Anthropic,
Ollama, Bedrock, ...) fell to unknown:genai whenever the call had no tools,
so the session lost its framework.

Cause: detect_spring_ai only accepted a bare chat span in the
org.springframework.boot scope when `gen_ai.system=openai`. A model
starter's chat observation carries only `gen_ai.*`, with its own provider as
`gen_ai.system`; `spring.ai.*` keys appear only when tools are attached.

Fix: any `gen_ai.operation.name` in the Boot scope is Spring AI. The scope's
other spans (HTTP server/client) carry no operation name.

Seen in: docs_spring-ai_a / _b / _agent (chat spans without tools carry no
spring.ai.* key).
…I SDK evidence

Symptom: any span with `gen_ai.operation.name=agent_step` was stamped
vercel_ai_sdk, whatever emitted it. The LangChain and LlamaIndex guides
tried `agent_step` for their step spans and had to switch to
`invoke_workflow` because the spans turned into Vercel AI SDK spans.

Cause: detect_vercel_ai_sdk accepted `agent_step` on its own
(ai_session.rs:1210). The op is the AI SDK's name for its step span, not a
value only the SDK can write.

Fix: drop the clause. The SDK's own step spans are still detected through
its `gen_ai` scope (previous commit), and a step op from any other emitter
takes the generic path.

Seen in: docs_langchain_a / docs_llamaindex_a guide processors (step spans).
…e valid UTF-8

Symptom: LangChain traced through LangSmith's OTel export showed hex blobs
as its transcript: `gen_ai.prompt` / `gen_ai.completion` arrived as hex
strings on every span, so the transcript was unreadable, turns had no label
and tool results read "not captured".

Cause: any_value_string (apps/ingest/src/telemetry.rs:4019) hex-encoded
every OTLP bytesValue. LangSmith 0.14 sends those two keys as bytes holding
UTF-8 JSON.

Fix: a bytes value that is valid UTF-8 is stored as that text; anything else
keeps the hex form. The local-mode encoder in apps/cli (a port of the Rust
encoder) does the same.

Seen in: docs_langchain_ls (50 spans, both keys bytesValue).
… of escaped strings

Symptom: a structured attribute value, such as `gen_ai.input.messages` sent
as an OTLP array of maps (the form the GenAI semconv describes), was stored
as an array of escaped JSON strings, so no reader could decode the messages.
Guides had to tell emitters to send JSON strings.

Cause: any_value_string (apps/ingest/src/telemetry.rs:4020-4027) turned
every array element and map value into a string first, so each nesting level
was re-encoded as a string inside the outer JSON.

Fix: arrays and maps render through any_value_json, which keeps nested
arrays and maps as JSON. Scalars keep their string form, so every flat array
or map (the only shapes in the captures: `gen_ai.response.finish_reasons`,
`agno.tools`, ...) is stored byte for byte as before. The local-mode encoder
in apps/cli does the same.

Seen in: none of the 113 captures sends a nested value on a span (checked
every array, map and bytes attribute); the fix is for SDKs that emit the
structured form.
Symptom: every "Test Connection" click in OpenRouter's Broadcast settings
created a junk agent session (trace:e6d594d2..., vendor openrouter, 3 spans,
0 LLM calls).

Cause: detect_openrouter matched on the scope alone, and the scope forces
predicate evaluation, so the dashboard's attribute-less
`openrouter-connection-test` span was stamped like a generation.

Fix: an OpenRouter span needs a `gen_ai.operation.name`. Every generation,
provider attempt and moderation span in the capture carries one; the
connection-test span carries no attributes at all.

Seen in: capture `openrouter` (3 connection-test spans on trace id 0...01).
@maple-review-bot

maple-review-bot Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Maple review

Confidence 4/5 · likely safe to merge
Detection widenings and the new decoding rules all have tests and the capture replay; the only defect left is the JS port's map key order.
quality 98/100 · 1 note · tests covered · risk medium

Extends AI-session detection (OpenInference/.NET/TS scopes, Strands under a custom service name, Vercel v7 and Spring AI from an operation name, bare OpenRouter spans) and makes bytes and nested structured attributes decode as text/JSON instead of hex and escaped strings. The Rust and TS behaviour is tested for both, and the change is otherwise safe to merge.

  • scope_facts maps openinference.instrumentation.langchain/.llama_index, Experimental.Microsoft.Agents.AI and the @arizeai/ openai-agents scope to vendors
  • detect_strands also fires when the scope and gen_ai.system/provider.name equal service.name
  • detect_vercel_ai_sdk/detect_spring_ai accept any gen_ai.operation.name in their scope; detect_openrouter now requires one
  • any_value_string decodes UTF-8 bytes as text and keeps nested arrays/maps as JSON

Findings

Note · F1 · anyValueJson emits map keys in insertion order, Rust sorts them

correctness · apps/cli/src/server/otlp/encode.ts:262-264

The new function is documented as a port of Rust any_value_json (encode.ts:253-256, "exactly as the Rust encoder does" at :223), but Object.fromEntries keeps the sender's key order while Rust's serde_json::Map is a BTreeMap, so serde_json::to_string sorts keys — the Rust test at telemetry.rs:4511 expects [{"parts":[{"content":"hi","type":"text"}],"role":"user"}] for the same payload the new JS test at encode.test.ts:277 expects as [{"role":"user","parts":[{"type":"text"}]}]. A structured attribute such as gen_ai.input.messages therefore renders in a different key order in local mode than on the ingest path, and integer-like keys are reordered a third way by JS object ordering.

Sort each map's entries by key before serializing (serde_json's BTreeMap order), or drop the parity claim from the doc comment and the test; note that `JSON.stringify` hoists integer-like keys, so sorting the entries alone does not reproduce Rust's byte order.
What was checked
  • Rust bytes-as-text and nested-JSON arms match the old scalars-as-strings output for flat arrays and maps (any_value_json, telemetry.rs:4036)
  • The new strands clause cannot fire on empty values: matches_service_name is only set when the scope name is non-empty (ai_session.rs:308)
  • The broadened Vercel/Spring rules stay behind their own scope flags and are ordered after every framework vendor (ai_session.rs:1025), and no TS copy of this detection exists to keep in sync
Copy all findings (1)
Findings from an automated review of commit d08eaf7da94fb3f0d663a91eda275f7c0cd29748. Verify each one against the current code before changing anything, fix only those that still apply, and keep each fix to the lines it names.

---

F1 · Note · correctness · apps/cli/src/server/otlp/encode.ts:262-264
`anyValueJson` emits map keys in insertion order, Rust sorts them
The new function is documented as a port of Rust `any_value_json` (encode.ts:253-256, "exactly as the Rust encoder does" at :223), but `Object.fromEntries` keeps the sender's key order while Rust's `serde_json::Map` is a `BTreeMap`, so `serde_json::to_string` sorts keys — the Rust test at telemetry.rs:4511 expects `[{"parts":[{"content":"hi","type":"text"}],"role":"user"}]` for the same payload the new JS test at encode.test.ts:277 expects as `[{"role":"user","parts":[{"type":"text"}]}]`. A structured attribute such as `gen_ai.input.messages` therefore renders in a different key order in local mode than on the ingest path, and integer-like keys are reordered a third way by JS object ordering.
Suggested fix: Sort each map's entries by key before serializing (serde_json's BTreeMap order), or drop the parity claim from the doc comment and the test; note that `JSON.stringify` hoists integer-like keys, so sorting the entries alone does not reproduce Rust's byte order.

d08eaf7 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple to ask about one.

@coderabbitai

coderabbitai Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 373ef8d1-7d38-478d-862c-385fa5d1a41c

📥 Commits

Reviewing files that changed from the base of the PR and between f8b99d8 and 5f8fa10.

📒 Files selected for processing (4)
  • apps/cli/src/server/otlp/encode.test.ts
  • apps/cli/src/server/otlp/encode.ts
  • apps/ingest/src/ai_session.rs
  • apps/ingest/src/telemetry.rs

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The CLI and ingest OTLP converters now decode qualifying UTF-8 bytes and preserve nested arrays and maps as JSON. AI span detection now recognizes additional instrumentation scopes, applies updated provider classification rules, and checks additional session ID attributes.

Changes

AnyValue Formatting

Layer / File(s) Summary
Recursive AnyValue conversion
apps/cli/src/server/otlp/encode.ts, apps/cli/src/server/otlp/encode.test.ts, apps/ingest/src/telemetry.rs
The CLI and ingest converters return readable text for qualifying UTF-8 bytes and hex for invalid or disallowed control-containing bytes. They preserve nested arrays and maps as JSON, with scalar values represented as strings. Tests cover byte conversion and nested values.

AI Span Detection and Sessions

Layer / File(s) Summary
Framework scopes and session identifiers
apps/ingest/src/ai_session.rs
Scope matching includes additional LangChain, LlamaIndex, Microsoft Agent Framework, and OpenAI Agents instrumentation. LangChain, LlamaIndex, and Vercel AI session lookup checks additional identifiers.
Provider classification rules
apps/ingest/src/ai_session.rs
OpenRouter, Strands, Spring AI, and Vercel AI classification predicates have changed. Tests cover updated classification rules and framework-specific spans.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Suggested reviewers: makisuo

Merge Risk: ⚪ Minimal · up to 5f8fa

The changed telemetry formatting and AI span attribution have no identified issue that blocks merging after normal checks.

Security Architecture Review

Security architecture risk: 🟡 Moderate · up to 5f8fa

Newly recognized agent spans may lose framework-specific detail decoding if the corresponding reader support is not deployed first. Ingestion remains tied to the authenticated organization, and this review did not establish a cross-tenant access path.

Retained concerns

  • Medium · reliability · inferred: If the new ingest vendor stamps are deployed before their reader registrations, affected spans persist under framework IDs but receive generic rather than framework-specific detail decoding. The PR identifies this prerequisite; its deployment state was not verified.
Security review details

Security Blast Radius

  • inferred — A sender able to submit telemetry can influence vendor attribution and session grouping within its authenticated organization. The inspected ingest path does not show those attributes redirecting storage to another organization.

Trust Boundaries and Controls

  • observed — Scope matching is exact for the newly named instrumentors, and re-ingest strips gateway-owned AI stamps before re-derivation. These controls do not authenticate the claimed instrumentation provider; organization routing is separately keyed to authenticated ingest identity.

Resilience and Maintainability Implications

  • observed — Session extraction is span-local, with no shared session cache in the stamping function. The changed CLI and ingest value converters recurse through nested values; effective input-depth and node bounds for those paths were not established.

Hardening Proposals

  • proposed — Verify that newly stamped vendor IDs have reader registrations before ingest rollout, and document whether session IDs are grouping labels or identity-bearing values for downstream authorization.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 69.44% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 36 functions across 4 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the two main changes: agent-session detection and attribute decoding. It is concise and specific.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

Devin Review

Comment on lines +340 to +341
"openinference.instrumentation.openai_agents"
| "@arizeai/openinference-instrumentation-openai-agents" => facts.openai_agents = true,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 OpenAI Agents session details disappear

When openai_agents_sdk stamps TypeScript OpenInference spans, resolveAiIntegration falls back to GenAI. OpenInference-only messages and operation kinds disappear from session details.

Learn more

The gateway stamps a vendor on each span, and the session reader uses that vendor to choose attribute decoders. The new scope marks TypeScript OpenAI Agents spans as openai_agents_sdk. The integration registry only registers the OpenInference decoder for unknown:openinference and openinference-openai, so resolveAiIntegration chooses the generic decoder instead. OpenInference-only fields then do not become GenAI fields even though they remain in storage.

Example: A TypeScript Agents span has openinference.span.kind=CHAIN and input.value containing a prompt. It previously carried unknown:openinference, which decoded that prompt; now it carries openai_agents_sdk, whose default integration does not read input.value.

Recommended fix: Register the OpenInference integration for the newly classified OpenInference vendors in AI_VENDOR_INTEGRATIONS, including openai_agents_sdk, langchain, and llamaindex where their spans use that dialect. Update projection/decoder tests for these stamps and ship this with the classification change.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Known and gated rather than fixed here: #1121 registers the OpenInference integration under openai_agents_sdk, langchain and llamaindex, and this PR is gated to merge after it (merge-order note at the top of the PR body). The registry change lives in #1121 to avoid two branches editing AI_VENDOR_INTEGRATIONS.

Comment thread apps/ingest/src/ai_session.rs Outdated
Comment on lines +1208 to +1210
|| (c.scope.matches_service_name
&& (c.ev.gen_ai_provider_name == c.resource.service_name
|| c.ev.gen_ai_system == c.resource.service_name))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Generic AI spans mislabeled as Strands

When a service-named scope reports its service as provider, detect_strands labels its spans Strands without SDK evidence. Generic AI spans then appear under the wrong vendor.

Learn more

A scope identifies the component that creates spans, while service.name identifies the process. The new branch accepts equality between those names and a GenAI provider or system value as sufficient evidence for Strands. Neither value identifies the SDK: an application may name its own tracer and GenAI provider after the service. The ordered vendor predicates then classify that span as Strands before the generic tier can handle it.

Example: A custom app emits a chat span under scope my-agent, sets resource service.name=my-agent, and sets gen_ai.provider.name=my-agent. The span gets vendor strands, though no Strands SDK is present; unknown:genai was the generic classification.

Recommended fix: Require an additional Strands-specific marker for the custom-service fallback, or establish a scope/attribute signature that distinguishes Strands TS from other service-named tracers; test the same three matching strings from a non-Strands emitter.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5f8fa10: the custom-service rule now also requires gen_ai.event.start_time, a non-semconv key the Strands TS tracer writes on every span. A test covers an app that names its tracer and provider after its service without that key (stays unknown:genai).

… not unknown

Review fix for #18: effect-lint rejects a function returning `unknown`.
anyValueJson now returns the named AttrJson type (string, array or map of
the same).
@maple-review-bot

maple-review-bot Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Maple review

Confidence 5/5 · safe to merge
The TS port mirrors the Rust arms closely; the decoders are exercised by new Vitest cases, so only the BOM divergence is open.
quality 96/100 · 2 notes · tests covered · risk low

The local-mode OTLP encoder now decodes bytesValue as UTF-8 text when valid and keeps nested arrays/maps as JSON, matching the Rust gateway changes in this PR. One small divergence from the Rust port remains.

  • bytesText returns UTF-8 text for valid-UTF-8 bytes attributes, hex otherwise

Findings

Note · F2 · UTF8 decoder strips a leading BOM that Rust's str::from_utf8 keeps

correctness · apps/cli/src/server/otlp/encode.ts:66

TextDecoder with the default ignoreBOM: false removes a leading U+FEFF from the decoded bytes (verified: EF BB BF 41 decodes to "A", not "\uFEFFA"), while the Rust arm this ports, std::str::from_utf8 in telemetry.rs:4022, keeps it. A bytesValue carrying a BOM-prefixed payload loses its first character in local mode but not through the ingest gateway, so the same span decodes differently in the two paths.

const UTF8 = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true })

Still open from earlier reviews

What was checked
  • Empty and all-zero bytes take the same path in both arms (encode.ts:72, telemetry.rs:4022)
  • Reusing the module-level UTF8 decoder is stateless without stream: true
  • Map key order still differs from spent serde_json; that is open finding F1, not refiled
Copy all findings (1)
Findings from an automated review of commit 2d4ae4202efbd32d05555732ae416aee5617abfa. Verify each one against the current code before changing anything, fix only those that still apply, and keep each fix to the lines it names.

---

F2 · Note · correctness · apps/cli/src/server/otlp/encode.ts:66
`UTF8` decoder strips a leading BOM that Rust's `str::from_utf8` keeps
`TextDecoder` with the default `ignoreBOM: false` removes a leading U+FEFF from the decoded bytes (verified: `EF BB BF 41` decodes to `"A"`, not `"\uFEFFA"`), while the Rust arm this ports, `std::str::from_utf8` in `telemetry.rs:4022`, keeps it. A `bytesValue` carrying a BOM-prefixed payload loses its first character in local mode but not through the ingest gateway, so the same span decodes differently in the two paths.
Replace those lines with:
const UTF8 = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true })

2d4ae42 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple to ask about one.

…ey are valid UTF-8

Review fix for #16. Binary that happens to be valid UTF-8 was stored as
control-character text: all-zero bytes became NULs instead of "", and
[0x01, 0x02, 0x7f] became "\u0001\u0002\u007f" instead of "01027f".

The text path now also requires no control characters other than tab, LF
and CR, on both the Rust encoder and its apps/cli port. The port's
TextDecoder also keeps a leading BOM (`ignoreBOM`), as Rust's from_utf8 does.
@maple-review-bot

maple-review-bot Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Maple review

Confidence 4/5 · likely safe to merge
Bytes and nested-structured attribute decoding changed in both the ingest gateway and the local CLI port, each covered by new unit tests on both sides.
quality 98/100 · 1 note · tests covered · risk medium

Adds bytesText/anyValueJson to the local CLI OTLP encoder as ports of the new Rust any_value_string arms, so valid-UTF-8 bytes attributes decode as text and nested arrays/maps stay JSON. Both ports are tested and match, except the map-key ordering already reported.

  • any_value_string decodes a BytesValue as text unless it holds a non-whitespace control character (telemetry.rs:4022)
  • any_value_json keeps nested arrays and maps as JSON instead of escaped strings (telemetry.rs:4044)
  • CLI bytesText and anyValueJson mirror those arms with TextDecoder(ignoreBOM: true) (encode.ts:75, encode.ts:265)

Still open from earlier reviews

Fixed since the last review

  • F2 · UTF8 decoder strips a leading BOM that Rust's str::from_utf8 keeps
What was checked
  • ignoreBOM: true keeps the leading U+FEFF while fatal: true throws on invalid UTF-8, matching str::from_utf8, so F2 is fixed (encode.ts:67)
  • BINARY_CONTROL is C0 minus tab/LF/CR plus DEL and C1, the same set char::is_control yields, and U+FEFF is category Cf so both ports keep it (encode.ts:69)
  • Rust Value::Object collects into a sorted map (no preserve_order in apps/ingest/Cargo.toml:44), confirming the key-order divergence rather than a wrong Rust side

f3e1be3 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple to ask about one.

…m-service Strands TS rule

Review fix for #1. The custom-service fallback accepted any span whose
scope and gen_ai provider (or system) both equal service.name, so an app
that names its own tracer and provider after its service would be read as
Strands.

The rule now also requires `gen_ai.event.start_time`, a non-semconv key the
Strands TS tracer writes on every span (docs_strands_ts: invoke_agent, chat,
execute_tool).
@maple-review-bot

maple-review-bot Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Maple review

Confidence 4/5 · likely safe to merge
The one place to watch is the merge-order gate: without #1121 the reclassified OpenInference OpenAI-Agents spans decode through the default GenAI integration.
quality 98/100 · 1 note · tests covered · risk medium

The head commit narrows the custom-service Strands rule in apps/ingest/src/ai_session.rs to spans carrying gen_ai.event.start_time, closing the false-positive the bot raised on service-named scopes. The ai_session.rs delta reads correctly and is covered by tests; safe to merge, gated on #1121 per the PR body.

  • detect_strands needs gen_ai.event.start_time for the service-named TS rule
  • gen_ai_event_start_time evidence bit set from gen_ai.event.start_time
  • Two tests cover the positive and the negative Strands cases

Still open from earlier reviews

What was checked
  • Both new scope names are in SCOPE_NAMES (ai_session.rs:291), so the two-byte scope screen admits them
  • matches_service_name cannot fire on an empty service.name; scope_facts returns early for an empty scope (ai_session.rs:308)
  • No new entrypoint, outbound call, log or metric in the delta; observability coverage is empty

5f8fa10 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple to ask about one.

@JeremyFunk
JeremyFunk merged commit 754e173 into main Sep 29, 2026
40 checks passed
@JeremyFunk
JeremyFunk deleted the fix/agent-sessions-ingest-detection branch September 29, 2026 11:47
JeremyFunk added a commit that referenced this pull request Sep 29, 2026
…blind runs

Blind runs of 11 frameworks (a fresh agent applying only the skill to an
open-source example) surfaced stale Maple limitations and missing setup
guidance.

- Remove statements fixed by #1120, #1121, #1122 and #1127 (2x token totals,
  Unidentified vendors, dropped plain-text tool payloads, OpenInference
  transcripts, check-headline caveats) from the skills and docs pages.
- Add to every skill: wrong-region 401 hint, load .env before the exporter,
  fail fast on a missing key, verification without Maple access, a driver for
  apps without a scriptable entry point, and a non-crashing TS shutdown.
- Apply the verified framework-specific fixes for vercel-ai-sdk,
  cloudflare-agents, mastra, langchain, openai-agents, google-adk,
  claude-agent-sdk and pydantic-ai.
JeremyFunk added a commit that referenced this pull request Sep 30, 2026
…1115)

* docs(agent-tracing): per-framework agent tracing guides and skills (WIP)

* docs(agent-tracing): apply verifier fixes from end-to-end runs of every guide

* docs(agent-tracing): editorial pass, cross-links from instrumentation and onboarding

* docs(agent-tracing): align guides with what Agent Sessions shows for each framework

* docs(agent-tracing): cut the human guides to a 5-minute setup, move detail into the skills

* docs(agent-tracing): trim the Vercel AI SDK setup to three packages, keep the span processor variant for serverless

* docs(agent-tracing): drop package trivia from the Vercel AI SDK install step

* docs(agent-tracing): say when to use the OpenTelemetry guide instead of listing languages

* docs(agent-tracing): cut claims a reader setting up tracing doesn't need

* docs(agent-tracing): one quick-setup wording across guides, with the EU region hint

* feat(docs): render install commands as npm/pnpm/bun and pip/uv tabs

* docs(agent-tracing): TypeScript for LangChain.js, OpenAI Agents, ADK, Cloudflare Agents and Genkit; filter guides by language

* docs(agent-tracing): list guides per language on the overview without a selector

* docs(agent-tracing): list the any-language guide once, under other languages and frameworks

* docs(agent-tracing): say what each Cloudflare Agents package is for

* skills(agent-tracing): tell agents how to send redacted feedback on a skill

* skills(agent-tracing): send feedback through the MCP only

* skills(agent-tracing): drop the feedback section for now

* skills(agent-tracing): drop fixed Maple gaps, add setup gotchas from blind runs

Blind runs of 11 frameworks (a fresh agent applying only the skill to an
open-source example) surfaced stale Maple limitations and missing setup
guidance.

- Remove statements fixed by #1120, #1121, #1122 and #1127 (2x token totals,
  Unidentified vendors, dropped plain-text tool payloads, OpenInference
  transcripts, check-headline caveats) from the skills and docs pages.
- Add to every skill: wrong-region 401 hint, load .env before the exporter,
  fail fast on a missing key, verification without Maple access, a driver for
  apps without a scriptable entry point, and a non-crashing TS shutdown.
- Apply the verified framework-specific fixes for vercel-ai-sdk,
  cloudflare-agents, mastra, langchain, openai-agents, google-adk,
  claude-agent-sdk and pydantic-ai.

* docs(agent-tracing): drop stale payload rules, build the Claude SDK env per call

- opentelemetry: tool results may be plain strings; Maple no longer drops
  plain-text tool payloads.
- claude-agent-sdk: build the telemetry env per query() and fail fast on a
  missing key, matching the skill.

* docs(agent-tracing): drop setup steps Maple no longer needs

- GenAI semconv flag is recommended, not required, for LangChain (Python), LlamaIndex and smolagents; openai-agents keeps it for agent lanes and finish reasons
- smolagents: stop zeroing run-span token usage
- Vercel AI SDK / Cloudflare Agents: runtimeContext groups sessions, drop enrichSpan
- Genkit: pass string tool results through unwrapped
- LangChain.js: correct the GenAiSpans rationale
- Strands TS: note zero tokens with api: "chat" behind OpenAI-compatible gateways

* skills(agent-tracing): pass LangChain.js tool results through as plain text

* docs(agent-tracing): drop workarounds and caveats the agent-session fixes made stale

- Session keys: every vendor now falls back to gen_ai.conversation.id; drop the
  "Maple ignores gen_ai.conversation.id" lines (agno, crewai, dspy, smolagents,
  spring-ai, strands) and the OpenInference Haystack session.id caveat.
- Tokens: usage counts only on the model-call span, so drop Strands'
  gen_ai_use_latest_invocation_tokens, the per-request TS agent rationale and
  the 1.54 floor, pydantic-ai's aggregated-usage warning and the Anthropic cache
  double-count caveats. Hand-written spans send semconv totals (input includes
  cache, output includes reasoning); the Anthropic/Gemini mappings and the ADK
  TS processor follow that.
- Cost: LiteLLM's litellm.cost.total and Pydantic AI's operation.cost are read;
  drop the LiteLLM turn-cost recipe.
- Detection: LangChain.js, Genkit and .NET Semantic Kernel get their framework
  label; OpenAI Agents TS gets agent lanes; LangChain.js groups by session.id
  without GenAiSpans' conversation-id copy.
- Classification: drop DSPy's adapter marker, Spring AI's advisor rename,
  LangChain's ChatPromptTemplate step, the MAF workflow.build instruction,
  Mastra scorer and LangChain turn-label caveats, and MapleSpanFixes' tool
  argument fix (the tool-errors view decodes arguments like the session page).

* skills(agent-tracing): leave ADK TS output tokens as reported; Maple adds thinking

* skills(agent-tracing): trim to what an implementing agent needs (#1174)

- cut human-guide links, backend background, tested-version notes, restated code
- drop the Go reference; other languages follow the generic steps
- Do-not lists keep only silent, non-obvious mistakes not stated in the steps
- inline the GenkitForMaple processor instead of pointing at the guide
- OTLP header: quoted literal space everywhere (every targeted SDK accepts it)
- add Cloudflare Agents and Genkit to the OpenTelemetry skill's framework list
- smolagents: enable_genai_semconv is required

* docs(agent-tracing): ADK header uses a quoted literal space, not %20

* docs(agent-tracing): keep tool error text out of Haystack spans with content off, note Spring AI's

* docs(agent-tracing): export ADK env vars, name MAPLE_INGEST_KEY, drop the removed Go reference

* skills(agent-tracing): tolerate malformed tool arguments, route provider SDKs through the router, raw skill URL

* docs(agent-tracing): tighten the OpenTelemetry guide's intro, content and check wording

* docs(agent-tracing): use the private ingest key from the environment, never inline

Agent tracing runs server-side, so guides and skills now use the private key (maple_sk_) as MAPLE_INGEST_KEY in the repo's secret/env convention. The user sets it themselves instead of pasting it into the prompt; skills create a gitignored .env and .env.example when the repo has none.

* docs(onboard): servers use the private ingest key from MAPLE_INGEST_KEY, browsers the public key

maple-onboard and the language style skills now read the private key (maple_sk_) from MAPLE_INGEST_KEY on servers, with the agent-tracing secret rules: repo secret/env convention, gitignored .env plus .env.example when there is none, fail fast when unset, never in source or asked for in chat. Browser and mobile code keep the inline public key. Landing docs and the agent-tracing overview prompt follow.

* docs(agent-tracing): warn and disable export when MAPLE_INGEST_KEY is unset

Instrumentation must never crash or block the app. Replace every throw,
exit, panic and ${VAR:?} on a missing key with one warning plus a skipped
Maple exporter, and never send an empty bearer.

* skills(agent-tracing): read the Spring key from Boot's environment so .env works

* Revert "skills(agent-tracing): read the Spring key from Boot's environment so .env works"

This reverts commit dc45d34.

* Revert "docs(agent-tracing): warn and disable export when MAPLE_INGEST_KEY is unset"

This reverts commit 307aa7a.

* Revert "docs(onboard): servers use the private ingest key from MAPLE_INGEST_KEY, browsers the public key"

This reverts commit 814e177.

* Revert "docs(agent-tracing): use the private ingest key from the environment, never inline"

This reverts commit 821d778.

* docs(agent-tracing): warn and disable export when the ingest key is unset or setup fails

Instrumentation must never crash or block the app. Code that reads
MAPLE_INGEST_KEY logs one warning and skips the Maple exporter instead of
throwing, exiting or panicking, and Go/Rust setup errors are logged, not
fatal.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant