Skip to content

feat(rivetkit): forward otel sdk warnings to the pino logger - #5733

Merged
NathanFlurry merged 9 commits into
stack/feat-rivetkit-trace-actor-to-actor-calls-zosqvyzxfrom
stack/feat-rivetkit-forward-otel-sdk-warnings-to-the-pino-logger-vmosovzz
Sep 23, 2026
Merged

NathanFlurry merged 9 commits into
stack/feat-rivetkit-trace-actor-to-actor-calls-zosqvyzxfrom
stack/feat-rivetkit-forward-otel-sdk-warnings-to-the-pino-logger-vmosovzz

Conversation

@eersnington

@eersnington eersnington commented Sep 15, 2026 •

Copy link
Copy Markdown
Member

sends otel sdk warnings and errors from core to the actor's pino logger. also handles for warns and errors. also is safe as it will release the logging callback on shutdown so it doesn't keep node running

@the-company-company the-company-company Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 1 medium-severity finding

Reviewed commit a550f80.

Comment thread rivetkit-typescript/packages/rivetkit-napi/src/telemetry.rs
@eersnington
eersnington force-pushed the stack/feat-rivetkit-forward-otel-sdk-warnings-to-the-pino-logger-vmosovzz branch from a550f80 to e34c47e Compare September 16, 2026 01:18
@eersnington
eersnington force-pushed the stack/feat-rivetkit-trace-actor-to-actor-calls-zosqvyzx branch from bc77372 to 384b32b Compare September 16, 2026 01:18

@the-company-company the-company-company Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues found

Reviewed commit e34c47e.

@claude

claude Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Review: forward OTel SDK warnings + workflow run/step tracing

Large but well-organized change (Rust core telemetry, NAPI bridge, TS runtime/workflow-engine, docs). The design follows the existing invocation/span patterns in telemetry.rs closely (link-not-parent for new traces, Drop-as-abandoned semantics, versioned BARE storage for the persisted run context), and test coverage for the new workflow spans is thorough (chaining across sleeps, ray propagation, retried-step attempts, nested SQL/queue spans).

A couple of things worth a look before merge:

1. finally { await span.finish(outcome) } can mask the real workflow result/error

rivetkit-typescript/packages/workflow-engine/src/index.ts (traceRun, ~line 923-944):

```ts
try {
const result = await span.run(() => run(span));
outcome = runOutcomeFromState(result.state);
return result;
} catch (error) {
if (error instanceof EvictedError) outcome = "cancelled";
throw error;
} finally {
await span.finish(outcome);
}
```

If span.finish(outcome) throws (NAPI bridge exception, native panic bridged as a JS error, etc.), that exception replaces whatever the try/catch was about to return or throw, including a legitimate workflow failure or a successful result. The rest of the codebase treats telemetry as strictly best-effort (WorkflowRunInvocation::finish/finish_workflow_span in Rust already swallow persistence errors with tracing::warn!, and the docs state "Export ... never fails actor work"), so this call site is the odd one out. In practice WorkflowRunOutcome::parse only fails on a bad string, which shouldn't happen given the TS union type, so the risk is narrow, but a finally that awaits something fallible is worth guarding (e.g. wrap span.finish in try/catch + log) so a telemetry-layer hiccup can never override the actual workflow outcome.

2. Process-wide SDK log sink vs. per-registry lifecycle

rivetkit-typescript/packages/rivetkit-napi/src/telemetry.rs keeps the sink in a single global static SINK, and rivetkit-typescript/packages/rivetkit/src/registry/napi-runtime.ts calls setTelemetryLogSink(...) on every createRegistry() while shutdownTelemetry() is invoked per-runtime on registry shutdown (registry/index.ts: [...runtimes].map(r => r.shutdownTelemetry?.())). Since the sink is process-wide by design (per the comment in telemetry.rs), if a single Node process ever hosts more than one registry concurrently:

  • the later createRegistry() call's sink silently replaces the earlier one, so SDK warnings from the first registry's actors get attributed to the second registry's logger, and
  • stopping any one registry uninstalls the sink for all the others still running, which silently falls back to raw Rust log formatting for their SDK warnings.

This is diagnostics-only (doesn't affect actor behavior) and may just be an accepted limitation of "one telemetry pipeline per process," but it's worth confirming that's the intended deployment model, or noting the caveat in docs-internal/engine/rivetkit-telemetry.md.

Everything else looked solid

  • telemetry.rs: ray-id handling correctly moved from an immutable inner.ray_id to a lock-guarded state.ray_id everywhere (no stale direct reads remain), and start_queue_receive's workflow-takes-message-ray logic matches the documented "a workflow run takes the ray of each queue message it receives" behavior.
  • WorkflowTraceContextV1/key reservation (rivetkit-core::keys::WORKFLOW_TRACE_CONTEXT_KEY = tuple-encoded 5, matching the // 5 is reserved for RivetKit's workflow trace context comment in workflow-engine/src/keys.ts) is a correctly cross-referenced, non-colliding reservation in the shared _rivet_wf_kv table.
  • New versioned struct follows the existing RunWakeAt/LastPushedAlarm pattern exactly (manual match on version, empty converters vec), compliant with the "never round-trip serialize/deserialize for version converters" rule.
  • runOutcomeFromState/WorkflowRunOutcome/WorkflowStepOutcome matches are exhaustive (no _ fallthrough on enums), per CLAUDE.md.
  • The "EvictedError only on cancellation" comment in traceRun checks out: executeWorkflow's only unhandled EvictedError rethrow is the storage.state === "cancelled" check; the graceful evict() path is fully absorbed into setEvictedState and returned as a normal result, so it never hits that catch.
  • New sql_efficiency.rs case for _rivet_wf_kv lookups follows the required test-coverage policy for new internal SQL queries.
  • vi.waitFor calls in actor-telemetry.test.ts all carry the required adjacent justification comment.

No security concerns; the data policy section explicitly excludes SQL text/bindings/state/raw error messages from spans, and the new attributes (rivet.workflow.run.outcome, rivet.workflow.step.*) are all bounded enum-like values, consistent with the metrics cardinality rules.

@eersnington
eersnington force-pushed the stack/feat-rivetkit-forward-otel-sdk-warnings-to-the-pino-logger-vmosovzz branch from e34c47e to 0f3841f Compare September 16, 2026 17:51
@eersnington
eersnington force-pushed the stack/feat-rivetkit-trace-actor-to-actor-calls-zosqvyzx branch from 384b32b to 7f1d947 Compare September 16, 2026 17:51
@eersnington
eersnington force-pushed the stack/feat-rivetkit-forward-otel-sdk-warnings-to-the-pino-logger-vmosovzz branch from 0f3841f to c2a3397 Compare September 16, 2026 18:08
@eersnington
eersnington force-pushed the stack/feat-rivetkit-trace-actor-to-actor-calls-zosqvyzx branch 2 times, most recently from 41a28e0 to 945ecce Compare September 16, 2026 18:16
@eersnington
eersnington force-pushed the stack/feat-rivetkit-forward-otel-sdk-warnings-to-the-pino-logger-vmosovzz branch from c2a3397 to e10de14 Compare September 16, 2026 18:16
@eersnington
eersnington force-pushed the stack/feat-rivetkit-trace-actor-to-actor-calls-zosqvyzx branch from 945ecce to 8395b1b Compare September 16, 2026 18:24
@eersnington
eersnington force-pushed the stack/feat-rivetkit-forward-otel-sdk-warnings-to-the-pino-logger-vmosovzz branch from e10de14 to 5fd3076 Compare September 16, 2026 18:24
@eersnington
eersnington force-pushed the stack/feat-rivetkit-trace-actor-to-actor-calls-zosqvyzx branch from 8395b1b to d485500 Compare September 16, 2026 18:34
@eersnington
eersnington force-pushed the stack/feat-rivetkit-forward-otel-sdk-warnings-to-the-pino-logger-vmosovzz branch from 5fd3076 to 2c70404 Compare September 16, 2026 18:34
@eersnington
eersnington force-pushed the stack/feat-rivetkit-trace-actor-to-actor-calls-zosqvyzx branch from d485500 to f9dcaf3 Compare September 16, 2026 18:42
@eersnington
eersnington force-pushed the stack/feat-rivetkit-forward-otel-sdk-warnings-to-the-pino-logger-vmosovzz branch from 2c70404 to a78f5c5 Compare September 16, 2026 18:42
@eersnington
eersnington force-pushed the stack/feat-rivetkit-trace-actor-to-actor-calls-zosqvyzx branch from f9dcaf3 to ae9c32e Compare September 16, 2026 19:13
@eersnington
eersnington force-pushed the stack/feat-rivetkit-forward-otel-sdk-warnings-to-the-pino-logger-vmosovzz branch 2 times, most recently from 8443f5d to 39bc2f6 Compare September 16, 2026 20:28
@eersnington
eersnington force-pushed the stack/feat-rivetkit-trace-actor-to-actor-calls-zosqvyzx branch 2 times, most recently from 241f2c1 to 9d7ac37 Compare September 17, 2026 15:14
@eersnington
eersnington force-pushed the stack/feat-rivetkit-forward-otel-sdk-warnings-to-the-pino-logger-vmosovzz branch from 39bc2f6 to 403bbe7 Compare September 17, 2026 15:14
@eersnington
eersnington force-pushed the stack/feat-rivetkit-trace-actor-to-actor-calls-zosqvyzx branch from 9d7ac37 to aeeb217 Compare September 18, 2026 22:40
@eersnington
eersnington force-pushed the stack/feat-rivetkit-forward-otel-sdk-warnings-to-the-pino-logger-vmosovzz branch from 403bbe7 to a249107 Compare September 18, 2026 22:40
@eersnington
eersnington force-pushed the stack/feat-rivetkit-trace-actor-to-actor-calls-zosqvyzx branch from aeeb217 to e143d11 Compare September 19, 2026 01:01
@eersnington
eersnington force-pushed the stack/feat-rivetkit-forward-otel-sdk-warnings-to-the-pino-logger-vmosovzz branch from a249107 to 86c5005 Compare September 19, 2026 01:01
@eersnington
eersnington force-pushed the stack/feat-rivetkit-trace-actor-to-actor-calls-zosqvyzx branch from e143d11 to 6f7295c Compare September 21, 2026 22:56
@eersnington
eersnington force-pushed the stack/feat-rivetkit-forward-otel-sdk-warnings-to-the-pino-logger-vmosovzz branch from 86c5005 to 80a413c Compare September 21, 2026 22:56
@eersnington
eersnington force-pushed the stack/feat-rivetkit-trace-actor-to-actor-calls-zosqvyzx branch from 6f7295c to b965bf6 Compare September 22, 2026 15:28
@eersnington
eersnington force-pushed the stack/feat-rivetkit-forward-otel-sdk-warnings-to-the-pino-logger-vmosovzz branch from 80a413c to 2bc183b Compare September 22, 2026 15:28
feat(rivetkit): otel trace workflow runs and steps
…-tracing-docs-mqqwnuwl

docs(rivetkit): add actor tracing docs
@NathanFlurry
NathanFlurry merged commit d54f0d1 into stack/feat-rivetkit-trace-actor-to-actor-calls-zosqvyzx Sep 23, 2026
6 of 10 checks passed
@NathanFlurry
NathanFlurry deleted the stack/feat-rivetkit-forward-otel-sdk-warnings-to-the-pino-logger-vmosovzz branch September 23, 2026 08:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants