Skip to content

feat(rivetkit): otel trace workflow runs and steps - #5753

Merged
NathanFlurry merged 5 commits into
stack/docs-rivetkit-add-actor-tracing-docs-mqqwnuwlfrom
wip/workflow-tracing
Sep 23, 2026
Merged

NathanFlurry merged 5 commits into
stack/docs-rivetkit-add-actor-tracing-docs-mqqwnuwlfrom
wip/workflow-tracing

Conversation

@eersnington

@eersnington eersnington commented Sep 19, 2026 •

Copy link
Copy Markdown
Member

Adds traces for rivetkit/workflow. Sits on top of stack/docs-rivetkit-add-actor-tracing-docs-mqqwnuwl #5734

await ctx.step("charge-card", () => { throw new UserError("declined", { code: "card_declined" }) });
orderFlow/workflow          run 2  link -> run 1  outcome=sleeping  ray=<queue message ray>
└── orderFlow/charge-card   attempt=1  outcome=retry   ERROR  error.type=user.card_declined
orderFlow/workflow          run 3  link -> run 2  outcome=completed
└── orderFlow/charge-card   attempt=2  outcome=ok
    └── rivet.sqlite.execute
  • Each run is its own trace and links back to the run before it. Each step attempt is a span in that trace.
  • SQLite, c.client() calls, queue sends, and logs inside a step nest under the step's span.
  • Replayed steps are not traced again.
  • The last run's span and ray are saved in _rivet_wf_kv, so runs stay linked across actor sleep. No schema migration.
  • A run uses the ray of the queue message that woke it. A workflow that receives no queue message has no ray, because actor creation rays do not reach the actor.
  • New experimental hooks c.run.startWorkflowSpan(), c.run.startWorkflowStepSpan(name, attempt), and c.run.runOutsideWorkflowSpan(body), for @rivet-dev/workflows to call later.
  • Docs mention the ray limitation.

@railway-app

railway-app Bot commented Sep 19, 2026

Copy link
Copy Markdown

This PR was not deployed automatically as @eersnington does not have access to the Railway project.

In order to get automatic PR deploys, please add @eersnington to your workspace on Railway.

@eersnington
eersnington changed the base branch from main to stack/docs-rivetkit-add-actor-tracing-docs-mqqwnuwl September 19, 2026 01:35
@eersnington eersnington changed the title [WIP]: OTel traces for Workflow [WIP] feat(rivetkit): OTel traces for Workflow Sep 19, 2026
@eersnington
eersnington force-pushed the stack/docs-rivetkit-add-actor-tracing-docs-mqqwnuwl branch from 061c0ec to a182e43 Compare September 21, 2026 22:56
@claude

claude Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Review (updated)

Re-reviewed against the latest push. The only functional delta since the last review is the fix(rivetkit): finish a step span after its result is written commit, which moves stepSpan?.finish("ok") in context.ts's success path to after await this.flushStorage() instead of before it. Traced through the surrounding code (success path and all three error branches: StepTimeoutError, CriticalError/RollbackError, and the generic retry/exhausted path) and the span now consistently closes only once the step's outcome is actually durable, with every path still finishing the span exactly once (the Drop impl on the Rust side remains a safe no-op backstop via the state.finished guard in finish_with). This is a correct, well-scoped fix.

Checking off items from the prior review against the current diff:

  • Formatting (long unwrapped line): resolved — start_workflow_step_span in both rivetkit-core/src/actor/context.rs and rivetkit-napi/src/actor_context.rs is properly multi-line now.
  • Stale PR description ("will be moved" wording): resolved — the description now says the trace context "is saved in _rivet_wf_kv" directly.
  • Duplicated outcome unions (WorkflowOutcome/WorkflowStepOutcome defined independently in workflow-engine/src/driver.ts, rivetkit/src/actor/config.ts, and mirrored as Rust enums with their own parse/as_label): still the case. Understandable given the layering, but still no compiler check across the boundary if a variant is ever added/renamed.
  • Cross-language key-encoding test gap: still open. Nothing asserts that Rust's WORKFLOW_TRACE_CONTEXT_KEY = [6, 1, 0x15, 5] and the TS side's reservation of KEY_PREFIX slot 5 (fdb-tuple's encoding of a length-1 tuple [5]) actually agree beyond the two hand-written comments pointing at each other. A drift here wouldn't fail any test, it would just silently stop linking workflow runs across sleep (or collide with a future workflow-engine key). Low cost to add a small Rust test asserting the literal against a known-good tuple encoding, or a driver test that round-trips through both layers.
  • Perf note (informational): still applies — ActorWorkflowDriver.telemetry is always constructed, so every workflow pass pays the ctx.startWorkflowSpan() NAPI round-trip (including the SQL read of the previous trace context) even when tracing is fully disabled at the collector level, since tracing::enabled! still gates the actual SQL read/write inside core but not the NAPI call itself. Not a regression, just worth knowing for very high-frequency retry workflows.

Also spot-checked a few things that looked risky at a glance and turned out fine:

  • WorkflowRunInvocation/WorkflowStepSpan both call finish_with from an explicit finish() and (redundantly, safely) from Drop; the state.finished guard makes the second call a no-op, so no double-recording of otel.status_code/error.type.
  • InvocationState.ray_id mutation in start_queue_receive (added to let a workflow run take on the ray of the queue message that woke it) locks, mutates, reads, and explicitly drop(state)s before span.set_parent(...), so there's no lock held across anything resembling an await and no reentrant-lock risk with record_ray.
  • NapiCoreRuntime#invocationContext.exit(run) for runOutsideWorkflowSpan matches Node's documented AsyncLocalStorage.exit() semantics (suppresses the store across the async continuations of run, not just its synchronous portion), so the workflow engine's own history-storage I/O correctly stays untraced even across awaits.

No correctness, security, or test-coverage blockers. The workflow test suite in actor-telemetry.test.ts (run-linking chain across sleep, step replay-once semantics, per-attempt retry reporting, ray isolation/propagation across messages, log/trace-id correlation inside steps) continues to cover the new behavior well.

@eersnington eersnington changed the title [WIP] feat(rivetkit): OTel traces for Workflow feat(rivetkit): otel trace workflow runs and steps Sep 22, 2026
@eersnington
eersnington marked this pull request as ready for review September 22, 2026 14:03

@the-company-company the-company-company Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 1 medium-severity finding

Reviewed commit 1134011.

Comment thread rivetkit-typescript/packages/workflow-engine/src/context.ts Outdated
@eersnington
eersnington force-pushed the stack/docs-rivetkit-add-actor-tracing-docs-mqqwnuwl branch from a182e43 to 465da0d Compare September 22, 2026 15:28

@the-company-company the-company-company Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues found

Reviewed commit f71c35a.

@NathanFlurry
NathanFlurry merged commit e614052 into stack/docs-rivetkit-add-actor-tracing-docs-mqqwnuwl Sep 23, 2026
7 of 12 checks passed
@NathanFlurry
NathanFlurry deleted the wip/workflow-tracing branch September 23, 2026 08:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants