feat(rivetkit-core): trace queue sends and receives

[eersnington]

• Connecting queue receives to their sends, including after restarts
• Placing queue spans under the current invocation, or starting a new trace for calls from run()

[eersnington]

Stack for rivet-dev/actors

• #5734 docs(rivetkit): add actor tracing docs
• #5733 feat(rivetkit): forward otel sdk warnings to the pino logger
• #5732 feat(rivetkit): trace actor to actor calls
• #5730 feat(rivetkit-core): trace queue sends and receives 👈
• #5731 feat(rivetkit-core): trace raw http requests
• #5729 feat(rivetkit-core): trace scheduled invocations
• #5726 feat(rivetkit): send trace context from clients
• #5723 feat(rivetkit): pass trace context to js and logs
• #5727 feat(rivetkit): trace sqlite operations
• #5721 feat(rivetkit): trace actor invocations
• #5728 fix(pegboard-gateway): forward request ray id to actors
• #5724 test(rivetkit): let driver runtimes take extra environment
• #5722 fix(rivetkit): disable services in driver test runtimes
• main

Get stack: forklift get 5730
Push local edits: forklift submit
Merge when ready: forklift merge 5730

_{change wqpoztoq}

[the-company-company]

🟠 1 medium-severity finding

_{Reviewed commit 23d24a3.}

[the-company-company]

🟠 Medium · Preserve queue-send telemetry in the WASM callback context

The new invocation telemetry is discarded before the WASM onQueueSend callback receives its context. Consequently, actorCtx.queue.send() runs on the unbound actor context, so ActorContext::send persists an empty TraceOrigin; a later queue receipt cannot link to the queue-send producer span. This makes the advertised queue trace propagation fail for WASM actors even though the core span is created.

Construct the callback's WasmActorContext with ctx.inner().clone().with_invocation_telemetry(invocation_telemetry) (while retaining the shared wrapper state), as the NAPI payload path does.

[claude]

Review: feat(rivetkit-core): trace queue sends and receives

Traced through the full path: schema migration to persistence to span creation to NAPI/TS wiring to tests. This is well-executed; no correctness bugs found.

What I checked closely

• Schema migration (internal_storage/schema.rs): _rivet_queue gets three new nullable columns via ALTER TABLE in MIGRATION_V2. Confirmed INTERNAL_SCHEMA_VERSION is still 1 on main, so this migration has not shipped yet, so it is safe to extend in this stacked PR rather than adding a new migration version.
• Column/param ordering: INSERT_QUEUE_MESSAGE_SQL columns (id, name, body, created_at, ray_id, traceparent, tracestate) line up correctly with trace_context_params() append order, and LOAD_QUEUE_MESSAGES_* select order matches read_trace_context(row, 4, ...) in decode_queue_message_rows.
• persist_queue_messages defaulting to IncomingTraceContext::default(): this is only used by the KV to SQLite migration path (migrate_kv_to_sqlite/mod.rs), where no trace context ever existed, so defaulting is correct rather than a lost-data bug.
• Span lifecycle on QueueSend dispatch failure (task.rs): when send_actor_event fails and enqueue_failed is logged, the invocation is dropped without an explicit .finish() call, but ActorInvocation Drop impl calls finish_dropped(), and separately Reply::drop delivers a DroppedReply error through the on_reply hook either way. The span still closes with an error status; no leak.
• _receive_spans in try_receive_batch (queue.rs): spans are created only after selected is confirmed non-empty (the empty case returns before span creation), and the vec is dropped at the end of the same function call, closing the spans immediately. This matches the doc comment on start_queue_receive (it closes when the caller drops it, which try_receive_batch does as it hands the message back). A point-in-time span by design, not a bug.
• Ray ID selection in start_queue_receive: when inside an invocation, it correctly uses the receiving invocations ray ID (not the senders), falling back to message.trace_context.ray_id when there is no active invocation (e.g. the run() loop). This matches both the doc comment and the new driver test assertions.
• Telemetry scoping: ActorContext invocation telemetry is stored per-handle (Self(Arc<Inner>, Option<ActorInvocationTelemetry>)), not shared mutable state, so concurrent invocations on the same actor cannot cross-contaminate ray IDs or spans.
• NAPI wiring: QueueSendPayload.telemetry is correctly threaded into ActorContext::with_invocation_telemetry in actor_factory.rs, and napi-runtime.ts was fixed to route queue operations through #actorContextForOperation instead of asNativeActorContext, so nested ctx.queue.send() calls from user code now correctly inherit the current invocation trace context (previously would not have).
• Wasm parity: invocation_telemetry is discarded (_) in rivetkit-wasm/src/lib.rs for QueueSend, but this matches the pre-existing behavior for Action and HttpRequest dispatch in the same file. Wasm does not bind invocation telemetry to context for any dispatch type yet, so this is not a new gap introduced by this PR.
• Match exhaustiveness: otel_kind, InvocationSubject::name()/record_attributes(), and InvocationType::as_label() all enumerate every enum variant explicitly, per the repo convention against _-fallthrough on enums.
• Test coverage: the new driver test (actor-telemetry.test.ts) covers both cases described in the design: a receipt inside a receiving action (parented under that action span, ray ID from the receiving invocation) and a receipt from a run() consumer with no wrapping invocation (root span, ray ID from the original sender). Both assert the add_link back to the sending span.

Minor, non-blocking observations

• queue.rs::send() clones trace_context once to pass into persist_queue_message and once for the returned QueueMessage. Given IncomingTraceContext is just a few Option<String> fields and this is not a hot path (single-writer per actor), not worth restructuring.
• IncomingTraceContext fields were widened from pub(crate) to pub, reasonable given it is now re-exported from lib.rs, just flagging for awareness since it slightly grows the public surface.

Nice work overall. The design comments in telemetry.rs (particularly on start_queue_receive) made it easy to verify the intended behavior against the implementation.

[the-company-company]

🟠 1 medium-severity finding

_{Reviewed commit c90f7b2.}

[the-company-company]

🟠 Medium · Preserve queue-send telemetry in the WASM callback context

The new invocation telemetry is discarded before the WASM onQueueSend callback receives its context. Consequently, actorCtx.queue.send() runs on the unbound actor context, so ActorContext::send persists an empty TraceOrigin; a later queue receipt cannot link to the queue-send producer span. This makes queue trace propagation fail for WASM actors even though the core span is created.

Construct the callback's WasmActorContext with ctx.inner().clone().with_invocation_telemetry(invocation_telemetry) while retaining the shared wrapper state, as the NAPI payload path does.

[the-company-company]

🟠 1 medium-severity finding

_{Reviewed commit f5d2327.}

[the-company-company]

🟠 Medium · Preserve queue-send telemetry in the WASM callback context

The new invocation telemetry is discarded before the WASM onQueueSend callback receives its context. Consequently, actorCtx.queue.send() runs on the unbound actor context, so ActorContext::send persists an empty IncomingTraceContext; a later queue receipt cannot link to the queue-send producer span. This makes queue trace propagation fail for WASM actors even though the core span is created.

Construct the callback's WasmActorContext with ctx.inner().clone().with_invocation_telemetry(invocation_telemetry) while retaining the shared wrapper state, as the NAPI payload path does.

[the-company-company]

🟠 1 medium-severity finding

_{Reviewed commit fd34b25.}

[the-company-company]

🟠 Medium · Preserve queue-send telemetry in the WASM callback context

The new invocation telemetry is discarded before the WASM onQueueSend callback receives its context. Consequently, actorCtx.queue.send() runs on the unbound actor context, so ActorContext::send persists an empty IncomingTraceContext; a later queue receipt cannot link to the queue-send producer span. This makes queue trace propagation fail for WASM actors even though the core span is created.

Construct the callback's WasmActorContext with ctx.inner().clone().with_invocation_telemetry(invocation_telemetry) while retaining the shared wrapper state, as the NAPI payload path does.

[the-company-company]

🟠 1 medium-severity finding

_{Reviewed commit 16e4368.}

[the-company-company]

🟠 Medium · Preserve queue-send telemetry in the WASM callback context

The new invocation telemetry is discarded before the WASM onQueueSend callback receives its context. Consequently, actorCtx.queue.send() runs on the unbound actor context, so ActorContext::send persists an empty IncomingTraceContext; a later queue receipt cannot link to the queue-send producer span. This makes queue trace propagation fail for WASM actors even though the core span is created.

Construct the callback's WasmActorContext with ctx.inner().clone().with_invocation_telemetry(invocation_telemetry) while retaining the shared wrapper state, as the NAPI payload path does.

[claude]

Review: feat(rivetkit-core): trace queue sends and receives

Independently traced the full path (schema → persistence → span creation → NAPI/TS wiring → tests) against this PR's current commit. This is well-executed and consistent with the existing invocation-telemetry pattern from the scheduled-fire tracing feature; no correctness bugs found.

What I checked closely

• Schema migration: _rivet_queue gets ray_id/traceparent/tracestate via the existing MIGRATION_V2 in schema.rs (already staged on this stacked branch for _rivet_schedule_events). Column order in INSERT_QUEUE_MESSAGE_SQL / LOAD_QUEUE_MESSAGES_* matches trace_context_params() append order and read_trace_context(row, 4, ...) read order.
• persist_queue_messages defaulting to IncomingTraceContext::default(): only used by the KV→SQLite migration path, where no trace context ever existed, so defaulting is correct, not a lost-data bug. (A one-line comment there would save the next reader the trip to migrate_kv_to_sqlite/mod.rs to confirm this.)
• Span lifecycle on QueueSend dispatch failure (task.rs): if send_actor_event fails, the event (including reply) is dropped, Reply::drop delivers a DroppedReply error through on_reply, and invocation.finish(...) still runs. No leaked/unclosed span.
• _receive_spans in try_receive_batch: created once per selected message, each with its own trace_context, and dropped at the end of the same function call (i.e. it's a correlation marker for the handoff, not a span wrapping consumer processing). This matches the start_queue_receive doc comment and the new test assertions.
• Ray ID / parent selection in start_queue_receive: inside an active invocation (e.g. consumeJob), it parents to that invocation and uses its ray ID; with no active invocation (e.g. the run() loop), it falls back to a root span using message.trace_context.ray_id. In both cases it unconditionally adds the OTel link back to the sender's span via parse_remote_parent. Matches both branches asserted in the new driver test.
• Send-side trace context capture (queue.rs::enqueue_message): reads self.invocation_telemetry() at send time, so a /queue/send HTTP dispatch captures the new queue.send producer invocation, while c.queue.send() called from inside an action/request correctly captures that ambient invocation instead. This depends on NAPI's QueueSendPayload.telemetry → ctx.with_invocation_telemetry(...) wiring in actor_factory.rs, which mirrors the pre-existing ActionPayload pattern exactly.
• napi-runtime.ts: queue operations (send/nextBatch/waitForNames/enqueueAndWait) now route through #actorContextForOperation instead of asNativeActorContext, so nested ctx.queue.* calls from user code correctly inherit the ambient application span, consistent with how SQL/waitUntil already worked. native.ts's queue-send dispatch handler is now wrapped in runtime.runWithActorInvocationContext, matching the existing action/HTTP dispatch pattern, which looks like a genuine fix for a prior gap.
• Wasm parity: invocation_telemetry is discarded (_) in rivetkit-wasm/src/lib.rs, and wasm-runtime.ts's runWithActorInvocationContext/actorInvocationTraceContext are pre-existing no-op stubs for every dispatch type (action, HTTP, queue). So wasm doesn't get invocation-scoped telemetry threading yet, but that's a pre-existing, general limitation this PR doesn't regress or need to fix.
• Exhaustiveness: otel_kind, InvocationSubject::name()/record_attributes(), and InvocationType::as_label() all enumerate every variant explicitly (no _ => fallback), per repo convention.
• IncomingTraceContext visibility (flagged as an open question in an earlier pass of this review): widening its fields from pub(crate) to pub, plus the lib.rs re-export, is actually required here, not just a nice-to-have. QueueMessage/CompletableQueueMessage are pub structs re-exported from lib.rs, and their new pub trace_context: IncomingTraceContext field means the field's type has to be at least as visible as the field itself, or the crate wouldn't compile with IncomingTraceContext staying crate-private. So this is a correct, necessary change.

Test coverage

The new actor-telemetry.test.ts case ("links a queue receipt to the send that produced it") covers both the "receipt nested under an active invocation" case and the "receipt as a root span from run()" case, asserting the OTel link back to the originating send span by trace/span ID rather than just presence. Rust-side changes are exercised transitively: the modified SELECTs are covered by the existing sql_efficiency suite since those tests reference the shared SQL constants directly, and the PersistedQueueMessage/ActorEvent::QueueSend match-arm updates in the two Rust integration test fixtures are exhaustiveness-driven (compile-time enforced), not new behavior needing new assertions.

Minor, non-blocking

• queue.rs::enqueue_message clones trace_context once for persist_queue_message and once for the returned QueueMessage. Given it's a few Option<String> fields on a single-writer-per-actor path, not worth restructuring.

No security or performance concerns beyond what's already accepted in the sibling scheduled-fire tracing feature this PR extends (traceparent/tracestate from client headers are parsed/normalized through the OTel W3C propagator, not stored raw, before being persisted; ray IDs are bounded and character-restricted via bounded_ray_id).

[the-company-company]

🟠 1 medium-severity finding

_{Reviewed commit fb29ff8.}

[the-company-company]

🟠 Medium · Preserve queue-send telemetry in the WASM callback context

The new invocation telemetry is discarded before the WASM onQueueSend callback receives its context. Consequently, actorCtx.queue.send() runs on the unbound actor context, so ActorContext::send persists an empty IncomingTraceContext; a later queue receipt cannot link to the queue-send producer span. This makes queue trace propagation fail for WASM actors even though the core span is created.

Construct the callback's WasmActorContext with ctx.inner().clone().with_invocation_telemetry(invocation_telemetry) while retaining the shared wrapper state, as the NAPI payload path does.

[the-company-company]

🟠 1 medium-severity finding

_{Reviewed commit fa09a2e.}

[the-company-company]

🟠 Medium · Preserve queue-send telemetry in the WASM callback context

The new invocation telemetry is discarded before the WASM onQueueSend callback receives its context. Consequently, actorCtx.queue.send() runs on the unbound actor context, so ActorContext::send persists an empty IncomingTraceContext; a later queue receipt cannot link to the queue-send producer span. This makes queue trace propagation fail for WASM actors even though the core span is created.

Construct the callback's WasmActorContext with ctx.inner().clone().with_invocation_telemetry(invocation_telemetry) while retaining the shared wrapper state, as the NAPI payload path does.

[the-company-company]

🟠 1 medium-severity finding

_{Reviewed commit c3ef082.}

[the-company-company]

🟠 Medium · Preserve queue-send telemetry in the WASM callback context

The new invocation telemetry is discarded before the WASM onQueueSend callback receives its context. Consequently, actorCtx.queue.send() runs on the unbound actor context, so ActorContext::send persists an empty IncomingTraceContext; a later queue receipt cannot link to the queue-send producer span. This makes queue trace propagation fail for WASM actors even though the core span is created.

Construct the callback's WasmActorContext with ctx.inner().clone().with_invocation_telemetry(invocation_telemetry) while retaining the shared wrapper state, as the NAPI payload path does.

[the-company-company]

🟠 1 medium-severity finding

_{Reviewed commit 4a4b929.}

[the-company-company]

🟠 Medium · Preserve queue-send telemetry in the WASM callback context

The new invocation telemetry is discarded before the WASM onQueueSend callback receives its context. Consequently, actorCtx.queue.send() runs on the unbound actor context, so ActorContext::send persists an empty IncomingTraceContext; a later queue receipt cannot link to the queue-send producer span. This makes queue trace propagation fail for WASM actors even though the core span is created.

Construct the callback's WasmActorContext with ctx.inner().clone().with_invocation_telemetry(invocation_telemetry) while retaining the shared wrapper state, as the NAPI payload path does.

[the-company-company]

🟠 1 medium-severity finding

_{Reviewed commit f3329fb.}

[the-company-company]

🟠 Medium · Preserve queue-send telemetry in the WASM callback context

The new invocation telemetry is discarded before the WASM onQueueSend callback receives its context. Consequently, actorCtx.queue.send() runs on the unbound actor context, so ActorContext::send persists an empty IncomingTraceContext; a later queue receipt cannot link to the queue-send producer span. This makes queue trace propagation fail for WASM actors even though the core span is created.

Construct the callback's WasmActorContext with ctx.inner().clone().with_invocation_telemetry(invocation_telemetry) while retaining the shared wrapper state, as the NAPI payload path does.

[the-company-company]

🟠 1 medium-severity finding

_{Reviewed commit 3219ea9.}

[the-company-company]

🟠 Medium · Preserve queue-send telemetry in the WASM callback context

The new invocation telemetry is discarded before the WASM onQueueSend callback receives its context. Consequently, actorCtx.queue.send() runs on the unbound actor context, so ActorContext::send persists an empty IncomingTraceContext; a later queue receipt cannot link to the queue-send producer span. This makes queue trace propagation fail for WASM actors even though the core span is created.

Construct the callback's WasmActorContext with ctx.inner().clone().with_invocation_telemetry(invocation_telemetry) while retaining the shared wrapper state, as the NAPI payload path does.

[the-company-company]

🟠 1 medium-severity finding

_{Reviewed commit e987d4f.}

[the-company-company]

🟠 Medium · Preserve queue-send telemetry in the WASM callback context

The new invocation telemetry is discarded before the WASM onQueueSend callback receives its context. Consequently, actorCtx.queue.send() runs on the unbound actor context, so ActorContext::send persists an empty IncomingTraceContext; a later queue receipt cannot link to the queue-send producer span. This makes queue trace propagation fail for WASM actors even though the core span is created.

Construct the callback's WasmActorContext with ctx.inner().clone().with_invocation_telemetry(invocation_telemetry) while retaining the shared wrapper state, as the NAPI payload path does.

[the-company-company]

🟠 1 medium-severity finding

_{Reviewed commit 1de582e.}

[the-company-company]

🟠 Medium · Preserve queue-send telemetry in the WASM callback context

The new invocation telemetry is discarded before the WASM onQueueSend callback receives its context. Consequently, actorCtx.queue.send() runs on the unbound actor context, so ActorContext::send persists an empty IncomingTraceContext; a later queue receipt cannot link to the queue-send producer span. This makes queue trace propagation fail for WASM actors even though the core span is created.

Construct the callback's WasmActorContext with ctx.inner().clone().with_invocation_telemetry(invocation_telemetry) while retaining the shared wrapper state, as the NAPI payload path does.

[the-company-company]

🟠 1 medium-severity finding

_{Reviewed commit f582735.}

[the-company-company]

🟠 Medium · Preserve queue-send telemetry in the WASM callback context

The new invocation telemetry is discarded before the WASM onQueueSend callback receives its context. Consequently, actorCtx.queue.send() runs on the unbound actor context, so ActorContext::send persists an empty IncomingTraceContext; a later queue receipt cannot link to the queue-send producer span. This makes queue trace propagation fail for WASM actors even though the core span is created.

Construct the callback's WasmActorContext with ctx.inner().clone().with_invocation_telemetry(invocation_telemetry) while retaining the shared wrapper state, as the NAPI payload path does.

[NathanFlurry]

Reviewed as part of the tracing stack.