Repository navigation
[OTLP] Export the tracer's sampling decision in OTLP traces - #9498
Merged
Merged
Conversation
Emit _sampling_priority_v1, _dd.p.* trace tags and _dd.*_psr rates on OTLP spans with the same placement rules as the MessagePack formatter, and set the span sampled flag from the trace chunk's sampling priority. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anna-git
added
the
AI Generated
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
OTLP integration snapshots need updated baselines and normalization of the newly exported, run-dependent trace-ID attribute.
2 open issues
What changed in this PR
Preserves the tracer’s sampling decisions in OTLP exports so the Agent can honor them.
Changes:
- Exports sampling priorities, trace tags, and sampling rates.
- Derives sampled flags from trace-chunk priority.
- Adds unit tests for attribute placement, encoding, and flags.
| File | Description |
|---|---|
| tracer/test/Datadog.Trace.Tests/OpenTelemetry/Traces/OtlpTracesProtobufSerializerTests.cs | Tests flags and sampling attribute types. |
| tracer/test/Datadog.Trace.Tests/OpenTelemetry/Traces/OtlpTracesJsonSerializerTests.cs | Tests local sampling flags and omitted flags. |
| tracer/test/Datadog.Trace.Tests/OpenTelemetry/OtlpMapperTests.cs | Tests sampling attributes, placement, and limits. |
| tracer/src/Datadog.Trace/OpenTelemetry/Traces/OtlpTracesProtobufSerializer.cs | Uses chunk priority for protobuf flags. |
| tracer/src/Datadog.Trace/OpenTelemetry/Traces/OtlpTracesJsonSerializer.cs | Uses chunk priority for JSON flags. |
| tracer/src/Datadog.Trace/OpenTelemetry/OtlpMapper.cs | Emits sampling attributes and trace tags. |
Review effort: Balanced
BenchmarksBenchmark execution time: 2026-10-24 00:23:26 Comparing candidate commit 209e651 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 72 metrics, 0 unstable metrics, 9 known flaky benchmarks, 117 flaky benchmarks without significant changes.
|
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (9498) and master. ✅ No regressions detected |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Snapshots difference summaryThe following differences have been observed in committed snapshots. It is meant to help the reviewer. 28 occurrences of : + {
+ "key": "_dd.limit_psr",
+ "value": {
+ "doubleValue": 1.0
+ }
+ },
+ {
+ "key": "_dd.rule_psr",
+ "value": {
+ "doubleValue": 1.0
+ }
+ },
+ {
+ "key": "_sampling_priority_v1",
+ "value": {
+ "intValue": "2"
+ }
+ },
258 occurrences of : + "flags": 1
32 occurrences of : + {
+ "key": "_dd.p.dm",
+ "value": {
+ "stringValue": "-3"
+ }
+ },
+ {
+ "key": "_dd.p.ksr",
+ "value": {
+ "stringValue": "1"
+ }
+ },
2 occurrences of : + "attributes": [
+ {
+ "key": "_dd.limit_psr",
+ "value": {
+ "doubleValue": 1.0
+ }
+ },
+ {
+ "key": "_dd.rule_psr",
+ "value": {
+ "doubleValue": 1.0
+ }
+ },
+ {
+ "key": "_sampling_priority_v1",
+ "value": {
+ "intValue": "2"
+ }
+ }
+ ],
+ "flags": 1
3 occurrences of : + {
+ "key": "_sampling_priority_v1",
+ "value": {
+ "intValue": "2"
+ }
+ },
+ {
+ "key": "_dd.rule_psr",
+ "value": {
+ "doubleValue": 1.0
+ }
+ },
+ {
+ "key": "_dd.limit_psr",
+ "value": {
+ "doubleValue": 1.0
+ }
+ },
4 occurrences of : + {
+ "key": "_sampling_priority_v1",
+ "value": {
+ "intValue": "2"
+ }
+ },
+ {
+ "key": "_dd.p.dm",
+ "value": {
+ "stringValue": "-3"
+ }
+ },
+ {
+ "key": "_dd.p.ksr",
+ "value": {
+ "stringValue": "1"
+ }
+ },
+ {
+ "key": "_dd.rule_psr",
+ "value": {
+ "doubleValue": 1.0
+ }
+ },
+ {
+ "key": "_dd.limit_psr",
+ "value": {
+ "doubleValue": 1.0
+ }
+ },
1 occurrences of : + "attributes": [
+ {
+ "key": "_sampling_priority_v1",
+ "value": {
+ "intValue": "2"
+ }
+ },
+ {
+ "key": "_dd.rule_psr",
+ "value": {
+ "doubleValue": 1.0
+ }
+ },
+ {
+ "key": "_dd.limit_psr",
+ "value": {
+ "doubleValue": 1.0
+ }
+ }
+ ],
+ "flags": 1
1 occurrences of : + {
+ "key": "_dd.p.dm",
+ "value": {
+ "stringValue": "-3"
+ }
+ },
+ {
+ "key": "_dd.p.ksr",
+ "value": {
+ "stringValue": "1"
+ }
+ },
[...]
+ "flags": 1
1 occurrences of : + "attributes": [
+ {
+ "key": "_sampling_priority_v1",
+ "value": {
+ "intValue": "2"
+ }
+ },
+ {
+ "key": "_dd.p.dm",
+ "value": {
+ "stringValue": "-3"
+ }
+ },
+ {
+ "key": "_dd.p.ksr",
+ "value": {
+ "stringValue": "1"
+ }
+ },
+ {
+ "key": "_dd.rule_psr",
+ "value": {
+ "doubleValue": 1.0
+ }
+ },
+ {
+ "key": "_dd.limit_psr",
+ "value": {
+ "doubleValue": 1.0
+ }
+ }
+ ],
[...]
+ "flags": 1
1 occurrences of : + "attributes": [
+ {
+ "key": "_sampling_priority_v1",
+ "value": {
+ "intValue": "2"
+ }
+ },
+ {
+ "key": "_dd.p.dm",
+ "value": {
+ "stringValue": "-3"
+ }
+ },
+ {
+ "key": "_dd.p.ksr",
+ "value": {
+ "stringValue": "1"
+ }
+ },
+ {
+ "key": "_dd.rule_psr",
+ "value": {
+ "doubleValue": 1.0
+ }
+ },
+ {
+ "key": "_dd.limit_psr",
+ "value": {
+ "doubleValue": 1.0
+ }
+ }
+ ],
2 occurrences of : + "attributes": [
+ {
+ "key": "_sampling_priority_v1",
+ "value": {
+ "intValue": "2"
+ }
+ },
+ {
+ "key": "_dd.p.dm",
+ "value": {
+ "stringValue": "-3"
+ }
+ },
+ {
+ "key": "_dd.p.ksr",
+ "value": {
+ "stringValue": "1"
+ }
+ },
+ {
+ "key": "_dd.rule_psr",
+ "value": {
+ "doubleValue": 1.0
+ }
+ },
+ {
+ "key": "_dd.limit_psr",
+ "value": {
+ "doubleValue": 1.0
+ }
+ }
+ ],
+ "flags": 1
1 occurrences of : + "attributes": [
+ {
+ "key": "_dd.p.dm",
+ "value": {
+ "stringValue": "-3"
+ }
+ },
+ {
+ "key": "_dd.p.ksr",
+ "value": {
+ "stringValue": "1"
+ }
+ }
+ ],
+ "flags": 1
106 occurrences of : + {
+ "key": "_dd.limit_psr",
+ "value": {
+ "doubleValue": 1.0
+ }
+ },
+ {
+ "key": "_dd.p.dm",
+ "value": {
+ "stringValue": "-3"
+ }
+ },
+ {
+ "key": "_dd.p.ksr",
+ "value": {
+ "stringValue": "1"
+ }
+ },
+ {
+ "key": "_dd.rule_psr",
+ "value": {
+ "doubleValue": 1.0
+ }
+ },
+ {
+ "key": "_sampling_priority_v1",
+ "value": {
+ "intValue": "2"
+ }
+ },
|
The upper 64 bits of the trace id are already part of the 128-bit OTLP traceId. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anna-git
force-pushed
the
anna.yafi/otlp-export-sampling-decision
branch
from
October 22, 2026 12:00
b4d1dd3 to
816044a
Compare
The http/json exporter writes flags after parentSpanId while the protobuf payload renders it last, so tests sharing a snapshot across protocols disagreed once local spans carry flags. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g decision Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![datadog-datadog-us1-prod[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
More details
Sampling attributes follow the existing MessagePack placement rules, and both OTLP serializers use the chunk sampling decision for span flags. The specialist returned no reportable regression; runtime tests remain unverified because the required .NET 11 preview SDK was unavailable.
🤖 Bits Code Review · Commit 421dbe9
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 421dbe9eeb
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
zacharycmontoya
approved these changes
Oct 22, 2026
zacharycmontoya
left a comment
zacharycmontoya
left a comment
Contributor
There was a problem hiding this comment.
LGTM with one more small suggestion
mcculls
removed their request for review
October 22, 2026 20:39
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… order Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anna-git
force-pushed
the
anna.yafi/otlp-export-sampling-decision
branch
from
October 23, 2026 15:11
21d8b9b to
f44063a
Compare
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
anna-git
enabled auto-merge (squash)
October 26, 2026 11:48
andrewlock
approved these changes
Oct 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
When traces are exported over OTLP (
OTEL_TRACES_EXPORTER=otlp), send the tracer's sampling decision, the same way the Datadog (MessagePack) export does:_sampling_priority_v1on chunk orphans, sent as an OTLP int attribute (the Agent converts int and double attributes to the same span metric)_dd.p.*tags (_dd.p.dm,_dd.p.ksr, ...) on the first span of each chunk, other trace tags on the local root_dd.agent_psr/_dd.rule_psr/_dd.limit_psron the local rootflags"sampled" bit now comes from the trace chunk's sampling priority. Before, it only used the span context's priority, which is only set for propagated contexts, so local spans never got the flag.Reason for change
The OTLP path dropped every sampling-decision marker. The Agent could not see the tracer's decision, so it ran its own OTLP probabilistic sampler and stamped
_dd.p.dm=-9with priority 1. Decisions made by remote/adaptive sampling rules (dm-11/-12, priority 2) were applied by the tracer but not attributed by the backend, so adaptive sampling could not steer services that export over OTLP.This complements DataDog/datadog-agent#58471, which makes the Agent keep the tracer's
_dd.p.dmfrom OTLP span attributes.Implementation details
OtlpMapper.EmitAttributesFromSpanwrites the new attributes with the same conditions and placement asSpanMessagePackFormatter:TraceTagWriter<TState>struct, the OTLP counterpart of the MessagePack one. Trace tags go through the same tag processors.OtlpTracesProtobufSerializer/OtlpTracesJsonSerializer:flagsusesTraceChunk.SamplingPriority ?? Span.Context.SamplingPriority.Test coverage
OtlpMapperTests:dm,ksrand psr rates for each sampling mechanismOtlpTracesProtobufSerializerTests/OtlpTracesJsonSerializerTests:OtlpMapperTests,OtlpTraces*,SpanMessagePackFormatterTestsandApiOtlpTestsall pass locally (203 tests, net11.0).dm -9, priority 1dm -11/-12, priority 2flags. They have been updated in this PR:tracer/test/snapshots/Otlp*(AspNetCore MinimalApis/Mvc21/Mvc31, AspNetMvc5, AspNetWebApi2, OwinWebApi2),OpenTelemetrySdkTests.SubmitsOtlpTraces*,OpenTelemetryHttpClientTests.SubmitsOtlpTraces*andOpenTelemetryWebRequestTests.*.Other details
Follow-ups, not in this PR:
dd=member (s:,t.dm:) into the OTLP spantrace_state, appended to the existingot=member, as dd-trace-go and libdatadog do. The Agent already gets the decision from the attributes, so this mainly matters for OTLP consumers other than the Agent._dd.tracer_krandprocess_id, which the MessagePack path also writes on the local root, are still not exported over OTLP._dd.p.*trace tags are classified case-insensitively in the OTLP export, matchingTagPropagation.ParseHeader, which accepts the prefix case-insensitively and keeps the original casing.SpanMessagePackFormatter.TraceTagWriterand header injection inTagPropagationstill compare ordinally, so an incoming_DD.P.*key is only written on the local root and isn't propagated downstream on those paths. Intentionally left unchanged here.SpanMessagePackFormatterandOtlpMapper. They could move into a shared helper.🤖 Generated with Claude Code