opentelemetry plugin emits no CLIENT span for the upstream call, so APISIX is disconnected on APM service maps

[alice101-dev]

Summary

The opentelemetry plugin emits exactly one span per request, with
kind = span_kind.server, and never a CLIENT span for the call APISIX makes
to the upstream.

Tracing itself is unaffected — the waterfall is complete and correctly
parented. But APM tools that build a service map from CLIENT → SERVER span
pairs cannot draw the gateway → upstream hop, so APISIX shows up as an island
next to the services it actually proxies to.

I would like to check whether adding such a span is something the project
would accept, and if so where the right place to start and finish it is.

What it looks like

Same application, same collector, same APM (HyperDX / ClickStack). Only the
gateway differs.

Fronted by APISIX — two disconnected clusters. api-gateway is joined to
the callers in front of it but not to orders-service behind it.

Fronted by Kong — one continuous chain, browser through to the backends.

Screenshots of both are in the comment below.

Why

Every cross-service parent/child pair in the trace data, with the span kinds:

parent	parent kind	child	child kind	edge drawn
orders	Client	inventory	Server	✅
orders	Client	payments	Server	✅
storefront-web	Client	api-gateway	Server	✅
api-gateway	Server	orders	Server	❌
api-gateway	Server	auth	Server	❌

The two missing edges are exactly the two whose parent is APISIX. Context
propagation is fine — the child spans exist and are correctly parented to the
gateway span. There is simply no CLIENT span for the proxy hop to anchor the
edge to.

In the source

apisix/plugins/opentelemetry.lua, checked on 3.17.0 and on master — one
request span:

local ctx = tracer:start(upstream_context, span_name, {
    kind = span_kind.server,
    attributes = attributes,
})

span_kind.client does not appear in the file at all.

The internal tracer (apisix/tracer.lua) can add child spans, but each one is
server or internal and describes an APISIX phase rather than the upstream
call:

apisix.phase.access          server
apisix.phase.header_filter   server
apisix.phase.body_filter     server
http_router_match            internal
resolve_dns                  internal

The plugin schema exposes only sampler, additional_attributes and
additional_header_prefix_attributes, so it cannot be enabled through
configuration either.

For comparison, Kong's OpenTelemetry plugin emits a kong.balancer span for
this hop, which is what makes the second screenshot connect end to end.

What I am proposing

A CLIENT span around the upstream request — started before proxying, finished
when the upstream response completes — as a child of the existing SERVER
span, with the usual client attributes (server.address, server.port,
http.response.status_code, upstream/node identity).

Besides fixing service maps, it would make the time APISIX spends waiting on
the upstream visible as its own span. Today that is only inferable from the gap
between the gateway span and the upstream service's server span.

I am happy to open a PR if the direction is agreeable. The part I would want
guidance on is which phase should own the span's start and finish
(before_proxy / body_filter / log), given that the plugin currently
builds its child spans from ngx.ctx.tracing at header_filter time.

Environment

• APISIX 3.17.0 (behaviour also present on master at the time of writing)
• Official apache/apisix container image, Kubernetes
• apisix-ingress-controller 2.1.0, Gateway API
• OpenTelemetry Collector → ClickHouse (HyperDX / ClickStack)

[alice101-dev]

Screenshots of the two service maps (Clickstack). Same application, same collector — only
the gateway differs.

Fronted by APISIX — api-gateway is joined to the callers in front of it,
but not to orders-service behind it:

Fronted by Kong — one continuous chain, browser through to the backends:

[vynazevedo]

One implementation detail matters for the service-map result: the CLIENT span needs to exist before the upstream request is sent, and its context needs to be injected into that request. Adding a child span retrospectively would leave the upstream SERVER span parented to the gateway SERVER span, so the missing edge could remain.

In 3.17.0, the plugin injects the gateway span's context in rewrite. Also, inject_core_spans() is called from log in that tag, rather than header_filter.

For the normal NGINX proxy path, I'd start with this design:

• Create the CLIENT span in before_proxy, parented to the saved gateway SERVER context, and inject the CLIENT context before dispatch. The hook runs after upstream selection.
• Keep explicit handles for both spans. The current log handler uses context:current():span(); leaving a CLIENT context attached would otherwise make it update/finish the wrong span and attach the core spans under the wrong parent.
• Finish the CLIENT span exactly once, with a log fallback for errors/aborts. If you finish it at body-filter EOF, document that this measures the proxied response lifecycle, including downstream backpressure, rather than pure upstream processing time.

I'd test the exported parent IDs as well as span kinds: gateway SERVER → gateway CLIENT → upstream SERVER, plus a local rejection with no upstream call, a connection failure, and a retry. A single logical CLIENT span across retries is a different contract from one span per attempt; it would help to choose that explicitly. Plugins that bypass NGINX upstream handling also need separate consideration.

That's a code-reading suggestion, not a tested patch or a statement about whether maintainers will accept the change.