opentelemetry plugin: uncaught Lua error on worker shutdown via vendored opentelemetry-lua batch_span_processor

[yeganeahmadnejad]

Description

The vendored opentelemetry-lua dependency (pinned opentelemetry-lua = 0.2-6 in apisix-master-0.rockspec) has a bug in batch_span_processor.lua's create_timer(): when ngx.timer.at() fails because the worker is exiting -- expected near the end of every graceful shutdown/deploy -- it falls back to a synchronous span export (self:flush_all(), opening a cosocket).

Since spans finish inline inside whatever phase the opentelemetry plugin is running (body_filter_by_lua* or log_by_lua* for HTTP spans -- see apisix/plugins/opentelemetry.lua's phase_func), that synchronous cosocket call hits APISIX's own phase guard in apisix/patch.lua and crashes:

2026/09/20 09:31:05 [error] 70#70: *1277095 failed to run log_by_lua*: /usr/local/apisix/apisix/patch.lua:406: API disabled in the context of log_by_lua*

Full traceback (captured on 3.18.0):

batch_span_processor.lua:97: create_timer(): failed to create timer: process exiting while logging request
stack traceback:
	[C]: in function 'error'
	.../resty/core/socket/tcp.lua:298: in function 'original_tcp'
	.../apisix/patch.lua:406: in function 'ngx_socket_tcp'
	.../resty/http.lua:227: in function 'new'
	.../opentelemetry/trace/exporter/http_client.lua:36: in function 'do_request'
	.../opentelemetry/trace/exporter/otlp.lua:58: in function 'export_spans'
	.../opentelemetry/trace/batch_span_processor.lua:45: in function 'process_batches'
	.../opentelemetry/trace/batch_span_processor.lua:215: in function 'flush_all'
	.../opentelemetry/trace/batch_span_processor.lua:99: in function 'create_timer'
	.../opentelemetry/trace/batch_span_processor.lua:183: in function 'on_end'
	.../opentelemetry/trace/recording_span.lua:98: in function 'finish'
	/usr/local/apisix/apisix/plugins/opentelemetry.lua:559: in function 'phase_func'
	/usr/local/apisix/apisix/plugin.lua:1442: in function 'common_phase'
	/usr/local/apisix/apisix/init.lua:1213: in function 'http_log_phase'

Steps to reproduce

1. Enable the opentelemetry plugin with tracing: true (or the default per-route tracing) on any route -- including one with no other plugins, e.g. just a health-check route.
2. Deploy/restart the worker (trigger a graceful shutdown) while traffic is hitting that route.
3. Near the end of the outgoing worker's shutdown, the last span(s) to finish will hit the race and crash with the above error.

Because this only requires the worker to be exiting -- not any specific plugin config -- it fires ~consistently on every rolling deploy of any release with tracing enabled.

Root cause & fix

Filed and fixed upstream in opentelemetry-lua: yangxikun/opentelemetry-lua#107, fix at yangxikun/opentelemetry-lua#106. Once merged/released there, bumping the pinned opentelemetry-lua version here would pick up the fix. In the meantime, APISIX could consider wrapping phase_func's span-finishing call in a pcall as defense in depth, since this class of "third-party tracer code doing something disallowed in a restricted phase" isn't fully preventable from the library side alone.

Environment

• APISIX 3.18.0 (Ubuntu-based image), opentelemetry-lua = 0.2-6
• OpenResty/CentOS or Ubuntu base, not relevant to the phase-restriction behavior (standard OpenResty body_filter_by_lua*/log_by_lua* cosocket restriction)