fix(observability): make Phoenix OTLP tracing work for programmatic sourcehunt runs - #243
Merged
Merged
Conversation
Phoenix authenticates OTLP trace ingestion with an Authorization: Bearer header. Clearwing sent PHOENIX_API_KEY as an "api_key" header, which Phoenix rejects with 401. The BatchSpanProcessor swallows the export failure, so traces were dropped silently with no error surfaced. Send "Authorization: Bearer <key>" instead. Standard OTEL_EXPORTER_OTLP header variables still take precedence when set.
…t runs SourceHuntRunner.run()/arun() are reached directly by the eval harness, the sourcehunt agent tool, notebooks and per-repo campaign runs, all of which bypass the CLI and web entrypoints that call ObservabilityIntegration.bootstrap_from_env(). Without a bootstrap the process-wide tracer provider stays the no-op proxy, so instrumented LLM spans are silently dropped and never reach Phoenix. Bootstrap from env at the top of arun() -- the single async entry every run funnels through -- and force_flush() at the run() boundary so each run's spans are exported before the caller exits or advances to the next repo. Both are no-ops when OTLP export is not configured and bootstrap is idempotent, so CLI/web behavior is unchanged.
jorge-garcia-le
marked this pull request as ready for review
September 23, 2026 18:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phoenix/OTLP traces were not appearing for sourcehunt runs. Two independent bugs were responsible; this branch fixes both.
1. OTLP ingestion was authenticated with the wrong header
Phoenix authenticates OTLP over
Authorization: Bearer <token>. Clearwing sent anapi_keyheader, which Phoenix rejects with401. Because export runs underBatchSpanProcessor, the failure is swallowed — no crash, no log, spans silently dropped.otel.py: whenPHOENIX_API_KEYis set (and no standard OTLP headers already override), sendauthorization: Bearer <key>instead ofapi_key.2. Programmatic runs never bootstrapped tracing
OTLP tracing is wired up in
clearwing:main()(CLI) andcreate_app()(web). Callers that driveSourceHuntRunnerin-process bypass both entrypoints, so the global provider stayed the no-opProxyTracerProviderand every span was discarded before auth even mattered.runner.py:arun()(the single async entry all runs funnel through, including the syncrun()wrapper and direct.arun()callers) now callsObservabilityIntegration.bootstrap_from_env()as its first statement. It is idempotent (process-wide singleton) and a no-op unless OTLP export is configured, so it stays safe under the CLI/web paths that already bootstrap.runner.py:run()callsforce_flush()in afinallyso a top-level invocation drains its spans before returning to the caller (e.g. before a campaign advances to the next repo), without disconnecting the shared provider.Scope / behavior
clearwing sourcehunt ...) and the--machine-fdsubprocess path already bootstrapped viamain(); they are unaffected except for the auth fix and an eager per-run flush.Testing
tests/test_otel.pyupdated for the Bearer header.