release(0.13.5): perf — cancellable flush sleep + CI hygiene - #60
Merged
Conversation
The Transport flush loop used `time.sleep(self.config.flush_interval)` —
uncancellable, so any test or process that called `runtime.shutdown()`
while the thread was mid-sleep blocked on `thread.join()` for the full
default 5s flush_interval. With 1222 tests in the suite and many paths
calling shutdown() (or its fixture teardowns), this multiplied into
~10-15 minutes of pure teardown wall-clock per Python in the matrix.
Replace the bare sleep with `Event.wait`, which returns the instant
`stop()` sets the event. `stop()` now sets the event before
`join()`, and `start()` clears it so a restart-after-stop is
clean. Pin contract in tests/test_transport.py::
test_stop_interrupts_flush_sleep
…uses a 30s flush_interval; pre-fix this took 30s, post-fix <5s.
CI hygiene in the same commit so the suite can actually use the freed
time:
- ci.yml / publish*.yml: enable pip cache (`cache: pip` +
`cache-dependency-path: pyproject.toml`) — saves ~60-90s of cold
install per matrix leg.
- ci.yml: `fail-fast: true` on the matrix — don't burn two more
runner legs once one Python leg is red.
- ci.yml / coverage / publish*.yml: install `pytest-xdist>=3.6` and
pass `-n auto` to pytest. `pytest-xdist` is also added to
`[project.optional-dependencies.dev]` so a local
`pip install -e .[dev]` brings it in.
- pyproject.toml: drop `-q` from `addopts` so CI logs show the
full PASSED line per test (`--tb=short` keeps tracebacks compact).
`-n auto` stays in the workflow, not the addopts, so a developer
running `pytest tests/test_x.py` gets a single process.
No public API change. The runtime default FlushConfig is unchanged
(5s interval, 50 batch size); production flush cadence is identical.
The fix only shortens the worst-case shutdown latency.
Pairs with the preceding release/0.13.5 commits: * perf(ci): cancel flush-thread sleep (transport.py:816) * remove redundant docs (drift.md, sdk-v3-migration-gaps.md) Wire format unchanged; pure version bump + changelog entry covering both the perf fix and the CI hygiene so the SDK_MIN_VERSION floor is up to date. No on-wire breaking change; backends on 1.0.0 keep working unchanged. Recommended upgrade path: 0.13.4 -> 0.13.5.
…ace respx PR #60 landed the cancellable-sleep fix in Transport._flush_loop and expected CI wall-clock to drop to 3-5 minutes. The first green run on PR #60 (PR #60 run #1) actually took 9m 47s — the test step dominated by a retry storm: Request failed (attempt 5/11), retrying in 8.46s: ConnectError Request failed (attempt 6/11), retrying in 9.16s: ConnectError ... Circuit breaker OPEN. Batch of 10 events will be re-queued. Root cause: `tests/conftest.py:reset_runtime` teardown nulled the runtime reference WITHOUT calling `runtime.shutdown()`. The transport flush thread therefore kept running across tests, the buffer drained through httpx with no respx context active, and the xdist workers spent the next 9 minutes retry-sending the buffer against the real (unreachable in CI) backend. `_retry_with_backoff (max_retries=10, max_delay=10s)` is 65s of pure sleep per failed batch, and with 4 xdist workers and many buffered batches this multiplied into 9m 47s — i.e. a CI-noise fix that hid a deeper lifecycle bug. Pre-fix CI was already paying this cost (5s shutdown-sleep × 200+ tests ≈ 17 min of teardown per Python leg); the retry storm was always there but masked by the dominant 5s cost. PR #60's 5s fix exposed it. Fix: add `flush: bool = True` to both `Transport.stop()` and `NullRunRuntime.shutdown()`. When False, the transport thread is cancelled WITHOUT a final `_do_flush()` / `_persist_to_wal()`. `tests/conftest.py:reset_runtime` teardown now calls `inst.shutdown(flush=False)` before nilling the reference. This makes the conftest teardown a true no-op for the buffer — the test that wrote the events is responsible for asserting on what it cared about. The production default (`flush=True`) is preserved, so the `nullrun.shutdown()` audit contract ("drain in-flight events") is unchanged. Pins: * `tests/test_transport.py::test_stop_flush_false_skips_final_flush ` — buffers an event, calls `stop(flush=False)` with no respx active, asserts the call returns in <1s AND the buffer is left untouched. Pre-fix this would have hung for 65s+ on the first retry. * `tests/test_init_contract.py::TestShutdownFlushKwarg:: test_runtime_shutdown_flush_false_skips_final_flush` — same contract at the `NullRunRuntime` level: `shutdown(flush=False )` propagates the `flush=False` flag to `Transport.stop()`. Public API additions: * `Transport.stop(timeout=10.0, flush: bool = True)` — `flush =False` is the new flag. * `NullRunRuntime.shutdown(flush: bool = True)` — propagates. * `nullrun.shutdown(timeout=2.0, flush: bool = True)` — passes `flush` through to the runtime. No on-wire or production behaviour change. CI step is expected to drop from ~9m 47s (PR #60 run #1) to ~30-60s on the next run.
Member
Author
Follow-up commit
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
maltsev-dev
added a commit
that referenced
this pull request
Jul 11, 2026
#61) * perf(ci): cancel flush-thread sleep so shutdown() returns in ms, not 5s The Transport flush loop used `time.sleep(self.config.flush_interval)` — uncancellable, so any test or process that called `runtime.shutdown()` while the thread was mid-sleep blocked on `thread.join()` for the full default 5s flush_interval. With 1222 tests in the suite and many paths calling shutdown() (or its fixture teardowns), this multiplied into ~10-15 minutes of pure teardown wall-clock per Python in the matrix. Replace the bare sleep with `Event.wait`, which returns the instant `stop()` sets the event. `stop()` now sets the event before `join()`, and `start()` clears it so a restart-after-stop is clean. Pin contract in tests/test_transport.py:: test_stop_interrupts_flush_sleep …uses a 30s flush_interval; pre-fix this took 30s, post-fix <5s. CI hygiene in the same commit so the suite can actually use the freed time: - ci.yml / publish*.yml: enable pip cache (`cache: pip` + `cache-dependency-path: pyproject.toml`) — saves ~60-90s of cold install per matrix leg. - ci.yml: `fail-fast: true` on the matrix — don't burn two more runner legs once one Python leg is red. - ci.yml / coverage / publish*.yml: install `pytest-xdist>=3.6` and pass `-n auto` to pytest. `pytest-xdist` is also added to `[project.optional-dependencies.dev]` so a local `pip install -e .[dev]` brings it in. - pyproject.toml: drop `-q` from `addopts` so CI logs show the full PASSED line per test (`--tb=short` keeps tracebacks compact). `-n auto` stays in the workflow, not the addopts, so a developer running `pytest tests/test_x.py` gets a single process. No public API change. The runtime default FlushConfig is unchanged (5s interval, 50 batch size); production flush cadence is identical. The fix only shortens the worst-case shutdown latency. * remove redundant docs * chore(release): bump version 0.13.4 -> 0.13.5 Pairs with the preceding release/0.13.5 commits: * perf(ci): cancel flush-thread sleep (transport.py:816) * remove redundant docs (drift.md, sdk-v3-migration-gaps.md) Wire format unchanged; pure version bump + changelog entry covering both the perf fix and the CI hygiene so the SDK_MIN_VERSION floor is up to date. No on-wire breaking change; backends on 1.0.0 keep working unchanged. Recommended upgrade path: 0.13.4 -> 0.13.5. * fix(tests): stop transport flush thread between tests so it doesn't race respx PR #60 landed the cancellable-sleep fix in Transport._flush_loop and expected CI wall-clock to drop to 3-5 minutes. The first green run on PR #60 (PR #60 run #1) actually took 9m 47s — the test step dominated by a retry storm: Request failed (attempt 5/11), retrying in 8.46s: ConnectError Request failed (attempt 6/11), retrying in 9.16s: ConnectError ... Circuit breaker OPEN. Batch of 10 events will be re-queued. Root cause: `tests/conftest.py:reset_runtime` teardown nulled the runtime reference WITHOUT calling `runtime.shutdown()`. The transport flush thread therefore kept running across tests, the buffer drained through httpx with no respx context active, and the xdist workers spent the next 9 minutes retry-sending the buffer against the real (unreachable in CI) backend. `_retry_with_backoff (max_retries=10, max_delay=10s)` is 65s of pure sleep per failed batch, and with 4 xdist workers and many buffered batches this multiplied into 9m 47s — i.e. a CI-noise fix that hid a deeper lifecycle bug. Pre-fix CI was already paying this cost (5s shutdown-sleep × 200+ tests ≈ 17 min of teardown per Python leg); the retry storm was always there but masked by the dominant 5s cost. PR #60's 5s fix exposed it. Fix: add `flush: bool = True` to both `Transport.stop()` and `NullRunRuntime.shutdown()`. When False, the transport thread is cancelled WITHOUT a final `_do_flush()` / `_persist_to_wal()`. `tests/conftest.py:reset_runtime` teardown now calls `inst.shutdown(flush=False)` before nilling the reference. This makes the conftest teardown a true no-op for the buffer — the test that wrote the events is responsible for asserting on what it cared about. The production default (`flush=True`) is preserved, so the `nullrun.shutdown()` audit contract ("drain in-flight events") is unchanged. Pins: * `tests/test_transport.py::test_stop_flush_false_skips_final_flush ` — buffers an event, calls `stop(flush=False)` with no respx active, asserts the call returns in <1s AND the buffer is left untouched. Pre-fix this would have hung for 65s+ on the first retry. * `tests/test_init_contract.py::TestShutdownFlushKwarg:: test_runtime_shutdown_flush_false_skips_final_flush` — same contract at the `NullRunRuntime` level: `shutdown(flush=False )` propagates the `flush=False` flag to `Transport.stop()`. Public API additions: * `Transport.stop(timeout=10.0, flush: bool = True)` — `flush =False` is the new flag. * `NullRunRuntime.shutdown(flush: bool = True)` — propagates. * `nullrun.shutdown(timeout=2.0, flush: bool = True)` — passes `flush` through to the runtime. No on-wire or production behaviour change. CI step is expected to drop from ~9m 47s (PR #60 run #1) to ~30-60s on the next run. * fix(langgraph): attach LLM spans to parent chain via callback run_id Sprint 2026-07-12 (multi-agent span attachment). Previously on_llm_end called runtime.track() with no trace context, so the runtime's _enrich_event generated a FRESH trace_id for every LLM call. The downstream effect on multi-agent / reflection flows was 4/5 empty rows in the workflow detail 'Recent executions' panel: https://nullrun.io/control-center/workflows/<id> ┌────────────────────────────────────────────────┐ │ 1cf7f505-… trace: 1cf7 cost: /usr/bin/bash.00 │ ← orchestration span only │ c4be95fe-… trace: c4be cost: /usr/bin/bash.00 │ ← orchestration span only │ 9295df0f-… trace: 9295 cost: /usr/bin/bash.00 │ ← orchestration span only │ 019f5060-… trace: 019f cost: $0.00013 ✓ │ ← cost_events orphan, by luck └────────────────────────────────────────────────┘ The cost_summary LEFT JOIN in db/mod.rs::get_execution_records_* keyed on cs.join_kind='trace_id' AND cs.join_id=u.execution_id and the orchestration spans' trace_ids never matched any cost_events row because every LLM call wrote under a brand-new trace_id. Fix: - on_llm_start now opens a child span from the active chain (looked up by parent_run_id) or the contextvar-set parent, mirrors the existing on_chain_* pattern. Stores the SpanContext under the LangChain run_id key. - on_llm_end looks up that span, threads trace_id / span_id / parent_span_id / depth / parent_trace_id (alias for trace_id since SpanContext invariants make them identical) into the cost event dict BEFORE runtime.track(). _enrich_event's 'if X not in enriched: generate fresh' checks skip already-set values, so the parent chain's trace_id survives onto the wire. - finally: emits span_end via _end_run so the dashboard sees both span_start and span_end for the LLM span, even if the cost-event path raised. Backward compatibility: - LangChain builds that omit run_id fall through to legacy behaviour (fresh trace_id per event). Tested by test_on_llm_without_run_id_is_silent_no_op. - Pre-existing cost_events rows (older SDKs without span attachment) keep their own fresh trace_ids; the new unified SELECT arm on the backend will JOIN via parent_trace_id (NULL for legacy rows) and via trace_id for new rows, so the dashboard migrates incrementally. Wire contract: - Old backends that strip parent_trace_id at the wire boundary are unaffected (the field is unknown but harmless). - New backends write it to cost_events.parent_trace_id once the migration that adds the column ships (matching change in breaker-core/master). Tests (test_langgraph_callback.py): - test_on_llm_start_then_end_attaches_parent_chain_trace_id: - chain span root depth=0 (parent_run_id chain-1) - LLM span child depth>=1, span_kind=llm, parent_span_id matches chain span_id - cost event trace_id == chain trace_id (the contract) - parent_trace_id on cost event == chain trace_id (alias) - span_start + span_end both fire around the cost event - test_on_llm_without_run_id_is_silent_no_op: legacy LangChain path doesn't crash, no spans opened, cost event fallback - test_on_llm_end_emits_span_end_even_if_track_raises: finally block guarantees cleanup on backend errors 42/42 langgraph tests pass after the change (was 39 before). * chore(release): 0.13.6 — multi-agent span attachment (parent_trace_id) Bump __version__ to 0.13.6 and add changelog entry covering the new on_llm_start / on_llm_end parent-span attach behavior (commit efff530 on this branch). No public API change. Wire format: backward-compatible. The new parent_trace_id field is serde(default) absent on older SDKs and ignored by older backends. Operators upgrading from 0.13.5 must upgrade both sides together (SDK to 0.13.6 + backend with migration 217); the SDK alone still works on 1.0.0 backends. Recommended upgrade path: 0.13.5 -> 0.13.6. SDK_MIN_VERSION_FOR_V3 unchanged (0.12.0).
maltsev-dev
added a commit
that referenced
this pull request
Aug 7, 2026
* perf(ci): cancel flush-thread sleep so shutdown() returns in ms, not 5s
The Transport flush loop used `time.sleep(self.config.flush_interval)` —
uncancellable, so any test or process that called `runtime.shutdown()`
while the thread was mid-sleep blocked on `thread.join()` for the full
default 5s flush_interval. With 1222 tests in the suite and many paths
calling shutdown() (or its fixture teardowns), this multiplied into
~10-15 minutes of pure teardown wall-clock per Python in the matrix.
Replace the bare sleep with `Event.wait`, which returns the instant
`stop()` sets the event. `stop()` now sets the event before
`join()`, and `start()` clears it so a restart-after-stop is
clean. Pin contract in tests/test_transport.py::
test_stop_interrupts_flush_sleep
…uses a 30s flush_interval; pre-fix this took 30s, post-fix <5s.
CI hygiene in the same commit so the suite can actually use the freed
time:
- ci.yml / publish*.yml: enable pip cache (`cache: pip` +
`cache-dependency-path: pyproject.toml`) — saves ~60-90s of cold
install per matrix leg.
- ci.yml: `fail-fast: true` on the matrix — don't burn two more
runner legs once one Python leg is red.
- ci.yml / coverage / publish*.yml: install `pytest-xdist>=3.6` and
pass `-n auto` to pytest. `pytest-xdist` is also added to
`[project.optional-dependencies.dev]` so a local
`pip install -e .[dev]` brings it in.
- pyproject.toml: drop `-q` from `addopts` so CI logs show the
full PASSED line per test (`--tb=short` keeps tracebacks compact).
`-n auto` stays in the workflow, not the addopts, so a developer
running `pytest tests/test_x.py` gets a single process.
No public API change. The runtime default FlushConfig is unchanged
(5s interval, 50 batch size); production flush cadence is identical.
The fix only shortens the worst-case shutdown latency.
* remove redundant docs
* chore(release): bump version 0.13.4 -> 0.13.5
Pairs with the preceding release/0.13.5 commits:
* perf(ci): cancel flush-thread sleep (transport.py:816)
* remove redundant docs (drift.md, sdk-v3-migration-gaps.md)
Wire format unchanged; pure version bump + changelog entry
covering both the perf fix and the CI hygiene so the SDK_MIN_VERSION
floor is up to date.
No on-wire breaking change; backends on 1.0.0 keep working
unchanged. Recommended upgrade path: 0.13.4 -> 0.13.5.
* fix(tests): stop transport flush thread between tests so it doesn't race respx
PR #60 landed the cancellable-sleep fix in Transport._flush_loop and
expected CI wall-clock to drop to 3-5 minutes. The first green run
on PR #60 (PR #60 run #1) actually took 9m 47s — the test step
dominated by a retry storm:
Request failed (attempt 5/11), retrying in 8.46s: ConnectError
Request failed (attempt 6/11), retrying in 9.16s: ConnectError
...
Circuit breaker OPEN. Batch of 10 events will be re-queued.
Root cause: `tests/conftest.py:reset_runtime` teardown nulled the
runtime reference WITHOUT calling `runtime.shutdown()`. The
transport flush thread therefore kept running across tests, the
buffer drained through httpx with no respx context active, and the
xdist workers spent the next 9 minutes retry-sending the buffer
against the real (unreachable in CI) backend. `_retry_with_backoff
(max_retries=10, max_delay=10s)` is 65s of pure sleep per failed
batch, and with 4 xdist workers and many buffered batches this
multiplied into 9m 47s — i.e. a CI-noise fix that hid a deeper
lifecycle bug.
Pre-fix CI was already paying this cost (5s shutdown-sleep × 200+
tests ≈ 17 min of teardown per Python leg); the retry storm was
always there but masked by the dominant 5s cost. PR #60's 5s fix
exposed it.
Fix: add `flush: bool = True` to both `Transport.stop()` and
`NullRunRuntime.shutdown()`. When False, the transport thread is
cancelled WITHOUT a final `_do_flush()` / `_persist_to_wal()`.
`tests/conftest.py:reset_runtime` teardown now calls
`inst.shutdown(flush=False)` before nilling the reference. This
makes the conftest teardown a true no-op for the buffer — the test
that wrote the events is responsible for asserting on what it
cared about. The production default (`flush=True`) is preserved,
so the `nullrun.shutdown()` audit contract ("drain in-flight
events") is unchanged.
Pins:
* `tests/test_transport.py::test_stop_flush_false_skips_final_flush
` — buffers an event, calls `stop(flush=False)` with no
respx active, asserts the call returns in <1s AND the buffer
is left untouched. Pre-fix this would have hung for 65s+ on
the first retry.
* `tests/test_init_contract.py::TestShutdownFlushKwarg::
test_runtime_shutdown_flush_false_skips_final_flush` — same
contract at the `NullRunRuntime` level: `shutdown(flush=False
)` propagates the `flush=False` flag to
`Transport.stop()`.
Public API additions:
* `Transport.stop(timeout=10.0, flush: bool = True)` — `flush
=False` is the new flag.
* `NullRunRuntime.shutdown(flush: bool = True)` — propagates.
* `nullrun.shutdown(timeout=2.0, flush: bool = True)` — passes
`flush` through to the runtime.
No on-wire or production behaviour change. CI step is expected to
drop from ~9m 47s (PR #60 run #1) to ~30-60s on the next run.
maltsev-dev
added a commit
that referenced
this pull request
Aug 7, 2026
#61) * perf(ci): cancel flush-thread sleep so shutdown() returns in ms, not 5s The Transport flush loop used `time.sleep(self.config.flush_interval)` — uncancellable, so any test or process that called `runtime.shutdown()` while the thread was mid-sleep blocked on `thread.join()` for the full default 5s flush_interval. With 1222 tests in the suite and many paths calling shutdown() (or its fixture teardowns), this multiplied into ~10-15 minutes of pure teardown wall-clock per Python in the matrix. Replace the bare sleep with `Event.wait`, which returns the instant `stop()` sets the event. `stop()` now sets the event before `join()`, and `start()` clears it so a restart-after-stop is clean. Pin contract in tests/test_transport.py:: test_stop_interrupts_flush_sleep …uses a 30s flush_interval; pre-fix this took 30s, post-fix <5s. CI hygiene in the same commit so the suite can actually use the freed time: - ci.yml / publish*.yml: enable pip cache (`cache: pip` + `cache-dependency-path: pyproject.toml`) — saves ~60-90s of cold install per matrix leg. - ci.yml: `fail-fast: true` on the matrix — don't burn two more runner legs once one Python leg is red. - ci.yml / coverage / publish*.yml: install `pytest-xdist>=3.6` and pass `-n auto` to pytest. `pytest-xdist` is also added to `[project.optional-dependencies.dev]` so a local `pip install -e .[dev]` brings it in. - pyproject.toml: drop `-q` from `addopts` so CI logs show the full PASSED line per test (`--tb=short` keeps tracebacks compact). `-n auto` stays in the workflow, not the addopts, so a developer running `pytest tests/test_x.py` gets a single process. No public API change. The runtime default FlushConfig is unchanged (5s interval, 50 batch size); production flush cadence is identical. The fix only shortens the worst-case shutdown latency. * remove redundant docs * chore(release): bump version 0.13.4 -> 0.13.5 Pairs with the preceding release/0.13.5 commits: * perf(ci): cancel flush-thread sleep (transport.py:816) * remove redundant docs (drift.md, sdk-v3-migration-gaps.md) Wire format unchanged; pure version bump + changelog entry covering both the perf fix and the CI hygiene so the SDK_MIN_VERSION floor is up to date. No on-wire breaking change; backends on 1.0.0 keep working unchanged. Recommended upgrade path: 0.13.4 -> 0.13.5. * fix(tests): stop transport flush thread between tests so it doesn't race respx PR #60 landed the cancellable-sleep fix in Transport._flush_loop and expected CI wall-clock to drop to 3-5 minutes. The first green run on PR #60 (PR #60 run #1) actually took 9m 47s — the test step dominated by a retry storm: Request failed (attempt 5/11), retrying in 8.46s: ConnectError Request failed (attempt 6/11), retrying in 9.16s: ConnectError ... Circuit breaker OPEN. Batch of 10 events will be re-queued. Root cause: `tests/conftest.py:reset_runtime` teardown nulled the runtime reference WITHOUT calling `runtime.shutdown()`. The transport flush thread therefore kept running across tests, the buffer drained through httpx with no respx context active, and the xdist workers spent the next 9 minutes retry-sending the buffer against the real (unreachable in CI) backend. `_retry_with_backoff (max_retries=10, max_delay=10s)` is 65s of pure sleep per failed batch, and with 4 xdist workers and many buffered batches this multiplied into 9m 47s — i.e. a CI-noise fix that hid a deeper lifecycle bug. Pre-fix CI was already paying this cost (5s shutdown-sleep × 200+ tests ≈ 17 min of teardown per Python leg); the retry storm was always there but masked by the dominant 5s cost. PR #60's 5s fix exposed it. Fix: add `flush: bool = True` to both `Transport.stop()` and `NullRunRuntime.shutdown()`. When False, the transport thread is cancelled WITHOUT a final `_do_flush()` / `_persist_to_wal()`. `tests/conftest.py:reset_runtime` teardown now calls `inst.shutdown(flush=False)` before nilling the reference. This makes the conftest teardown a true no-op for the buffer — the test that wrote the events is responsible for asserting on what it cared about. The production default (`flush=True`) is preserved, so the `nullrun.shutdown()` audit contract ("drain in-flight events") is unchanged. Pins: * `tests/test_transport.py::test_stop_flush_false_skips_final_flush ` — buffers an event, calls `stop(flush=False)` with no respx active, asserts the call returns in <1s AND the buffer is left untouched. Pre-fix this would have hung for 65s+ on the first retry. * `tests/test_init_contract.py::TestShutdownFlushKwarg:: test_runtime_shutdown_flush_false_skips_final_flush` — same contract at the `NullRunRuntime` level: `shutdown(flush=False )` propagates the `flush=False` flag to `Transport.stop()`. Public API additions: * `Transport.stop(timeout=10.0, flush: bool = True)` — `flush =False` is the new flag. * `NullRunRuntime.shutdown(flush: bool = True)` — propagates. * `nullrun.shutdown(timeout=2.0, flush: bool = True)` — passes `flush` through to the runtime. No on-wire or production behaviour change. CI step is expected to drop from ~9m 47s (PR #60 run #1) to ~30-60s on the next run. * fix(langgraph): attach LLM spans to parent chain via callback run_id Sprint 2026-07-12 (multi-agent span attachment). Previously on_llm_end called runtime.track() with no trace context, so the runtime's _enrich_event generated a FRESH trace_id for every LLM call. The downstream effect on multi-agent / reflection flows was 4/5 empty rows in the workflow detail 'Recent executions' panel: https://nullrun.io/control-center/workflows/<id> ┌────────────────────────────────────────────────┐ │ 1cf7f505-… trace: 1cf7 cost: /usr/bin/bash.00 │ ← orchestration span only │ c4be95fe-… trace: c4be cost: /usr/bin/bash.00 │ ← orchestration span only │ 9295df0f-… trace: 9295 cost: /usr/bin/bash.00 │ ← orchestration span only │ 019f5060-… trace: 019f cost: $0.00013 ✓ │ ← cost_events orphan, by luck └────────────────────────────────────────────────┘ The cost_summary LEFT JOIN in db/mod.rs::get_execution_records_* keyed on cs.join_kind='trace_id' AND cs.join_id=u.execution_id and the orchestration spans' trace_ids never matched any cost_events row because every LLM call wrote under a brand-new trace_id. Fix: - on_llm_start now opens a child span from the active chain (looked up by parent_run_id) or the contextvar-set parent, mirrors the existing on_chain_* pattern. Stores the SpanContext under the LangChain run_id key. - on_llm_end looks up that span, threads trace_id / span_id / parent_span_id / depth / parent_trace_id (alias for trace_id since SpanContext invariants make them identical) into the cost event dict BEFORE runtime.track(). _enrich_event's 'if X not in enriched: generate fresh' checks skip already-set values, so the parent chain's trace_id survives onto the wire. - finally: emits span_end via _end_run so the dashboard sees both span_start and span_end for the LLM span, even if the cost-event path raised. Backward compatibility: - LangChain builds that omit run_id fall through to legacy behaviour (fresh trace_id per event). Tested by test_on_llm_without_run_id_is_silent_no_op. - Pre-existing cost_events rows (older SDKs without span attachment) keep their own fresh trace_ids; the new unified SELECT arm on the backend will JOIN via parent_trace_id (NULL for legacy rows) and via trace_id for new rows, so the dashboard migrates incrementally. Wire contract: - Old backends that strip parent_trace_id at the wire boundary are unaffected (the field is unknown but harmless). - New backends write it to cost_events.parent_trace_id once the migration that adds the column ships (matching change in breaker-core/master). Tests (test_langgraph_callback.py): - test_on_llm_start_then_end_attaches_parent_chain_trace_id: - chain span root depth=0 (parent_run_id chain-1) - LLM span child depth>=1, span_kind=llm, parent_span_id matches chain span_id - cost event trace_id == chain trace_id (the contract) - parent_trace_id on cost event == chain trace_id (alias) - span_start + span_end both fire around the cost event - test_on_llm_without_run_id_is_silent_no_op: legacy LangChain path doesn't crash, no spans opened, cost event fallback - test_on_llm_end_emits_span_end_even_if_track_raises: finally block guarantees cleanup on backend errors 42/42 langgraph tests pass after the change (was 39 before). * chore(release): 0.13.6 — multi-agent span attachment (parent_trace_id) Bump __version__ to 0.13.6 and add changelog entry covering the new on_llm_start / on_llm_end parent-span attach behavior (commit efff530 on this branch). No public API change. Wire format: backward-compatible. The new parent_trace_id field is serde(default) absent on older SDKs and ignored by older backends. Operators upgrading from 0.13.5 must upgrade both sides together (SDK to 0.13.6 + backend with migration 217); the SDK alone still works on 1.0.0 backends. Recommended upgrade path: 0.13.5 -> 0.13.6. SDK_MIN_VERSION_FOR_V3 unchanged (0.12.0).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
release(0.13.5)
This is the 0.13.5 perf release — pairs the LangChain elif-chain
fix from 0.13.4 with a CI/runtime perf pass that the audit revealed.
What's in this PR
Three commits on top of
origin/master(note: the 0.13.4elif-chain fix already landed on master as PR #59, so the matching
c17a582version-bump commit is intentionally not cherry-picked — its changelog text was already on master and would have
been a no-op):
6ec7671— perf(ci):Transport._flush_loopswappedtime.sleepforthreading.Event.waitsoruntime.shutdown()returns in ms, not the fullflush_interval(5s default). Pin:tests/test_transport.py::test_stop_interrupts_flush_sleepuses 30s
flush_intervaland assertsstop() < 5s.84fc9d5— docs: removedocs/drift.mdanddocs/sdk-v3-migration-gaps.md(drift content already inCHANGELOG.mdand the per-release__version__.pyheaders).99bc197— chore(release): bump 0.13.4 → 0.13.5(
pyproject.toml+src/nullrun/__version__.py) with a fullchangelog block describing the perf fix + CI hygiene.
CI hygiene (in the perf commit)
setup-pythonaction:cache: pip+cache-dependency-path: pyproject.toml— saves ~60-90s ofcold install per matrix leg.
strategy.fail-fast: trueon the test matrix.pip install "pytest-xdist>=3.6"+pytest -n autoso thesuite uses all runner cores.
xdistalso added to[project.optional-dependencies.dev]for local parity.pyproject.toml: dropped the global-qfromaddoptssoCI logs surface the full
PASSEDline per test;--tb=shortkeeps tracebacks compact.-n autostays inthe workflow (not
addopts) so a developer runningpytest tests/test_x.pylocally gets a single process.Wire format
No change. The default
FlushConfigis unchanged (5sflush_interval, 50batch_size); production flush cadence isidentical. The fix only shortens the worst-case shutdown latency.
Compatibility
Backends on 1.0.0 keep working unchanged. No
SDK_MIN_VERSIONbump. Recommended upgrade path: 0.13.4 → 0.13.5.
Validation
test_stop_interrupts_flush_sleeppasses in 0.26s (was 30s).test_legacy_key_warning— 0.19s (was 5.18s).test_remote_states_race×4 — no 5.0s teardowns.test_state_compare_case_insensitive×10 — all <0.2s.ruff checkclean on changed files.mypyclean ontransport.py.Branch / commit policy
release/0.13.5cut fromorigin/master(notfrom a prior
release/*, per repo convention).Co-Authored-Bytrailers.user.name/user.emailfrom local git config.