Skip to content

Latest commit

 

History

History
2497 lines (2231 loc) · 274 KB

metrics.md

File metadata and controls

2497 lines (2231 loc) · 274 KB

Metrics Telemetry Module

Overview

Local-first metrics telemetry built on the OpenTelemetry SDK (Apache-2.0 / CNCF). The trunk is designed so future work is purely adding instrument calls at call sites — never changing this plumbing. Default OFF (telemetry.enabled: false): all metric call sites are cheap no-ops and nothing is written or exported, byte-identical to no telemetry (mirrors the mcp_gateway.enabled / skills.lazy_load opt-in convention). Any instrument that needs durable state to survive a process death — currently only the session-lifetime breadcrumb — must gate that state on the same consent, and fail closed when consent cannot be read, so this sentence stays literally true.

Source: the src/kiro_crew/metrics/ package. The tables below cover the modules that carry contract meaning; the package holds more (per-domain instrument modules for sessions, turns, tool calls, DB, process and inventory gauges, plus temporality.py and events.py), so read the directory rather than treating any list here as the inventory. Tests: test/metrics/.

Components

File Purpose
schema.py Namespace constants (NS_CORE = "kirocrew.", NS_GENAI = "gen_ai.", NS_APP_PREFIX = "app.") + validate_name / validate_attrs / redact guardrails. Documents the low-cardinality contract.
recorder.py MetricsRecorder — facade over the OTEL Meter. Every metric passes namespace + privacy guardrails BEFORE reaching an instrument. Instrument-cache creation is lock-guarded (atomic check-then-create). Best-effort: a telemetry failure never propagates to the caller. meter=None = no-op recorder.
provider.py Consent gate + process-global recorder (get_recorder()) + graceful shutdown() / reset_for_testing(). get_recorder() serves a memoized recorder and re-resolves the telemetry.enabled consent value every _CONSENT_RECHECK_SECS (30s), rebuilding when it moved — see "Recorder lifecycle & threading" below. Public consent surface: env_pin() / TELEMETRY_ENV_VAR. When enabled, wires a PeriodicExportingMetricReader to the local JSONL exporter. Installs one View per instrument from histogram_bounds() (_HISTOGRAM_BUCKETS_MS for millisecond instruments plus _HISTOGRAM_BUCKETS_BY_UNIT for the non-ms ones), each with its own ExplicitBucketHistogramAggregation boundaries (see below) — deliberately NOT a catch-all instrument_type=Histogram View.
local_exporter.py JsonlMetricExporter — appends one JSON line per export cycle to <dir>/metrics-YYYY-MM-DD-<pid>.jsonl (default dir ~/.kiro/crew/metrics). Per-PID single-writer shards keep append + rotation lock-free, so concurrent exporters do not lose DELTA cycles. A private .metrics.lock serializes only retention sweeps; pruning skips canonical shards owned by live PIDs or modified within the safety window. Bounded retention (rec #14): shards rotate before an append exceeds max_total_mb; closed/expired shards are pruned directly by age and oldest-first size. Pruning is throttled to at most once per 300s and fully best-effort. Dir mode is 0o700, file mode 0o600, and nothing egresses the host. Declares DELTA preferred_temporality for Counter/UpDownCounter/Histogram so daily aggregation is an element-wise sum across cycles/PIDs; the map itself lives in metrics/temporality.py, which owns it for the OTLP leg too (declaring it inline here is what let that leg ship CUMULATIVE for the same instruments with nothing failing). Observable counters are deliberately NOT mapped and would export CUMULATIVE: the delta baseline lives in the provider, which is rebuilt in-process on a telemetry consent change, so DELTA would re-emit the process-lifetime total once per rebuild. Nothing registers an observable counter today — the process CPU/GC readings and the inventory probe-failure count are lifetime-total GAUGES, which is what removed the last cumulative series — so the entry guards the next one; the aggregator still reduces cumulative streams window-relative (deterministic identity boundary + time-ordered legacy reset detection + first-in-window baseline) for the shards written before that switch. Process identity: each record is stamped once at resource level with kirocrew.process.start_time (schema.RESOURCE_ATTR_PROCESS_START_TIME) — the writing process's OS start-time token from platform_compat.own_process_start_time(), module-cached so provider rebuilds inside one process stamp the SAME value, and reboot-unique (Linux start ticks + boot UUID; macOS microsecond proc_pidinfo instant; Windows creation FILETIME). A read that cannot honor one-token-one-process (unreadable boot UUID, no libproc, 1s-only sources) emits NO token rather than an aliasable coarse one — a degraded token would merge lifetimes AND mute the reset heuristic that catches merges. The shard-filename PID plus this token identify a process beyond PID reuse, making the aggregator's cumulative reset detection deterministic. The stamp lands on the serialized JSONL line, never on the SDK Resource — that Resource also feeds the opt-in OTLP reader, and this host-local token must not egress. Fail-soft: when the platform read is unavailable the field is absent and the aggregator's legacy value heuristic applies. Resource level, not a metric attribute, so it never multiplies series cardinality.
temporality.py Single owner of the instrument-kind -> temporality map, for the local sink AND the opt-in OTLP leg. delta_preference() returns a fresh DELTA map for Counter/UpDownCounter/Histogram (ObservableCounter deliberately absent, gauges never applicable); otlp_preference() returns it, or None to stand aside when OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE is set. Exists because the map lived inline in local_exporter.__init__ while _build_otlp_reader passed nothing, so the two destinations reported the same instruments DELTA on disk and CUMULATIVE on the wire with nothing failing.
http_metrics.py Gateway HTTP observability (rec #1): record_boot_to_ready() (boot-to-ready histogram) + make_route_latency_middleware() (per-route latency, wired as the outermost middleware on both start_dashboard/start_api_server). Bounds route_template cardinality via collect_route_templates() (build-time snapshot) + route_template() (__unknown__ fallback); clamps method to a fixed allowlist and status_class to 1xx..5xx/other. Upgraded WebSocket connections and text/event-stream SSE responses are excluded because their handler elapsed time is connection/turn lifetime, not HTTP request latency. Best-effort — a telemetry failure never alters a response.

Recorder lifecycle & threading

get_recorder() returns a memoized recorder on a fast path guarded by a monotonic clock (_consent_recheck_due): once a recorder exists it is handed back directly until the recheck window elapses. Every _CONSENT_RECHECK_SECS (30s) the call hands a consent check to a worker (_schedule_consent_check_locked -> _consent_worker), which re-reads telemetry.enabled and rebuilds when it moved. That is what makes kirocrew config set telemetry.enabled true — a write from a SEPARATE process — take effect without a gateway restart. Inside the gateway the wait is skipped by the config watcher: watch_config(), registered at gateway boot, subscribes _on_config_change to the telemetry section, and any telemetry.* write awaits asyncio.to_thread(shutdown) so the next metric rebuilds from the new values. A config that cannot be READ yields "no change" rather than False, so a transient read error never tears down a working recorder; the worker stamps the recheck clock either way, so an unreadable config cannot turn every metric call into a fresh file read.

get_recorder() itself never reads config and never builds anything, apart from the very first build of the process, which has nothing to serve in the meantime. KiroCrewConfig.load() is a fingerprint-cache hit in the steady state (~0.3ms) but a full read plus schema validation when the file actually changed (~14ms), and the rebuild costs ~57ms of SDK import — neither belongs on the event loop, which the route-latency middleware drives on every HTTP request. The consequence to know: the recheck is eventual in BOTH directions. A change is noticed at the window boundary and lands a thread hop later, which is immaterial against the window it already sits behind. _check_in_flight keeps a busy window from spawning one worker per request, and is cleared in the worker's finally so a crash costs one window rather than stranding the check.

Consent resolution is env-first: env_pin() reads TELEMETRY_ENV_VAR (KIROCREW_TELEMETRY) and, when set, decides the effective state regardless of the config flag; _consent_enabled() falls back to telemetry.enabled.

_lock is never held across a provider or reader shutdown. _take_provider_locked() clears the globals under the lock and RETURNS the provider; the flush happens after release. A provider shutdown joins each reader's export thread (30s deadline) and, with telemetry.otlp_endpoint set, ends in a synchronous network POST — holding the lock across it would stall get_recorder() on lock ACQUISITION, and the route-latency middleware calls get_recorder() on the event loop for every HTTP request. This applies to every path that reaches a teardown, including the one that discards a superseded build.

Which thread flushes depends on who dropped the recorder. The consent worker flushes on its own thread, after releasing the lock. shutdown() flushes on the CALLER's thread, so process teardown and the config-watcher applier (which calls it via asyncio.to_thread) both observe completion — still not under the lock. Partially-constructed readers from a failed init are reaped off-thread the same way (_reap_readers_detached), because a reader shutdown performs a final export.

_build_recorder() writes no module state. It returns a _Build tuple (recorder, provider, resolved consent) that a caller installs under the lock via _install_locked(), so only the install is a critical section. Consent is recorded alongside the recorder even on a host that can never record (OTel absent), because leaving it unset would make every recheck window see a difference and rebuild a no-op recorder in a loop.

A _build_generation counter makes a mid-build disable stick. Each rebuild captures the counter and rechecks it before installing; _take_provider_locked() bumps it, so a disable that lands while a build is in flight is not undone by the build finishing afterwards — the superseded build's provider is flushed (outside the lock) and discarded instead. Every consent change starts its own worker rather than skipping when one is already running: skipping would leave the recorded consent updated while the recorder stayed a no-op, and the next recheck would then find no difference and never retry.

Only the FIRST build of a process is synchronous (_ever_built is still False), because there is no recorder to serve in the meantime and that path is not a steady-state request path. Every later build is a REbuild and goes off-thread, including one after shutdown() cleared the state — otherwise the config route's own write would put the SDK import back on the event loop. reset_for_testing() clears _ever_built, so a test's next build is synchronous and assertable without polling. shutdown() bumps the generation but does not stop an already-running consent worker, so reset_for_testing() also waits (bounded, _RESET_WAIT_BOUND_SECS) for _check_in_flight to clear before returning — raising if the bound expires — so a worker left running by an earlier test can never mutate module state underneath the next one.

One consequence of the detached flush: while it is still in its join, a re-enable can put a second exporter on the same per-PID shard, which the local exporter's single-writer assumption does not cover. Both sides swallow their IO errors, so the worst case is one dropped export cycle rather than a corrupt shard.

Guardrails (contract C4)

  • Namespace: core callers must use kirocrew.* or gen_ai.*; app callers must use app.<app_id>.* and cannot spoof the core/gen_ai namespaces (validate_name raises ValueError, the recorder swallows it, nothing is recorded).
  • Privacy: string attribute values pass redact() — AKIA/ASIA keys, SecretAccessKey=, private-key headers, 40+ char hex, JWT shapes, password=/token= patterns, base64-encoded credential variants, and a Shannon-entropy heuristic all yield "[REDACTED]". The first-party kiro_crew.security scrubbers (redact_credentials, redact_exfiltration_urls — both return (cleaned, warnings) tuples) are also consulted. Long non-suspicious strings are truncated to MAX_ATTR_VALUE_LEN (128).
  • Cardinality: metric names + attribute values must be low-cardinality constants; attribute count is capped at MAX_ATTR_COUNT (32). Instrument caches are keyed by name and never evicted.

Configuration

TelemetryConfig in config/sections_telemetry.py (parsed by config/loader_builders.py; section telemetry in ~/.kiro/crew/config.json):

Field Default Meaning
enabled false Main switch. Off = no-op recorder, nothing written. Editable from the dashboard (Settings → Privacy) as well as the config file, kirocrew config set, and the env var; re-resolved live, so a change takes effect without a restart.
local_dir "" JSONL shard dir; empty = ~/.kiro/crew/metrics. ~ expansion supported.
export_interval_seconds 60 Flush interval (floored to 1).
retention_days 0 Age pruning is disabled by default to preserve pre-existing history on upgrade. Set a positive day window to opt in (rec #14).
max_total_mb 0 Size pruning is disabled by default to preserve pre-existing history on upgrade. Set a positive opportunistic directory budget to opt in; protected active writers can temporarily exceed it (rec #14).
otlp_endpoint "" Opt-in OTLP/HTTP metrics endpoint (e.g. http://localhost:4318/v1/metrics). Empty = no network egress (default). When set, aggregated metrics are ALSO pushed to this collector in addition to the local JSONL sink; requires pip install "opentelemetry-exporter-otlp-proto-http==1.44.0" (rec #1). Metrics signal only: no spans or log records exist, so _build_otlp_reader logs a WARNING (destination name only, never the URL) when the endpoint path ends in /v1/traces or /v1/logs, and still builds the reader.

Field validation (TelemetryConfig.__post_init__): export_interval_seconds below 1 is floored to 1; negative retention_days / max_total_mb are clamped to 0 (cap disabled) rather than being interpreted as "prune everything".

Opt-in, retention bounds & egress (rec #14 / rec #1)

Default posture — nothing collected, nothing leaves the host. telemetry.enabled defaults false, so every metric call site is a cheap no-op and no file is written — including the session-lifetime breadcrumb, which is gated on the same consent and fails closed. Even once local collection is enabled, otlp_endpoint defaults empty and otlp_endpoints defaults to an empty list, so no data ever leaves the machine unless the operator explicitly sets an OTLP endpoint (or the build ships a collector of its own).

Easy opt-in (four equivalent ways):

  • Config flag: set "telemetry": {"enabled": true} in ~/.kiro/crew/config.json.
  • CLI: kirocrew config set telemetry.enabled true.
  • Dashboard: the recording switch in Settings → Privacy, which writes the same key through PATCH /api/config/kirocrew (telemetry.enabled is in _EDITABLE_CONFIG). That route refuses true with HTTP 409 when any OTLP destination resolves (telemetry.otlp_endpoint, a telemetry.otlp_endpoints entry, or one the build ships): _build_recorder attaches an OTLP reader whenever the active telemetry provider supplies a destination — which the public default does for any non-empty endpoint — so enabling from a switch offered as local-only would start network egress. The endpoint is chosen in the config file, so that is where enabling on such a host is done. Disabling is always allowed — a narrower local choice always composes. An unreadable config also fails closed with a 409. A successful write reaches the recorder through the config-watcher applier (watch_config -> _on_config_change -> asyncio.to_thread(shutdown)), not through the route itself, so the value applies on the very next metric rather than at the next recheck.
  • Env var: export KIROCREW_TELEMETRY=1 (also accepts true/yes/on; 0/false/no/off force-disables). The env var overrides the config flag and is handy for CI / containers / one-off debugging. It gates local collection only — it never enables network egress. Resolved by the public provider.env_pin() and consumed by provider._consent_enabled().

None of these requires a gateway restart: get_recorder() re-resolves consent every _CONSENT_RECHECK_SECS (30s) and rebuilds when it moved (see "Recorder lifecycle & threading"). The gateway process is where the session/turn/HTTP metrics are recorded; other kirocrew processes pick the value up on their own recheck or at their next start.

Any telemetry.* change rebuilds the recorder, not just enabled. provider.watch_config() registers one applier on the telemetry section, and its callback calls shutdown() — the next get_recorder() then builds a fresh recorder from the new config. That covers the settings the 30s consent recheck cannot see: the export cadence and every other value _build_recorder reads once. The recheck itself compares three things against the live recorder: enabled, the agent pair the harness label comes from (_agent_key), and a sha256 of telemetry.otlp_endpoint plus telemetry.otlp_endpoints (_endpoints_key). So a process with no config watcher (the MCP gateway daemon, through recheck_consent) still drops the reader of a list entry that was turned off or removed, within one window. Registration is idempotent per process, so a re-entered boot path cannot stack appliers that each rebuild. The rebuild is lazy by design — shutdown() only drops the memo, so the ~57ms of SDK import is paid on the next metric call rather than on the config watcher's own thread.

Beacon fields are the exception: they are read once for the boot heartbeat, so a change to them applies at the next start.

External OTLP egress (opt-in, off by default): egress destinations are pluggable. _build_recorder asks the active TelemetryProvider for them via otlp_destinations(cfg) and attaches one PeriodicExportingMetricReader per returned destination that names the "metrics" signal (provider._otlp_destinations resolves, provider._build_otlp_reader builds one reader). Each destination's host must also pass the security policy's capabilities.telemetry otlp_hosts list (otlp_governance.governed_destinations before the reader is built and otlp_governance.gate_exporter before each export; see governance.md), and a policy that pins capabilities.telemetry off drops every destination. The public DefaultTelemetryProvider turns a non-empty otlp_endpoint into exactly one destination and returns none when it is empty, so a standalone build behaves as it did when this module constructed that exporter itself. Readers are appended AFTER the local JSONL sink, so an edition can add destinations but can never remove or replace the sink the dashboard reads, and it supplies no cadence — the export interval stays telemetry.export_interval_seconds. Nor does it supply a TEMPORALITY: _build_otlp_reader passes the same DELTA map the local sink uses (metrics.temporality.delta_preference, the single owner for both sinks), so one MeterProvider's two destinations cannot describe the same instrument two different ways. It passes NOTHING when OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE is set, leaving that variable in full control: the exporter applies an explicit dict ON TOP of whichever base the variable chose (OTLPMetricExporterMixin._get_temporality in the otlp-proto-common package — the reader lives there, not in the SDK tree or the http exporter tree), so passing the map unconditionally would silently override an operator who asked for CUMULATIVE.

A destination carries name (a non-secret label used in logs, because the endpoint value never is), endpoint, signals, and optionally an authenticated session. The session is why the seam exists: requests re-evaluates Session.auth per request, so a credential that rotates during process lifetime (an OIDC/SSO id_token, STS credentials) is re-read on every export, where OTEL_EXPORTER_OTLP_HEADERS — the only injection point before this seam — freezes into the exporter's session at construction and starts returning 401 once it rotates. A destination built in code carries static headers on Session.headers; one declared as data names them in headers_env (below).

The method must be cheap and side-effect-free per call: it is read once per recorder build AND on every egress-posture read (the Privacy panel's status, and each telemetry.enabled config write), so an edition must build its transport once rather than acquiring a credential inside it.

Filtering is deny-by-default: a destination with an empty endpoint, or one aimed at a signal this core does not emit, is dropped rather than trusted. A provider that raises contributes nothing and is reported at WARNING, and telemetry keeps working local-only; the same is true when the platform context cannot be composed, because for an egress seam "no destinations" is the closed state.

The MCP gateway daemon resolves its provider without booting. gatewayd never runs boot_platform (it is not the composition process; see app_call.py), so on a non-standalone profile current_context() raises there, and a recorder that resolves destinations through it builds local-only (its log says platform context unavailable; OTLP egress disabled (local-only)). gatewayd._prepare_telemetry (from main()) builds the recorder before the event loop starts, composing nothing: a standalone host resolves the lazy default context, so its telemetry.otlp_endpoint is live from the first build, and the pool gauges exist from startup. A consenting host then composes ONLY the edition's telemetry provider on a background thread through platform.bootstrap.compose_telemetry_provider (the same profile resolution, admission gate, contract check and security floor as boot, shared with bootstrap_context through _checked_companion) and hands it to provider.use_telemetry_provider. NO context is installed, so every other current_context() reader in the daemon keeps the behavior it had (why the daemon composes rather than receiving destinations from its parent is recorded in platform-context.md, under mcp_gateway/gatewayd.py::_compose_and_apply). Composition builds the edition's whole context and keeps only .telemetry; because it is off the boot path and the event loop, a slow edition never delays the daemon. Only when it adds a destination the startup build lacked is the recorder dropped and rebuilt (on the provider's worker thread); the pool gauges, read live, lose nothing. Accepted limit: the startup window is local-only. Counters and histograms the daemon records before that rebuild (kirocrew.mcp.backend.acquire.duration, kirocrew.mcp.lazy_load.*, kirocrew.mcp.warm_pool.acquire) stay in the local sink and are never exported, because the swap replaces the MeterProvider and the OTel SDK cannot hand its cumulative state to a new one. On a non-standalone host that window is the composition time (about 1 s measured on an edition install) plus the rebuild, so a backend acquired in the first seconds after a daemon start is under-counted in the exported series. Making the first acquire wait on composition would put the edition's start-up cost on a session's first tool call, and replaying recorded points would need a second store; neither is worth a one-second gap per daemon start. An off-consent daemon composes nothing and exports nothing; a later opt-in composes lazily in the off-loop rebuild, and a factory that raises costs egress only. Consent is re-read only inside get_recorder(), which the daemon otherwise calls only when a backend is acquired, so daemon/sweepers.py::_telemetry_consent_sweeper forces a re-read (provider.recheck_consent) every gatewayd._CONSENT_SWEEP_SECS (half of _CONSENT_RECHECK_SECS, so 15 s): an idle daemon stops exporting within one tick plus the config read of a withdrawal, inside the 30 s window. The tick forces the read rather than calling get_recorder(), because that one rechecks only once a full window has passed since the worker's last stamp, which lands after the tick that started the read; a tick per window would re-read only every other tick (about 60 s). The resource is the one every process builds (_resource_attributes), with the daemon's own process.pid; its kirocrew.process.* gauges therefore start arriving as one more process per install, which is the per-process design described under "One publisher per install". Its resource carries kirocrew.process.role=gatewayd (absent on every other process), so a query separates the daemon's series, for example its own RSS as kirocrew.process.memory.rss_bytes{kirocrew.process.role="gatewayd"}, from the main process's without joining on process.pid.

Install support with pip install "opentelemetry-exporter-otlp-proto-http==1.44.0". If a destination is supplied but the package extra is not installed, telemetry degrades to local-only with a warning instead of crashing. The OTLP exporter sees the same data points as the local sink: the MetricsRecorder facade sanitises attributes before they reach ANY reader, and call sites are required to pass low-cardinality constants rather than prompts, content, tokens, paths or user ids. That sanitisation is defence in depth over the requirement, not a substitute for it, so egress is only as safe as the call sites feeding it.

Bounded local retention (rec #14, explicit opt-in): both destructive caps default to 0, so upgrading cannot delete existing telemetry history. Operators can opt in independently to age and/or size bounds:

  • Age cap — set retention_days to a positive window (for example 7); shards whose mtime is older than that window are eligible for deletion.
  • Total-size cap — set max_total_mb to a positive budget (for example 128); before an append would exceed the live-shard budget, the exporter rotates that shard and opens a fresh canonical writer. Closed shards are then deleted oldest-first until the combined size is under budget. The active writer is retained; with multiple process-local writers, enforcement remains opportunistic rather than a strict instantaneous directory-wide byte ceiling. In the worst case, live protected shards can temporarily approach the number of active writers times max_total_mb before those writers rotate and closed shards become eligible for oldest-first deletion.
  • Both caps are independently opt-in and can be disabled again by setting the value to 0.
  • After an operator enables a cap, before the first destructive plan in each exporter process, retention emits one fixed, path-free warning and defers deletion for a full 300-second prune interval. Operators can set either cap to 0 during that window. The notice is process-local; there is no persistent migration marker or format to carry into future releases.
  • Per-PID append + rotation are lock-free. A private cross-process .metrics.lock serializes only prune sweeps; contention skips pruning but never discards the DELTA payload already appended. Canonical shards owned by live PIDs or modified within the 300-second safety window are not deleted.
  • Pruning is throttled (≤ once per 300s), runs only AFTER a successful append, and considers only regular files matching the exact generated grammar metrics-YYYY-MM-DD-PID[-ROTATION_NS].jsonl; broad-prefix lookalikes, invalid dates, symlinks, and the lock sidecar are excluded. It is fully best-effort — a rotation/prune failure is logged and swallowed, never breaking export.

Must never be recorded: prompts, message/tool content, token counts, filesystem paths, user ids, and secrets. telemetry.otlp_endpoint is schema-sensitive so credential-bearing collector URLs are masked by config API/UI consumers as well as omitted from logs.

That is a contract on call sites — they emit only low-cardinality enum-like attribute values — backed at the MetricsRecorder facade by the schema.py guardrails (see below), which redact a string matching a known credential shape, or clearing the entropy backstop, to "[REDACTED]" before it reaches an instrument. Namespace validation is exhaustive; attribute redaction is defence in depth with a bounded reach (schema.py documents where the entropy backstop can and cannot fire), so it narrows the blast radius of a call site that breaks the contract rather than making one impossible.

Tests: test/metrics/test_local_exporter.py (retention: direct age cap, oldest-first size cap, live-writer protection, live-shard rotation, non-blocking prune lock, append survives prune contention, both-disabled, broad-prefix/malformed shard lookalikes ignored, export-then-prune never raises), test/metrics/test_provider.py (default-off, env-var opt-in/opt-out, OTLP None by default = no egress, OTLP reader built when endpoint set, degrade when extra missing, plus TestConsentRecheck: out-of-band enable/disable, the rebuild and the flush both off the calling thread, a mid-rebuild disable not undone by the build, the hot path not reading config every call, an unreadable config keeping the live recorder, shutdown() applying a change without waiting out the window and without holding the lock across the flush, and the env pin still winning after a config edit), test/test_collection_status_endpoint.py (GET /api/telemetry/collection: effective state, env/overlay pins, endpoint presence without the endpoint string), test/metrics/test_schema.py (redaction / namespace).

The singleton _consolidate session maps to the bounded consolidation source label through messaging.link's exact-key table. Similarly prefixed keys such as _consolidateXYZ and _consolidate:123 remain other; this key is not a namespace.

Resource attributes: what identifies a series (and what egresses)

The SDK Resource is the payload contract for both sinks: it labels every series on the local JSONL line AND, when otlp_endpoint is set, everything pushed to that collector. _resource_attributes() in provider.py is the single place it is built, and every value there is either a closed set or an explicit clamp, because a resource attribute is a label on EVERY series this process exports — one unbounded value there multiplies every instrument.

Attribute Value Bound
service.name kirocrew constant
service.instance.id the persisted anonymous install id (beacon.install_id) one per install
process.pid os.getpid() one per live process
kirocrew.process.role gatewayd in the MCP gateway daemon (provider.mark_gatewayd_process, called by gatewayd._prepare_telemetry before its first build); ABSENT in every other process, so their resources are unchanged one constant value
kirocrew.harness the install's CONFIGURED default harness: global agent.acp_backend in its config spelling (any id in ACP_BACKENDS_KNOWN, for example kiro, kas, claude, codex), or claude_code when agent.provider is claude_code (metrics/harness.py::set_configured, read from the same config load as the telemetry section) CLOSED: ACP_BACKENDS_KNOWN + claude_code / unknown / other; one value per process
service.version build version, release-clamped via beacon.release major.minor.patch, no build stamp
os.type linux / darwin / windows CLOSED, else other
host.arch amd64 / arm64 / x86 CLOSED, else other
process.runtime.name cpython / pypy / jython / ironpython CLOSED, else other
process.runtime.version beacon.python_minor() major.minor, never the patch
host.cpu.logical_count os.cpu_count() small integer

Two identities, deliberately separate. service.instance.id counts DEVICES: a random UUID persisted on disk, never derived from hostname or username (which on a corporate desktop routinely embed the employee's alias), and stable across restarts so "this install over time" survives them. process.pid counts PROCESSES: one install runs several telemetry-enabled processes at once (the gateway plus spawned agents/apps each build their own recorder), and with an install-scoped resource alone their gauges and cumulative counters would interleave into one corrupted series at a backend. Collapsing the two would report one machine's 6-8 processes as 6-8 machines.

Why the install id may egress while kirocrew.process.start_time may not. The start-time token is a host-local process fingerprint that exists only to make the aggregator's cumulative-reset detection deterministic; it has no fleet-level question to answer, so it is stamped on the JSONL line and kept off the Resource (see local_exporter.py above). The install id is the opposite: it is the only thing that can answer "how many distinct devices", it is random rather than derived, and it is the SAME id the beacon already sends. Note that omitting the key does not yield an unlabelled resource — the SDK substitutes its own per-process uuid4, i.e. a fresh series set per restart — so the choice is between a stable anonymous id and per-restart churn, not between an id and no id.

The id is MINTED in _build_recorder()'s live branch, immediately before the resource is assembled, so the first export already carries it and no startup window can leak the SDK's substitute. That branch runs only under consent True, so an install with telemetry off creates nothing. Consequence to know: enabling metrics is what creates the beacon's install-id file, even on a host that has the beacon itself disabled.

Deliberately ABSENT: the distribution channel (the beacon's data-minimization pass removed its channel field, because channel sharply narrows the crowd a stable id hides in — stamping it on every metric payload would quietly undo that decision) and an install type (no reliable detection today; a guessed label would be confidently wrong). Both are guarded by a regression test.

Tests: test/metrics/test_resource_attrs.py.

The harness label

Which agent harness ran the work, in one closed vocabulary owned by metrics/harness.py. The values are DERIVED from agent_sdk/backends.py::ACP_BACKENDS_KNOWN in their config spelling (backend_to_stored, so kiro-cli is kiro), plus claude_code (the agent.provider == "claude_code" seam, which wins over the backend id), unknown (nothing to resolve at the emit point) and other (an id outside the known set, for example one an edition registered). A harness added to ACP_BACKENDS_KNOWN joins the set with no edit here. Two readings:

Reading Where Value Answers
kirocrew.harness RESOURCE attribute, every series the install's configured default (global agent.acp_backend) how many installs chose which harness
harness POINT attribute on kirocrew.session.started and kirocrew.turn.duration only the session's EFFECTIVE harness, per-session and per-agent overrides included sessions and turns per harness

Refresh. A resource is fixed for the life of a MeterProvider, so the resource value is read in _build_recorder() from the same config load as the telemetry section. watch_config() subscribes the rebuild to agent.acp_backend and agent.provider as well as telemetry, so a change to either drops the recorder and the next metric rebuilds with the new label (the gateway; a short-lived process keeps the value it booted with). A process that registers no watcher, such as the MCP gateway daemon, gets the same rebuild from the consent recheck: the build records the raw agent.acp_backend / agent.provider pair it read (_built_agent_key), and _consent_worker rebuilds a live recorder when the pair on disk differs. Downstream that reads as a new series set from the change on.

Series bound. The resource attribute adds no series inside a process: it has one value per process, so it only partitions installs. The point attribute multiplies kirocrew.session.started by at most the size of the set (len(HARNESS_VALUES): the known backends plus three), and in practice by the number of harnesses one install actually runs, usually 1 or 2 (the default plus a member override). On kirocrew.turn.duration it is close to free: each harness serves its own model ids (model_id_namespace), so model already mostly determines harness, and the real growth is bounded by the (model, harness) pairs an install runs, not by the product of both sets. It is deliberately NOT on kirocrew.session.duration or the kirocrew.turn.tokens / credits / cost_usd family yet.

Process-level metrics (gateway boot, HTTP requests, MCP pool, crew log health, inventory gauges) have no session, so they carry only the resource attribute.

Tests: test/metrics/test_harness_label.py.

Per-destination narrowing: the export allowlist, point attributes, resource

The table above is the resource EVERY sink gets, and the roster below is the instrument set every sink gets. That is the right answer for the local JSONL sink and for an operator's own collector: both are the operator's own machine talking to the operator's own backend. It is the wrong answer for a destination that may receive only part of it -- a product destination reporting how Kiro Crew is used, whose exported set is a reviewed list rather than everything the process happens to record.

So OtlpDestination carries three optional NARROWING fields. Each can only reduce or label what THAT destination receives; none of them can widen anything, reach the local sink, or touch another destination. All three default to today's behaviour, so an existing destination's payload is byte-identical.

Field Default What it does
allow: frozenset[str] | None None A metric whose name is not in the set is dropped before encoding. None sends every metric collected. Full instrument names, kirocrew. prefix included.
point_attributes: Mapping[str, str] empty Fixed pairs added to every exported data point of this destination.
resource_allow: frozenset[str] | None None This destination's batches carry a Resource reduced to these keys. A key the shared resource does not set is simply absent.

metrics/destination_filter.py enforces all three as an exporter WRAPPER that _build_otlp_reader puts around that destination's own exporter. The descriptor declares the narrowing; the core performs it. Four properties follow from where the wrapper sits:

  • Containment is structural. A wrapper only ever sees batches handed to its own exporter. The local sink and every other destination read their own, and a destination that narrows nothing is handed the unwrapped exporter object -- there is no wrapper in its path to differ by.
  • Nothing shared is mutated. A collected batch arrives as SDK dataclasses whose nested objects a sibling exporter may also hold, so every narrowed level is rebuilt with dataclasses.replace rather than assigned in place. That also keeps it correct as the SDK adds fields (NumberDataPoint.exemplars arrived that way).
  • It fails closed. If the narrowing itself raises, the batch is DROPPED and the first occurrence logs one WARNING (the rest are debug, because a per-minute reader would otherwise write a line per interval forever). Forwarding the unnarrowed batch is the one outcome that would turn a bug here into an egress of data the destination was never allowed.
  • Temporality stays a core decision. The wrapper mirrors the wrapped exporter's _preferred_temporality and _preferred_aggregation, which PeriodicExportingMetricReader reads at construction -- answering None there would silently put that destination back on the SDK's CUMULATIVE default while the local sink stayed on DELTA.

A point-attribute key is refused, not dropped. Keys are checked at construction against destination_filter.ALLOWED_POINT_ATTRIBUTE_KEYS (ClientType only today), and an unreviewed key raises. _build_otlp_reader turns that into "this destination gets no reader", so the cost is one destination's egress and local collection continues. Dropping the label instead would publish a series the receiving side cannot route, with only a debug line to say why. Keys are a reviewed non-secret set and are named in that log; attribute VALUES are never logged, least of all beside an endpoint that may carry a credential. On a key collision the destination's fixed value WINS over one already on the point: the value is a constant fixed at construction and a receiving side routes on it, so a call site must not be able to steer it.

PRODUCT_METRICS in metrics/product_metrics.py is the reviewed set a product destination exports under. Names are spelled as literals there because metrics.turns imports metrics.provider at module scope and the recorder reads the set, so importing the declaring modules would close a cycle; test/metrics/test_product_metrics.py resolves every name against its declaring constant instead, and fails on a name with no instrument or on one that is a histogram or an up/down counter. Counters and gauges only: a fixed-bucket histogram's boundaries may not survive a receiving backend, so a distribution is exported as a bucket-labelled counter.

Being a real counter is not sufficient. Every attribute value on an exported series has to come from a closed enum, which the kind check cannot see, so a metric carrying an open value is held out by name with its reason: kirocrew.turn.tokens is wanted and its instrument exists, but its model value is the configured model id, drawn from no set this repo controls -- on a non-Kiro backend it can be a provider resource id, so its range is whatever that provider names and whatever the person pointed the backend at. provider is configured the same way. It waits for a bucketed or enumerated stand-in for those two. A negative test asserts each holdout stays absent and that its name resolves to a real instrument, so the guard cannot pass on a typo and putting a name back means deleting a line that states why it was out.

Tests: test/metrics/test_destination_filter.py (one provider with the real local sink plus narrowed and un-narrowed destinations attached at once: the allowed metric only, the fixed attribute on every point, the reduced resource, both other sinks untouched; then the nearby inputs -- an empty allowlist, an allowlist naming an unknown metric, observable gauges, existing attributes preserved, a collision on the fixed key, two destinations with different allowlists, a resource key the build does not set, a refused key, and a raise inside the narrowing), test/metrics/test_product_metrics.py (allowlist membership, instrument kind, the open-attribute holdouts, and selftests that drive each rejection those checks rely on).

OTLP endpoint entries: telemetry.otlp_endpoints

Every OTLP destination built from config is an entry, parsed by metrics/otlp_endpoints.py into one OtlpDestination each, in this order:

  1. telemetry.otlp_endpoint, when non-empty: the first entry, named telemetry.otlp_endpoint, not validated and carrying no narrowing, so the descriptor is byte-identical to the one the default provider returned before the list existed.
  2. telemetry.otlp_endpoints: the operator's list. The loader keeps only the shape (a list of objects); each entry is checked when destinations resolve.

DefaultTelemetryProvider.otlp_destinations returns otlp_endpoints.config_destinations(cfg). A build that ships its own collector returns it from its provider's otlp_destinations, after config_destinations(cfg) so the operator's entries stay. That is the whole lock mechanism: a destination code returns is not config, so no config value removes or disables it, and no new seam is needed. otlp_endpoints.label_sources marks each resolved destination otlp_endpoint, otlp_endpoints (equal to the descriptor an operator entry produces, each entry matched once) or provider (anything else, reported locked). Matching is by the whole descriptor, not the name, so an operator entry that reuses a build collector's name cannot relabel it.

Entry keys: name, endpoint (required, http(s):// with a host, at most MAX_ENDPOINT_LEN characters), enabled (a boolean; any other value refuses the entry, so a mistyped opt-out never exports), headers_env (at most MAX_HEADERS pairs), temporality ("" / cumulative / delta), allow (a list of names, or a preset name from otlp_endpoints.ALLOW_PRESETS: PRODUCT_METRICS), point_attributes, resource_allow. A malformed entry, including a shape no check anticipated, is skipped with one WARNING per process naming the entry and the field, never a value; it never stops collection. The warn-once memory keeps a 16-byte digest per warning, at most MAX_WARNED (256), and clears when full, so config revisions that are later discarded do not accumulate in a long-lived gateway. An unknown allow preset refuses the entry: an allowlist must not fail open into "send everything". At most MAX_ENTRIES (16) are read. These bounds are arbitrary safety bounds on threads and retained memory, not tuned values. There is no key for a literal header value: an entry carrying headers is refused with a warning that points at headers_env, so no credential has to be written into config.json, where the dashboard and the agent's settings diagnosis can read the config.

Three fields on OtlpDestination carry what an entry needs at reader build:

  • headers_env -- header name -> environment variable name, read by _build_otlp_reader when it builds the exporter. An unset or blank variable refuses that destination (no reader, one warning naming the variable), so a secret never sits on the descriptor or in config.json.
  • temporality -- "" keeps temporality.otlp_preference(); delta (otel_delta_preset) and cumulative (cumulative_preset) pin every instrument kind for that reader only, so the OTel variable cannot move it.
  • process_env_credentials -- True (the default, and the legacy otlp_endpoint entry) leaves the exporter's fallbacks in place: with no headers of its own it sends OTEL_EXPORTER_OTLP_METRICS_HEADERS / OTEL_EXPORTER_OTLP_HEADERS, and with no session it may load the OTel credential-provider session. Every otlp_endpoints entry is False: _build_otlp_reader passes a header map that stays truthy when empty (_OwnHeadersOnly, because the exporter reads headers or <env>) and a plain requests.Session, so the entry sends only its headers_env headers. Those variables were set for one collector; without this, adding a second entry with no headers would send the first collector's credential to the second host.

Gates are unchanged and apply to every entry because every entry is a destination: the consent gate in _build_recorder runs before any is resolved, otlp_governance.governed_destinations applies the otlp_hosts check, and gate_exporter re-checks each export. An enterprise build blanks both local keys in _edition_locked_view (CONFIG_PATH_LOCKS names both).

The config view masks the whole telemetry.otlp_endpoints value (core_config_view._masked_config_dict masks every string under a container marked sensitive), because entries can carry endpoint userinfo. The agent's settings diagnosis (change_cards_settings.non_default_settings, behind diagnose_settings) reports any path the schema marks sensitive, or that sits inside a sensitive field, only as {set: bool} (_schema_sensitive, through config.validation._is_sensitive_path), whatever its key names look like.

Disclosure: otlp_egress_listing(cfg) returns the labelled set before the policy check, and otlp_egress_active is bool() of it. A configured entry is listed even when its reader would be refused for a reason that can lift with no change by the person (an unset headers_env variable, a host the policy refuses today). GET /api/telemetry/collection returns it as otlp_destinations: [{name, source, locked}] -- names only, never an endpoint or host -- and Settings -> Privacy lists those names under its egress note.

Tests: test/metrics/test_otlp_endpoint_list.py (the legacy string unchanged, parsing, refusals and bounds, a build-shipped destination next to operator entries, the gates on it, reader headers and temporality, the disclosure listing, and loopback wire tests: two destinations where the second is narrowed and authenticated from the environment, and the OTel header variable reaching the legacy entry but not a list entry), test/test_change_cards_settings.py (the diagnosis withholds schema-sensitive fields), test/test_config_patch_otlp_endpoints.py (a list entry and a build-shipped destination each refuse the enable; the list is not PATCH-writable; the config view masks it).

Instrumented signals

Metric Type Attrs Site
kirocrew.acp.child_permission.denied counter surface (runtime / session_handle / subagent), reason (closed denial reason) acp/runtime_host_requests.py::_audit_denied_off_loop, acp/session_handle_permissions.py::_audit_handle_reject, and the subagent's permission narrator (subagent_manager/run.py::_SubagentNarrator); one point per backend-child permission request rejected before it can become a silent hang.
kirocrew.acp.child_permission.routed counter surface (runtime) acp/runtime_reader_loop.py when a child permission request reaches the mode-parity pipeline. Together with .denied, this accounts for handled child permission requests.
kirocrew.acp.dropped_frames counter method_class (permission / update / other) acp/runtime_frame_accounting.py::_note_dropped_frame; counts unroutable ACP frames without exposing backend method names. A non-zero permission series is the hang-regression signal.
kirocrew.acp.skill_view.fallback counter outcome (loaded_after_retry / refused_unloaded / refused_unprepared / refused_superseded) acp/runtime_mode_activation.py::_activate_mode_bracketed; one point per session/set_mode that did not land on the freshly prepared skill-view alias first time: loaded after a forced reload, or the session start refused because the host never loaded it, no view could be prepared, or a concurrent start adopted a newer view of the agent (or began one that did not complete) while this one's set_mode was pending. More than a trickle of refused_unloaded or refused_unprepared means the host is not loading published aliases; refused_superseded means session starts overlapped an edit to the same agent's spec, or a newer preparation that did not complete. Starting again uses the new view when it offers the agent; when the new view refuses the agent, starts keep failing until the spec is fixed.
kirocrew.turn.timeout.cause counter path (provider_timeout / dashboard_ceiling), awaiting_permission (bool), children_announced (bool) _on_complete in dashboard/chat_turn/turn_stats.py and dashboard/turn_dispatch.py; attributes the two timeout exits to a pending permission prompt and announced child work.
kirocrew.session.idle_expired counter turn_active (bool), orphaned (bool) session_cleanup.py; emitted after an idle/orphan sweep successfully resets a session.
kirocrew.session.startup.duration histogram (ms) outcome (ready / auth_required / error), spawned (bool), backend (kiro) + phase (total / spawn_init / session_new / session_load / set_model), channel (conversation source), resumed (bool) on the kiro path Two sites. claude: acp/client_start.py::AcpClient.ensure_ready() — times cold-start (spawn + session init) and emits in a finally so every exit path is measured, with no phase attr. kiro (default): providers/acp_runtime_start.py::_emit_kiro_startup_metric — one phase=total point PLUS one point per internal phase; spawned is unconditionally True because _start_kiro_runtime_impl always spawns a fresh runtime (the warm fast-path returns before reaching either site and is NOT measured). outcome defaults to "error" so an unexpected exception is never mislabeled "ready". Consumers MUST treat only the end-to-end point (phase absent or total) as a startup — the phase points are components of one startup. channel comes from messaging.link::telemetry_channel_of, a closed label set (an unrecognised key classifies as other, never the key itself) answering WHICH surface paid the cost; resumed separates the session/load path from session/new. session_load is recorded only when a resume was attempted, and session_new only when create_session actually ran, so a resumed startup never reports a near-zero session_new.
kirocrew.chat.first_token.duration histogram (ms) channel, first_turn (bool), resumed (bool) dashboard/chat_turn/turn_stats.py::_emit_ttft_metric; user-message-to-first-visible-token latency for top-level user prompts. Own bucket family _STARTUP_BUCKETS_MS.
kirocrew.session.pool.decision counter outcome (hit / miss_empty / bypass_member_capabilities / bypass_resume / bypass_stateless / bypass_member / bypass_app_launch / bypass_restricted_context / bypass_member_context / bypass_cwd / bypass_env / bypass_shared_scratch / bypass_backend_override / bypass_effort / bypass_model_at_launch / disabled / other), channel session_parts/runtimes.py::SessionManager._record_pool_decision, one point per get_or_create warm-pool decision. Exactly one reason is reported per decision — the disqualifiers form a disjunction, so branch order picks the reported reason, not the outcome. Deliberately a counter rather than an attribute on the startup histogram: "was the pool used" and "how long did startup take" are separate questions, and crossing them would multiply every phase series. bypass_resume quantifies how often a resume_sid disqualifies a session from the pool, and bypass_shared_scratch how often a dedicated subagent cold-starts to mount its parent's scratch. bypass_model_at_launch counts model-pinned chats on a harness that takes its model only at launch (ACP_BACKENDS_MODEL_AT_LAUNCH): the pooled process cannot switch, so it is discarded and the chat cold-starts on its pin. Values are pinned by session.POOL_DECISIONS, which lists every reason get_or_create assigns (a test reads the assignments), so no reason folds into other.
kirocrew.session.resume.outcome counter outcome (loaded / fallback_replay / no_session_file), channel providers/acp_runtime_start.py::_emit_kiro_startup_metric, emitted only when a resume was attempted. Distinguishes a lossless native resume from the degraded fallback (fresh session/new plus history replay on the Kiro Crew side, taken when session/load exhausts _RESUME_MAX_ATTEMPTS against a stale lock) and from a resume skipped because the session file was gone.
kirocrew.session.duration histogram (ms) end_reason (reset / removed / unclaimed / destroyed / discarded / shutdown / retired / recycled / evicted / crashed), session_source (via messaging.link::telemetry_channel_of) metrics/sessions.py::record_session_ended (or record_sessions_ended for a path that drains many keys at once) for every clean end; metrics/sessions.py::backfill_crashed_sessions for crashed. Why the reason is passed in, never derived. There is no single teardown funnel: SessionLifecycleService (its bodies in session_lifecycle_parts/teardown.py and session_lifecycle_parts/fleet.py) holds six end paths that each pop the registry themselves (reset / remove / remove_if_unclaimed / destroy / discard_conversation / close_all), and four more pop it outside all of those — retire_kiro_identity_sessions reports retired for the sessions it tears down on an identity change, reload_provider_factory reports retired too for the registry it clears on a provider switch (both are "this process is no longer the right host for these sessions", which is why they share a label rather than minting a second one), session_compaction.py::CompactionCoordinator._recycle_held reports recycled (context overflow replacing the provider in place), session_background.py::recycle_heartbeat also reports recycled (the background runtime replacing its own session at cycle end), and the two dead-provider evictions in session_allocation.py report evicted. So each site states its own end_reason, the same contract metrics/turns.py holds its callers to. Every path calls this while it still holds the session registry lock, in the same event-loop tick as its _sessions.pop: the pop and the histogram record are both in-memory and both happen before the call's single suspension point, and a later call lets a racing cold start register a SUCCESSOR under the reused key whose record the predecessor's teardown then consumes. reset is a PATH label rather than a cause — the idle sweep and a slot reset both reach teardown through it, and the finer causes are already counted by kirocrew.session.idle_expired and kirocrew.watchdog.recovery.outcome; the other four are separate precisely because they never take the reset route. EVERY registry removal must record an end, and that is a correctness requirement rather than a completeness one. An unrecorded removal does not merely lose a sample: the crumb below survives it, so the next boot reports the session as crashed — a failure that never happened, injected into the one population this histogram exists to measure. Four such paths shipped unrecorded and were caught in review, which is why the gate for this is test_every_registry_removal_records_an_end, a fail-closed AST walk over every module that mutates the registry (recognising pop, del and clear) rather than a hand-kept list of method names that a new path silently escapes. The crumb is what makes a crash countable. A start time held only in memory dies with the process, which is exactly the population most worth measuring, because a crash runs no teardown path at all. So record_session_started drops one small JSON file under <data home>/metrics/open-sessions/, named by SHA-256 digest of the session key PLUS the writing pid and the start time, with the key stored inside (keys carry channel ids and slot names and are not filesystem-safe; session_map.json already holds every key in the same data home, so this is not new exposure). The name identifies the writer and the generation, not just the key, and that is what makes every unlink here safe. A session key is not unique across processes — BACKGROUND_KEY is a fixed constant, so a kirocrew run and the gateway hold the same key at the same time with a live session each — so while the name was the digest alone they shared one file, and whichever ended first unlinked the other's record; the locks in this module are THREAD locks and never spanned processes, so nothing here could have prevented it. With the writer and the generation in the name, a crumb can only ever be removed by the session that wrote it. Three consequences follow. A clean end unlinks the ONE generation it can name, and an end with no live table entry unlinks NOTHING — the files sharing that digest belong to other processes or earlier runs, and removing them is precisely the bug the name now prevents. A start that displaces a live generation of the same key, and an eviction under _MAX_LIVE_SESSIONS, must each reap the generation they orphan, because once the table forgets a generation nothing can ever name its file again; both ride the same awaited worker hop as the start's own write, so those unlinks stay off the event loop without opening a second cancellation window. And the backfill legitimately finds SEVERAL files for one key, each a distinct session judged on its own recorded owner. Whatever survives to the next boot is a session that never ended cleanly, which backfill_crashed_sessions(started_before) emits as crashed and unlinks; it is also the only reader of these files, so an unparseable crumb is reaped there rather than by an end unlinking blind. A crumb that exists but cannot be READ (an OSError other than FileNotFoundError) is kept with one DEBUG line naming the error class: its owner cannot be judged, so it waits for a later boot (test_session_crumb_unreadable.py). started_before is the calling process's own start time, so a crumb at or after it is left alone. The crumb also records WHO owns it — the writing process's pid plus a per-process start identifier — and the backfill skips any crumb whose owner is still running. The cutoff alone was not enough: it only ever protected THIS process's own crumbs, and cli.py and eval/runner.py each build their own session manager against the same data home, so a kirocrew run session started before the gateway booted was read as a crash. That both invented a sample and deleted the live crumb, losing the real crash that session might later suffer. The start identifier is what makes pid reuse safe: a recycled pid belongs to a different process and must not look like the owner, and that includes reuse of the CURRENT process's own pid — a container hands the gateway PID 1 on every restart, so a crashed predecessor's crumb arrives carrying this process's pid, and skipping it on the strength of the pid alone meant its crash was never emitted and its file never cleaned on any later boot either. Liveness comes from platform_compat.pid_exists; the start identifier only refines it. get_process_start_id returns None for any process the caller may not introspect, and its own contract says a None must NOT be read as a mismatch — using it as the liveness test would judge dead every owner it cannot introspect and reap live sibling sessions. The ownership check fails CLOSED — a live pid whose identity cannot be read counts as running, so an ambiguous crumb waits for a later boot rather than becoming a fabricated crash; losing a real crash sample costs one data point, inventing one corrupts the population this instrument reports. A crumb with no ownership recorded (written before this field existed) falls back to the cutoff alone. A registration that never became a session is discarded, not ended: discard_session_start pops the table entry and unlinks that generation's crumb WITHOUT emitting, for the cancellation window between inserting a session into the registry and finishing its start record — a session that never lived has no lifetime to report, and it must not be left as a crumb the next boot calls a crash. It is deliberately not an eleventh end_reason, because those describe how a live session ENDED. Exactly-once accounting. A crumb is consumed at most once, so the teardown paths — which are NOT mutually exclusive, the idle sweep calls reset — cannot double-count between them, and a backfilled session cannot be counted again on the boot after. The clean path deliberately reads neither the crumb nor the popped entry's created_at: consuming a DIFFERENT record than the crash path would let a clean end's leftover crumb be re-emitted as crashed, and created_at is reset when a provider is recycled in place, so it measures provider age rather than registry residency. A crashed session's END time is its transcript mtime (<data home>/sessions/<stem>.jsonl, appended as the conversation runs, so it is the last moment the session was observably alive and the honest post-hoc maximum). A session with no transcript — a subagent leaves only a replay log — yields no end time and is unlinked WITHOUT an emit rather than recorded as a 0ms lifetime: absence reads as "no data", a recorded zero renders as a real zero. Two locks, split by what they may be held across. _live_lock guards the in-memory start table ONLY and is never held across I/O, because every end takes it on the event loop; _crumb_io_lock orders the two worker-thread paths that touch crumb FILES — the writer, and the backfill's read-decide-unlink — and nothing on the loop may take it. Coupling them was a real stall: the backfill held one lock across a crumb read plus an unlink on a worker while a teardown waited for that same lock on the loop. The end path needs no crumb lock at all, because ownership already partitions the work — the backfill skips every crumb whose owner is live, so the crumbs a session ends in THIS process and the crumbs the backfill reaps are disjoint sets. It is also worth stating that these are THREAD locks and so never protected anything against a sibling PROCESS; that is exactly what the ownership field is for. The write itself runs on a worker thread (asyncio.to_thread) and is AWAITED while the caller still holds the session registry lock, which is what satisfies two constraints that look opposed: the syscalls never run on the event loop, where a slow or network-homed data home would stall every gateway task behind one session insertion, and nothing can interleave, because starts, ends and successors all serialise on that registry lock. Three earlier shapes each satisfied one and broke the other — a fire-and-forget pool write let a late writer delete a successor's crumb (an ended-key set, then a generation token compared before and after the write, both failed, because every check still acted on a path the successor SHARED), and writing inline fixed the ordering by putting filesystem I/O on the loop. Naming the file after its writer and generation is what finally removed that shared path, so the post-write check now removes only this writer's own file. The END path's unlink runs on a worker thread too, and is AWAITED there for the same reason the write is. Inline on the event loop it would be one syscall against the data home on the paths a person is waiting on -- closing a tab, resetting a slot, shutting the gateway down -- so on a slow or network-backed home that single unlink would park every gateway task behind whichever session happened to be closing, felt as the whole gateway freezing rather than as a metrics problem. Queued fire-and-forget it would be worse than slow: shutdown_maintenance_executor drains with cancel_futures=True, so at close_all -- when teardown is heaviest -- the unlink is dropped outright and a cleanly ended session's crumb survives to be back-filled as crashed, inflating the exact population this instrument measures. Awaiting a worker hop is the only shape that is both off the loop and never dropped. The same-tick rule the inline version protected is unaffected, on three counts: the registry pop and the histogram sample both happen BEFORE the function's only suspension point, so a teardown path still accounts for its session immediately after its _sessions.pop; every caller awaits while it still holds the session registry lock, which is the lock a racing cold start must take to register and map a SUCCESSOR; and the hop is awaited so nothing is fired blind. Submitted is not started, so the hop is a BARE executor future awaited through asyncio.shield. On a saturated executor the work item is still queued, and cancelling the await reaches the underlying concurrent.futures.Future through futures._chain_future's _call_check_cancel, which cancels a work item that has not begun -- the same dropped-unlink failure that ruled out the maintenance pool, reached by a different queue. Shielding leaves the inner future alone, so nothing cancels the queued work and it runs when a thread frees up. Two details make abandoning it safe on the cancellation path, and both are why this is not the fire-and-forget shape the start path forbids. It is a Future and NOT a Task, so runners._cancel_all_tasks at loop teardown -- which iterates all_tasks() -- cannot reach it; shielding a to_thread COROUTINE would be wrong, because shield wraps a coroutine in a Task that teardown cancels, dropping a still-queued item all over again. And a worker finishing after the loop closed cannot raise "Event loop is closed", because _chain_future._call_set_state returns early when dest_loop.is_closed() instead of calling call_soon_threadsafe; the historical Windows failure was a start-path write never awaited at all, whereas here the normal path awaits to completion and only a cancellation abandons the future. Unlinking inline in the cancellation handler was tried and rejected in review: it fixed the drop but put a filesystem syscall back on the event loop, which is the thing this issue is about. The start path still must not use this shape, because its write has to be ORDERED under the caller's registry lock and so cannot be allowed to land late; an end's unlink can, because the crumb name encodes the writer and the generation, so a late unlink can only ever reach the file its own session wrote. If the executor is already shut down the hop cannot be scheduled at all, and the crumb is left to the boot backfill rather than unlinked on the loop -- reachable only once the loop itself is tearing down, and the backfill's ownership check means it is claimed only after this process is gone. A cancellation at that hop is ABSORBED by all three end paths, and that is what makes the new suspension point safe to give a teardown. The registry pop is the caller's point of no return: past it only the caller's remaining cleanup can finish the job, and destroy deletes the session-map entry in a finally BELOW its end call while discard_conversation and reset clear the resume SID on the line after the lock. Raising would exit those callers before that cleanup, leaving a permanently destroyed session's mapping behind for the next boot to resume -- a data-loss window the synchronous version did not have, because it had no suspension point there at all. A cancellation cannot undo a teardown, only leave it half done, so the correct answer is to let the teardown finish. That cancellation is genuinely DISCARDED rather than deferred -- a swallowed CancelledError is not re-delivered by asyncio, so the caller runs to completion unless something cancels it again -- which is the intended outcome rather than a side effect, because finishing a teardown that has already popped is exactly what the callers' finally blocks exist for. The consequence worth knowing is that a caller wanting a hard cap on teardown must impose it outside the pop, since a cancellation delivered after the pop is not honoured; no caller does today (every close_all await site is bare). The two guarantees sit in two places deliberately: whether a CALLER may be aborted is knowable only where its post-pop cleanup is known, so it lives in the end paths, while whether the UNLINK survives a cancelled hop is a property of the hop, so it lives in the helper. Two more consequences are worth stating. The sample is emitted BEFORE the hop, the opposite of kirocrew.session.started's counter, because the risks are opposite: a cancelled start is rolled back and never existed, while an end cannot be rolled back (the pop has already consumed the only record of that generation), so emitting first is what keeps the sample under cancellation and what measures the lifetime at the pop rather than a worker round-trip later. And the four paths that pop MANY keys at once -- close_all, drain_all_providers, retire_kiro_identity_sessions and reload_provider_factory -- call record_sessions_ended instead, which pops the whole set under one _live_lock hold and unlinks it in ONE hop: a hop per key would put a cancellation point BETWEEN two keys, leaving every key after it popped but unrecorded, and on close_all (a path cancellation reaches by design) that would turn one orderly shutdown into a burst of fabricated crashes -- strictly worse than the on-loop unlink it replaced. discard_session_start is a coroutine for the same reason, and absorbs its cancellation for a second one too: its callers re-raise the failure that started the rollback on the very next line, so letting a cancellation escape would displace that failure with one of its own making. The crumb is written only under telemetry consent, checked fail-closed: it exists solely to feed this instrument, which is a no-op without consent, so writing one on an unopted host would persist state nothing can read against a documented default of collecting nothing. Bounded throughout: _MAX_BACKFILL_EMITS (2000) caps a runaway crumb directory while still unlinking everything it walks, and _MAX_LIVE_SESSIONS (4096) drops the oldest half of the table, unlinking each dropped generation's crumb as it goes — both cost samples, never memory. Own bucket family _SESSION_BUCKETS_MS.
kirocrew.session.started counter session_source (via messaging.link::telemetry_channel_of), harness (the session's EFFECTIVE harness, closed set below) metrics/sessions.py::record_session_started, one increment per session admitted to the registry, called from the sites that insert into _sessions (session_allocation.py and session_allocation_parts/get_or_create.py cold-start and resume paths, session_background.py, and the compaction path that keeps a session alive) with the provider the entry was built around. harness is read off that provider by metrics/harness.py::harness_of (through providers/acp.py::_provider_backend, the shape check the session-map label uses), so a per-session or per-agent acp_backend override counts as its own harness; see "The harness label" below. The denominator the kirocrew.session.duration population is a subset of: not every registry removal records an end, so started counts admissions while duration counts the endings actually observed, and the gap between the two is itself readable. session_source is the SAME telemetry_channel_of derivation the duration histogram and metrics/turns.py use, so all three group by one bounded label set (an unrecognised key classifies as other, never the key itself). The increment and the _live_starts write are in-memory and therefore safe under the session registry lock; only the crumb touches disk, off-thread.
kirocrew.turn.duration histogram (ms) outcome (ok / timeout / cancelled / tool_stall / stale_recover / stall_exhausted / error / unclassified), session_source (via messaging.link::telemetry_channel_of), model + provider (omitted when the caller cannot resolve them), harness (the session's EFFECTIVE harness; always present, unknown when unresolved) harness is resolved at each owner from the live provider that ran the turn: owner (1) reads it off model_source, owner (2) off the turn's client, both through metrics/harness.py::harness_of. It is not on the row store. Two emit owners, at most one sample per turn. Only a completed or timed-out turn is sampled: a dashboard turn that ends without EVENT_COMPLETE (Stop, the turn ceiling, process death, auth loss, a busy prompt, a session not found, a terminal or transient error) still writes its usage row with a synthetic stop_reason through chat_runner._persist_abnormal_turn_usage, which passes emit_metric=False and emits no kirocrew.turn.* sample. On the complete path the sample is taken before the row's shielded persist, so a cancellation landing during the persist does not lose it. (1) dashboard/handlers/usage.py::persist_token_record_async — the call EVERY dispatch surface already makes once per turn at EVENT_COMPLETE, which is what makes cron, the heartbeat, memory consolidation, subagents, task-runner steps, workflow stages and the messaging channels present in this metric at all, so the page does not report the interactive median as the system's health. It reuses the built row's own duration_ms, so the row store and the histogram cannot disagree about one turn. (2) dashboard/chat_turn/turn_stats.py::_emit_turn_metric, called from _on_complete in the same file, which passes emit_metric=False to its persist call and emits itself — that persist sits behind usage_has_billing, so a turn that timed out having billed nothing writes no row and would otherwise lose its sample, and only this surface holds the EFFECTIVE session key (a linked channel conversation is attributed to its channel, not to the slot.key the row is filed under) and the spent-recovery-budget knowledge that labels stall_exhausted. metrics/turns.py::turn_outcome owns the mapping (""/None/end_turn/stop/completed → ok; the two watchdog stop reasons map to their own outcomes — checked BEFORE the timeout substring — so a recovered stall is never counted as a generic fault and the stall population stays visible; a stall arriving with its 3-attempt recovery budget already spent, or on a NESTED turn (_prompt_depth > 0, which the recovery branches never re-queue — it dies with "please retry"), labels stall_exhausted, which IS a terminal fault to the aggregator, so the recovered-stall exclusion cannot hide a session that dies needing user action). cancelled is a user pressing Stop, matched by EXACT equality against STOP_REASON_CANCELLED so the watchdog's "error: cancel unacked" still reaches the error branch; folding it into error would put every deliberate cancel into fault_rate's numerator, so it is excluded from the numerator while REMAINING in the denominator (the turn ran). unclassified marks a turn whose surface passed a bare TurnUsage and so had no stop reason to give; it is in neither the numerator nor the denominator of fault_rate (in the numerator it would invent a fault for every clean background turn; in the denominator alone it would dilute the rate towards zero as background traffic grows), and its count is readable as outcome["unclassified"]. One histogram powers turn latency p50/p90 AND fault rate. The value is duration_ms or elapsed_ms: the acp provider always reports TurnUsage.duration_ms == 0 (only claude_code fills it), so the caller must pass the locally measured wall clock as elapsed_ms or nothing is ever emitted. A still-zero value skips the emit deliberately — absence renders as "no data", whereas a recorded 0 would render as a plausible 0ms p50. What it measures: the wall clock starts at turn start, so a turn parked on an interactive tool-approval prompt counts operator thinking time. No finer-grained source exists on the acp path, so this is "turn wall-clock", not pure model latency — a high p90 can mean slow approvals rather than a slow model.
kirocrew.turn.tokens counter (token) direction (input / output), model + provider Same two owners and the same exactly-once split as kirocrew.turn.duration, emitted from metrics/turns.py::emit_turn_usage off the same built row, so volume and latency describe one population. ONE instrument with a direction attribute rather than an .input/.output pair: the two series then carry an identical attribute set by construction (a model added to one cannot be forgotten on the other), the dashboard's counter path already reports every attribute combination under by_attr, and direction is a two-value enum so the cardinality cost is exactly 2x. Positive-only. Each backend fills only the dimensions it bills in; the kiro/acp backend reports zero for every token field, so emitting the zeros would publish a full series of them per turn and a recorded 0 reads as a measured zero rather than as "this backend does not report here". A negative value is also dropped — a monotonic counter cannot take it back.
kirocrew.turn.credits histogram (credit) model + provider The turn's billed amount on a credit-billing backend (kiro/acp). Emitted only when non-zero, which is what makes it mutually exclusive with cost_usd on a given host (acp/types.py: "Consumers read whichever is non-zero"). Two instruments rather than one with a currency attribute because credits and dollars are not the same quantity: percentiles over the union have no unit. Own bucket family _CREDIT_BUCKETS (0.025–2500), calibrated against 17,240 real per-turn rows (min 0.03, p10 0.076, p50 6.8, p90 53, p99 155, max 658). NOT in _HISTOGRAM_BUCKETS_MS, and claimed by name in _aggregate so it never reaches the *_ms surface — see the bucket-boundaries section.
kirocrew.turn.cost_usd histogram (usd) model + provider The same amount on a dollar-billing backend (claude_code / bedrock), same non-zero gate, same owners. Own bucket family _USD_BUCKETS (0.001–100). This is the one bucket array with no local calibration — a credit-billing host reports zero for every cost_usd row by construction — so it is sized from published per-token pricing against the observed token range and is worth re-checking once a dollar-billing host reports. Reported through _amount_stats, which keeps six decimals so a sub-cent turn does not round to 0.0.
kirocrew.tool.call.duration histogram (ms) tool_kind (read / fetch / search / edit / write / create / delete / move / execute / think / switch_mode / client_built_in / mcp / other), outcome (completed / failed / cancelled) Round-trip latency of one tool call, keyed by kind and never by tool name. The tool NAME is deliberately not an attribute: MCP names are unbounded (any installed server contributes its own) and the recorder caches one instrument per distinct attribute value with no eviction, so a name-valued label is a cardinality bomb. tool_kind is the ACP kind normalised by metrics/tool_calls.py::classify_tool_kind and pinned to the closed TOOL_KINDS set; the kind field arrives verbatim from the agent, so an unrecognised or agent-authored kind — including an empty one — folds to other, which is what stops a tool name minting a series. An MCP-served call is labelled mcp whatever kind it reported, decided by the trusted _meta.kiro.mcpServerName rather than the reported kind: an MCP server's kind vocabulary is its own, and "this call left the process over MCP" is the more useful fact than a builtin's finer verb. outcome carries exactly the three terminal ACP statuses — a non-terminal update (pending / in_progress / absent) records nothing and leaves the clock running, so no fourth value is reachable. A <= 0 span skips the emit, the same rule kirocrew.turn.duration follows. The clock is time.perf_counter, NOT time.monotonic: on Windows the latter advances in ~15.6ms ticks, so any call completing inside one tick measured exactly 0.0 and was dropped by that guard — which silently hid every sub-tick tool call on the platform, i.e. most cached reads. perf_counter is the highest-resolution monotonic clock available everywhere, so the guard keeps meaning "unmeasurable" rather than "fast". Site: two parser layers, one registry, one sample per call. There is no single choke point, because the two backends parse through different code: the shared builders in acp/_dispatch_parts/session_update_parser.py and acp/_dispatch_parts/tool_result_events.py (_build_tool_call_event starts the clock once the trusted MCP identity is resolved; _build_tool_result_event stamps the terminal status BEFORE its output parse can return None, so an output-less completion is still measured), where the scope arrives as cache_scope forwarded by the session handle (acp/session_handle_updates.py, acp/session_handle_permissions.py, acp/session_handle_dispatch.py) as the frame's sessionId; and acp/client_tool_events.py::_extract_tool_call_update with its result-extracting sibling, which shape the same frames inline and scope on the client's own _session_id. Every other surface consumes the emitted AcpEvent stream downstream of these two, so instrumenting both parsers covers all of them. The registry is keyed `scope
kirocrew.crew.session_control.calls counter op (create / send / fork / stop / close / continue / broadcast / status / read_message / summary / retry / reload / rename / set_model / end_wait / adopt / release / generate_title / other), caller (the agent_kind enum: default / crewmate / conductor / pipeline_conductor / ledger_conductor / security_conductor / worker / team_lead / dashboard_author / dashboard_manager / knowledge / research / heartbeat / lite / guest / other), outcome (ok / error) Which session-control operations agents reach for, and how often they fail. Session control is the set of MCP tools an agent uses to act on ANOTHER live conversation — open one, send into it, stop its turn, read its tail, adopt it, close it. People use the dashboard for all of that, so every caller here is an agent. kirocrew.tool.call.duration cannot answer the question: it labels a call by ACP tool_kind and labels an MCP-served call mcp whatever kind it claimed, which is the right trade there and the wrong grain here — it folds all eighteen verbs, every other MCP server's tools and their outcomes into one series. op is the TOOL's verb, not the route's path segment, so /api/session-control/read is read_message (the session_read_message tool) and /set-model is set_model: the question is which tool an agent reached for, and the tool verb is the stable name of the two. The route table is Kiro Crew's own, so the list is closed by construction rather than by trust, but classify_op still folds an unlisted value to other so a route added without a row cannot mint a series — revive and history are registered routes the list deliberately does not name and both land in other today. caller is derived from agent_files.OWNED_KIRO_AGENT_FILES (file stem minus kirocrew-, - rewritten to _) rather than tabulated, so a new owned agent file joins the enum with no table to update and cannot fall silently into other; the bare kirocrew agent maps to default, and a mate's pinned DM session is identified by its SLOT KEY's shape (members.is_member_session_key) because such a slot's agent field carries the mate's own crew name and could only answer other. An agent name outside the owned set — a third-party agent, a user-authored template — is other: the raw name is free-form with no cardinality bound, and folding it before the recorder ever sees it is what keeps the series bounded. A BLANK agent on a resolved slot is default, not other: _ChatSlot takes agent: str = "" and an ordinary dashboard chat is minted without one, so the blank is what the largest caller population carries, and crew_log/emit_session.py already reads it that way (agent or _DEFAULT_AGENT). A caller whose slot could not be resolved AT ALL is other — same blank agent, different answer, and classify_caller's slot_found is where that line is drawn. outcome is decided by the status the route answered with (below 400 is ok) or error for a handler that raised. Ceiling 19 × 16 × 2 = 608 series, and a backend that should not carry all of them can drop outcome from its dimensions. Why not kirocrew.gateway.request.duration. That histogram already covers these twenty paths, labelled method × route_template × status_class, so per-route volume and per-route failures are readable there today and this counter is not the place to go for them. What no instrument carries is WHO called: it has no caller-identity label, and adding one would multiply every route in the gateway's bounded template set by sixteen rather than only these twenty. caller is the axis that makes this counter exist; op and outcome are the dimensions it is crossed with, because "which verbs does a conductor drive, and which of them fail for it" is one question rather than three. Site: one wrapper per route, outside the auth gate. metrics/session_control_calls.py holds the enums and the emit; dashboard/handlers/session_control.py::_counted wraps each of the twenty registered handlers and is the ONLY emit site, so the deferred route binder calling a handler cannot count a call twice. The wrapper sits OUTSIDE the handler's own _require_internal / member gate deliberately: a caller refused there made a session-control call that failed, and a wrapper placed under the gate would not see it. A handler that raises is counted error and its exception re-raised untouched, so this only ever ADDS the count — the response, the status and the SEL audit line stay the handler's own. The caller classification reads LIVE STATE ONLY (the in-memory slot registry, via get_slot and then caller_access.caller_slot_key's walk), so no disk read lands on a route an agent is waiting on, and an unresolvable caller is other rather than a failure. Best-effort throughout, behind telemetry.enabled: a disabled process holds a recorder with no meter and the emit returns before touching an instrument.
kirocrew.watchdog.action counter action (deferral / probe / cancel), verdict (working / dead / unknown / stuck_input), evidence_class (established_flat / mcp_flat / remote_flat / shell / shell_absent / platform_limited / wait / degraded), window (narrowed / extended / working_cap / standard), agent_override (bool) acp/session_handle_watchdog.py::AcpSessionHandle._emit_watchdog_metric, one point per watchdog DECISION in _dispatch_events: deferral from _log_working_deferral (rides its 10-min rate limit, so an hours-long WORKING build contributes a bounded handful of points, not one per tick), probe at the stale-probe send, cancel before _end_stalled_tool. evidence_class is _watchdog_evidence_class — a prefix/shape bucket of the free-form oracle evidence (pids/deltas/commands never emitted). window encodes the effective window selection: narrowed = a tool-branch evidence TAG reduced the suspect window below the build-scale default (90 min) — established_flat to the model-silent budget (minutes), shell_absent to the ordinary silence budget (stale_window_secs, 600s) because the shell command has no process to its name, platform_limited to the same stale_window_secs only when the tool layer classified the shell command prompt-shaped, remote_flat to remote_flat_probe_secs (off by default; opt-in) for an MCP tool blocked on its own remote call; extended = model-wait-branch established_flat extended the stale window from 600s (stale_window_secs) to 1800s (model_silent_probe_secs) for a non-streamed server-side think; working_cap = an opaque MCP tool's mcp subtree active WORKING reading ran past tool_stall_hard_cap_secs and got the tool-stall cancel; standard = ordinary window in all other cases. agent_override is the per-agent watchdog-override BOOLEAN from the WatchdogSettings snapshot — deliberately NOT the agent name (free-form ⇒ cardinality bomb; per-agent joins happen via the row store's agent + stop_reason fields below). Guardrail query: action=cancel, evidence_class=mcp_flat, window=standard must not increase — the narrowed window may only affect established_flat, shell_absent, a prompt-shaped command's platform_limited and remote_flat.
kirocrew.watchdog.idle.duration histogram (ms) action, evidence_class Same emit helper, same decision points; value = the branch's idle clock (_tool_idle / _stale_idle) at decision time, converted to ms at the emit site because the dashboard's generic aggregation reports a histogram under *_ms keys unless its emitting module declares a non-millisecond unit for it, and this one declares none (a seconds instrument would render 1000x off). Answers whether 900s is right for LLM-shaped stalls (idle-at-action distribution per evidence class). Own bucket family _WATCHDOG_IDLE_BUCKETS_MS (1s–4h, densest at the 300/900/3600-second window boundaries).
kirocrew.watchdog.recovery.outcome counter mechanism (stale_recover / tool_stall / throttle / transient_5xx), outcome (recovered / exhausted), attempt_bucket (1–3) dashboard/chat_turn/turn_stats.py::_emit_recovery_outcome, derived from the per-slot retry budgets the stop-reason branches maintain (slot._stale_recovery_retries / slot._tool_stall_retries). throttle and transient_5xx count the interactive ladder's verbatim re-sends of a top-level turn after a pre-output provider error (slot._turn_retries, via chat_turn/recovery.py::_turn_retry_outcomes): recovered when the re-sent turn lands ok, exhausted when it still ends on the terminal error card, one point per mechanism the turn used. exhausted emits in the stall branches when a budget hits its cap ("start a new chat"); recovered emits at the budget-reset block when a turn completes with outcome ok while a stall budget is armed — the stall branches return early, so an armed budget reaching that reset is by construction a completed recovery cycle (gated on ok so a user cancel of the recovery turn never counts as a recovery). attempt_bucket clamps to the 3-attempt cap (closed enum, mirrors the CLI's attempt_number_bucket). Every stall recovered point is one prevented hang. Fault accounting for the exhausted case lives on the turn histogram, not here: the final turn of an exhausted cycle labels stall_exhausted (see kirocrew.turn.duration), so a dead session counts toward fault_rate while this counter stays pure mechanism telemetry.
kirocrew.autonudge.loops.started counter kind (loop / monitor) Every inactive-to-active transition of a loop, so a start is counted for each active period and every stop below has a matching start. Taken at the same commit point and from the same baseline as kirocrew.autonudge.loops.stopped (autonudge_service/store.py::_record_stops_committed, which calls _record_starts_committed): an id active in the rows just written and absent from the last committed store is a start, the mirror of the stop diff. No mutation names this counter, so the prompt arm, the structured arm, a revival through update (a user resume, and the Issue Radar and Research Lab reconcilers re-arming an inactive loop on every pass), the restore after a failed session close, a replacement rollback that writes the displaced row back, and any activation path added later are all counted the same way, after their write has committed; test_an_activation_path_that_never_names_the_counter_is_counted pins that. A pause, a resume and a later stop therefore record two starts and two stops, and a re-save of a loop that was already active counts nothing. A restart counts nothing: _load seeds the baseline from the rows active on disk. A row _load resumes from the torn-deactivation shape (inactive, no stop reason, live deadline) was inactive on disk, so it counts once, at the first commit that writes it active, and its later stop is paired. A row _load could not parse is rewritten verbatim on every commit but never served, so it is skipped. A refused admission, a refused revival, a 409 on an existing record and a failed persist commit nothing and leave the counter untouched. kind is read from the same autonudge_stop_log::_summary the stop record carries: monitor when the row carries monitor state (a structured controller record, or a prompt loop whose probe gate watches a subject) and loop otherwise, so a start and its stop share one label. No loop id, slot key or target is an attribute.
kirocrew.autonudge.ticks counter kind (as above), outcome (delivered / quiet / refused / failed) autonudge_service/firing.py, one point per tick that reached a verdict about spending a turn: the probe gate's quiet branch in _timer records quiet, and _run_fire_cycle records the other three immediately after the fire callback settles. The ratio of quiet to delivered is the number to read -- it says whether loops idle or work, and whether the probe gate is earning the subprocess it costs. refused (the callback declined: the slot was mid-turn) is kept apart from failed (the callback RAISED) because a refusal is the system working as designed while an exception is a defect, and folded together a wedged fire callback is indistinguishable from a busy session. A tick that returns before deciding about a turn -- a spent cycle cap, a stand-down, a hold for a person -- records NOTHING here: it decided about the loop, which kirocrew.autonudge.loops.stopped already reports, and counting it would put events that were never candidates for a turn into the denominator this ratio is read against. The quiet emit therefore sits INSIDE the loop.active and loop.id in self._loops guard: a terminal verdict ("the watched pull request merged") also answers quiet while having just deactivated the loop, so emitting above the guard would report one event twice, once as a tick and once as a stop. refused carries the same guard: a fire callback that removes the loop and then returns False (the session is gone, the channel is unsupported) is counted only as the stop. What this counter does NOT observe: a structured monitor record is driven by its own typed controller (firing.py hands it to _on_monitor_tick and returns) and reaches neither emit site, so it contributes nothing here; its per-tick verdict is kirocrew.monitor.decisions, which is the controller's own decision and the only honest reading of that path. The consequence to know before building a dashboard: within ticks, kind=monitor means a probe-gated PROMPT loop, a narrower population than kind=monitor on the started and stopped counters, so a per-loop rate must not be taken as ticks over loops.started for that kind. The quiet-to-delivered ratio is read inside ticks against its own population and is unaffected. test/metrics/test_monitor_loop_counters.py pins both: a structured record's tick records no point, and a terminal quiet verdict records a stop and no tick.
kirocrew.autonudge.loops.stopped counter kind (as above), reason (closed: cycle_cap / runtime_budget / stop_sentinel / monitor_terminal / autonudge_stop / approval_stall / session_start_failures / consecutive_failures / structural_terminal / model_refused / sentinel_dropped / verdict_stall / agent_turn_budget / token_budget / provider_error_budget / invalid_monitor_record / unsupported_monitor_version / completion_evidence_unavailable / session_unavailable / manual / user_stop / session_close / success / blocked / budget / target_unavailable / removed / unspecified / other), ran (lt_1h / 1h_24h / gt_24h / other) The store's one commit point (autonudge_service/store.py::_record_stops_committed), from the same stop record the WARNING line and the stop file are written from (autonudge_stop_log::stop_records). Exactly once per active-to-inactive transition, by construction: a record exists only for a loop that was active in the previously committed store and is not active now, so a loop resumed and stopped again is counted twice here and twice in loops.started, so a second autonudge_stop on an already-stopped loop produces no record and no increment, a restart does not re-report a stop already counted, and the dozen paths that can stop a loop -- the cycle cap, a spent runtime budget, an unanswered approval, a restart mid-cycle, a user delete, a closed session, an agent's own call -- cannot double-count between them. The stored reason never travels as itself: it is model-authored on some paths and composed at the call site on others (campaign_<status>, orphaned_worker), so metrics/monitor_loops.py::STOP_REASONS maps it to the closed value above and anything unrecognised becomes other. That table is BUILT from the constants and the enum that write those strings, never retyped beside them -- the *_REASON constants in autonudge_service/model.py, the MONITOR_STOP_* constants in monitoring/models.py, the terminal MonitorOutcome a monitor row falls back to when it carries no stop reason of its own, and autonudge_stop_log's two defaults -- so a reason added to any of those sources is reported under its own name without an edit here. A structured monitor that stops on a provider verdict stores the observation's own reason code (pull_request_merged, pull_request_closed, the monitor_blocked default), which is per-provider and not in that table; when the stored reason maps to other and the row is still present with a terminal MonitorOutcome, the counter reports that outcome instead (metrics/monitor_loops.py::monitor_outcomes), so a merged PR monitor reads success and a closed one blocked. A retyped table silently reported the one member it had missed as other, which is what this derivation prevents and what test_every_stop_reason_constant_is_covered and test_every_terminal_outcome_is_covered pin. A spent runtime budget is reported as runtime_budget, the same string the store, the WARNING line and the stop file carry, so one name finds it in all four places. One stored reason is reported under a different spelling, spelled out here because a reported value that differs from the stored string must be findable in one place: the legacy approval_stalled joins the structured spelling approval_stall under one value, because the two are the same event recorded by the two record shapes. ran is a counter with a closed bucket attribute rather than a histogram (RFC rule 2), with edges at 3600s and 86400s: lt_1h is a runtime under one hour, 1h_24h is one hour up to but not including twenty-four, gt_24h is twenty-four hours or more. A row whose creation time was missing or unreadable yields ran=other rather than being folded into lt_1h, which would claim a short life there is no evidence for. The stop record's free-text detail is never read.
kirocrew.monitor.decisions counter decision (no_change / record_only / wake_actionable / stop_success / stop_blocked / retry_provider / stop_budget -- the members of monitoring.models.MonitorDecision), kind (a registered monitored kind: github_pull_request / gitlab_merge_request / azure_devops_pull_request / bitbucket_pull_request / gh-pr / github_workflow_run, else other) monitoring/decision.py::decide_monitor, once per call, at the one place every path through the policy converges -- after the stall streak is folded, so what is counted is the verdict the caller acts on rather than an effect a later step may still replace. The population is POLICY EVALUATIONS, which is narrower than every verdict a caller acts on: monitor_records.py::apply_monitor_probe short-circuits some ticks to a verdict of its own (a record whose outcome is already written, a reading it refuses) without consulting the policy, and those are deliberately uncounted -- the series answers "the policy was asked, and this is what it said", which is what makes a rate over the decisions readable. monitoring/shadow.py::run_shadow_probe also asks the policy and is counted the same way; it has no caller in the gateway today, so every point on this series comes from a live monitor. A pure policy module may carry this because recording is an in-memory add that reads no clock, file or socket and swallows its own failures, so the module stays deterministic for its callers and its tests. kind is checked against monitoring/registry.py, the catalogue every boundary that gates a kind already reads, so a kind from a store row written by a newer gateway has no entry and is reported as other; the target, the objective and every fingerprint stay out. metrics/monitor_loops.py::MONITOR_DECISION_VALUES is DERIVED from MonitorDecision rather than restated beside it: a second hand-written copy of a seven-member enum could only ever catch a new decision by reporting it as other while reading as though someone had checked.
kirocrew.context.section.duration histogram (ms) section (one fixed label per assembled block: preamble / profile / workspace / docs / steering / thread_history / stop_notes / memory / skills / lessons / provenance / finalize, plus episodic from the build_message site), custom (bool) Two sites, both first-turn only. ContextBuilder.build_session_context (context_assembly/session_start.py) emits one point per section from monotonic checkpoints taken as each block is appended; ContextBuilder.build_message (context_assembly/turn_message.py) emits section=episodic for the query-dependent episodic retrieval that runs as that method's sibling rather than one of its sections. Why per-section: the block is assembled AFTER the user's message arrives and the caller awaits it before dispatching the prompt, so every section lands directly on time-to-first-token; as one opaque interval the cost is unattributable and diagnosis degrades to guess-and-rebuild. The spread within a single build is the widest of any instrument here — string appends under a millisecond alongside a query-embedding section reaching seconds — which is why it takes _FAST_BUCKETS_MS (0.5ms..60s) rather than a startup ladder. custom is a bool rather than the agent name deliberately: a populated install has dozens of agents, and one series per agent per section would multiply series count for no diagnostic gain. Sections under 1ms are still recorded as points but omitted from the companion INFO line to keep it readable.
kirocrew.db.query.duration histogram (ms) store (memory / knowledge / vector / history / other), op (search / read / write / delete / migrate / connect / other), outcome (ok / error) metrics/db_metrics.py; times logical SQLite store operations rather than SQL statements. Raw SQL and search terms never become attributes.
kirocrew.crew_log.write.lost observable gauge (lifetime total) cause (dropped / overflowed / wake_dropped / other) metrics/crew_log_health.py::register_crew_log_health_instruments; entries the crew log never got, and fold wakes its queue could not hold. Every cause is detected on the crew log's APPEND thread or on a producer's own thread, and is already counted there (WriterStats.dropped / .overflowed, crew_log/eager.py::dropped_wakes), so this adds no call at those sites: one observable callback runs on the metrics reader's ticker thread, reads the three counters through sys.modules (importing nothing) and folds each delta into a monotonic total. A GAUGE whose reading is that process-lifetime total, not a counter, and for the reason the process CPU and GC readings became gauges: reading a total this module does not own can only be an async instrument, an observable counter would export CUMULATIVE, and test_no_instrument_exports_cumulative allows no cumulative series anywhere because one forces every consumer into stateful whole-hour aggregation. Declaring DELTA for it instead is the move metrics/temporality.py records as tried and reverted, since the delta baseline lives in the provider and a consent change rebuilds it. The loss RATE is the consumer's difference between two readings. The first collection is a BASELINE, so losses accrued before telemetry was enabled are not reported, and the totals are in memory only, so a restart re-counts nothing. dropped and overflowed are real holes in the log (a stuck disk vs a saturated one); wake_dropped is not, because the next read carries the fold forward, so it only means a dashboard saw the value late.
kirocrew.crew_log.fold.apply counter scope (slot / session / other), fold (the crew log's fold catalogue: status / usage / timeline / tools / approvals / subagents / class / outline / otel_usage / otel_session_tree / ledger / radar / work / panel / agentic / mistakes / workstreams / worktree / other), bucket (lt_10ms / lt_100ms / lt_1s / ge_1s, bounds exclusive, so exactly 10ms is lt_100ms) crew_log/eager.py::_advance and ::_advance_session close each pass through metrics/crew_log_health.py::note_fold_pass. One point per fold the eager worker's own pass ADVANCED over one coalesced batch, on that worker's thread and nowhere else: every record site is a thread-local check that returns at once on another thread, so the same fold code reached from a dashboard read records nothing. A session pass walks one unit's file once for every session fold, so the duration is the pass's and is attributed to each fold that moved. A pass that advanced nothing records no apply, whether an unchanged live cell answered it, the read path refused it, or it raised, so fold.start above fold.apply is the share of eager folding that produced no value. READ THIS BEFORE READING THAT RATIO: when a dashboard read reaches a growing cell first, the read does the folding and the worker's pass behind it finds nothing changed, so that fold counts as a start and not as an apply; the pair describes eager folding, not all folding. The bucket measures the pass end to end, INCLUDING any wait for the per-cell fold lock, because a pass held up behind a reader is slow eager folding. The clock is read once when the pass closes, before any point is recorded, so every fold of one pass lands in the same bucket and a first recorder build is not part of the duration. A bucket counter rather than a histogram, per the no-new-histogram rule: the edges live in crew_log_health.bucket_for (shared with savepoint.write) and in this row, nowhere else. The fold catalogue has ONE owner, crew_log/projection_catalogue.py::FOLD_NAMES, read into a frozen set at import, so a fold the crew log gains is labelled with no edit here and the names above are the catalogue as it stands rather than a second copy that could drift. A second literal copy with a check that raised on drift would be reached through the fold surface's module-scope import of the metrics module, which would turn a forgotten label edit into a failed import of every dashboard read and session fold.
kirocrew.crew_log.fold.start counter scope (as above), origin (warm / savepoint / cold / other) The branches that decide where a fold's starting state comes from, which are crew_log/projection_session.py::_fold_attempt and crew_log/projection_slot_memo.py::_fold_slot_warm_locked, leave a thread-local mark (crew_log_health.note_fold_origin), and the eager worker turns it into a point when it closes the pass. A mark rather than a record because two of those branches hold a per-cell fold lock and a first get_recorder() builds the telemetry provider synchronously, which would hold that lock against a waiting reader; the marks are keyed by scope because resolving a slot fold's units can fold a session log. Exactly one point per eager pass: a pass that folded twice after the file changed identity under it reports the origin it finally folded from, and a pass that raised before choosing one records nothing at all. Eager fold worker thread only, as fold.apply is. A rising cold share is what a savepoint that is no longer being kept looks like from the outside; warm includes a pass an unchanged live cell answered without folding anything, which is the case fold.apply leaves out.
kirocrew.embed.queue_wait / .inference histograms (ms) queue wait: none; inference: chars_bucket (<=128 / <=512 / <=2000 / <=6000 / >6000) embeddings_llama.py::_emit_embed_timing; separates time waiting behind another embed from model inference time.
kirocrew.telegram.api.duration histogram (ms) method (Bot API method), outcome (ok / timeout / rate_limited / error) telegram/client_http.py::_record_api_duration; awaited Bot API round-trip latency.
kirocrew.mcp.backend.acquire.duration histogram (ms) warm (bool — not was_spawned) mcp_gateway/daemon/diagnostics.py::_emit_backend_acquire_metric — ensure_backend pre-flight + lazy-spawn paths; acquire-only duration captured before attach_stub/create_task overhead.
kirocrew.mcp.lazy_load.count / .duration counter + histogram (ms) transport (stdio) mcp_gateway/daemon/diagnostics.py::_emit_lazy_load_metrics — legacy lazy-spawn path (also emits backend.acquire).
kirocrew.mcp.warm_pool.acquire counter result (hit / miss) mcp_gateway/prewarm.py::HotKeyStore.record_outcome (emitted outside the lock).
kirocrew.mcp_gateway.backends.live gauge (1) class (first_party / third_party / other) metrics/mcp_gateway_gauges.py::register_mcp_gateway_gauges, callbacks run only at reader collection and observe only in the process that called bind_pool (gatewayd, in run_gatewayd). Live pooled backend processes, read through BackendPool.live_pooled_backends: the same population the dashboard's pool snapshot (BackendPool.metrics_snapshot_async, served by daemon/control.py) counts, so the two saving figures agree. Draining backends (an old process finishing calls after a credential rotation) and connection-private backends (never shared) are excluded: no session can be pooled onto either. class is first_party for a name in mcp_discovery_managed._MANAGED_SERVER_NAMES, third_party for any other name, other when the name is missing; the name is dropped after classifying. first_party and third_party are always published (0 is a real reading), other only when non-zero.
kirocrew.mcp_gateway.sessions.attached gauge (1) class Same module and population: the summed stub attachments (Backend.refcount) on those backends. sessions.attached - backends.live is the number of backend processes pooling avoided. One runtime opens one stub per server however many sessions it hosts, so this is a lower bound on sessions and the saving derived from it is a lower bound too.
kirocrew.mcp_gateway.backends.rss_bytes gauge (By) class Same module and population: summed current RSS of each backend through mcp_gateway/pool.py::backend_rss_kb, the probe the dashboard's pool snapshot uses (the backend pid AND its descendants, so a launcher that forks the real server is measured with it; KiB, exported as bytes). A pid with no reading (it died, or the platform has no /proc subtree walk: macOS and Windows) adds 0 and is skipped; a class where no backend could be read publishes nothing, a gap rather than a fake zero. bytes_saved ~= (sessions.attached - backends.live) * backends.rss_bytes / backends.live. gatewayd's own RSS is kirocrew.process.memory.rss_bytes where the resource has kirocrew.process.role=gatewayd; the main process's has no kirocrew.process.role.
kirocrew.skill.lazy_load.count / .duration counter + histogram (ms) hit (bool); the counter also skill (closed, see the usage metrics section) skills_body_reads.py::load_skill (bound as SkillsLoader.load_skill) via _emit_lazy_load_metric (best-effort; never breaks skill loading).
kirocrew.subagent.spawned counter concurrency (int — the live running count INCLUDING this spawn, bounded by the manager's own concurrency cap, so a MAX over it is the high-water mark without a second instrument), batched (bool — the spawn is a member of a fan-out wave rather than a lone one) subagent_manager/admission/pump.py::_PumpMixin._log_spawned_impl — the confirmed-start funnel, reached only after a spawn is admitted, so a rejected or never-started spawn is not counted (a promise an admission-time increment could not make). The emit_counter import is function-local, not module-level: bind_component_globals rebinds each *_impl's __globals__ to the owning namespace, where a module-level import here would not be visible.
kirocrew.subagent.completed counter outcome (completed / failed / stopped / unknown -- SubagentInfo.outcome's own canonical three-way classification, restated as metrics/events.py::SUBAGENT_OUTCOMES because metrics sits below the sub-agent manager in the dependency order, and folded through subagent_outcome so a record whose outcome cannot be read lands on unknown rather than being guessed as failed) subagent_manager/terminal.py::TerminalCoordinator._record_terminal_metric, called from _report_terminal_impl past the point its finalize claim has been granted. That point, rather than either crew-log closer, is what makes one increment per run true: the reaper and the run loop's own finally both race to report and exactly one is let through, whereas a counter on subagent/completed and a second on subagent/failed would count a run twice on the paths that reach both. Gated on telemetry.enabled alone and NOT on the crew-log emitter, so the population is not narrowed to installs that happen to keep a crew log. The emit_counter import is function-local for the reason the row above gives.
kirocrew.webhook.requests counter outcome, over two disjoint halves plus a remainder. Recorded runs (webhooks.VALID_OUTCOMES): completed / timeout / error / rejected_capacity / unauthorized / disabled. Refusals the run store never sees (webhooks.METRIC_REFUSAL_OUTCOMES): throttled (the per-source auth-failure throttle), unavailable (a store the endpoint must read is unreadable, or a destination agent could not be verified), rejected_body (the request's own content refused -- too large, unreadable, unparseable, or a field of the wrong type). Remainder: other. The halves are disjoint by construction and pinned by a test, so one request is never counted under both. No free text reaches an attribute -- the run's detail, name, hook id, session key and token id are all excluded. This extends RFC 6.4's list, which named only the recorded-run outcomes. TWO emitters, both through webhooks.py::count_request, and neither is redundant. WebhookRunStore.record counts each recorded run, outside its own try, since the series counts the DECISION and not the history row. dashboard/handlers/hooks.py::api_hooks_agent counts the requests it turns away before a run exists, with an inline count_request immediately above each of its 21 refusing returns -- recording there is impossible or self-defeating (the unreadable store is what would be written; the throttle exists to stop doing work). Inline rather than through a shared helper because the error-code contract gate requires a literal code in a dict literal at the json_response site, which a helper taking the body as a parameter cannot provide; completeness is guaranteed instead by test_webhooks_request_counter.py::test_every_refusal_branch_is_counted, which reads the function's syntax tree and fails if any answer reaches a caller without reaching the counter. The accepted path is NOT counted at admission: its run records its own terminal outcome later, and counting both would double-count it.
kirocrew.inventory.webhooks.registered observable gauge none metrics/inventory_gauges.py::read_registered_webhooks -- entry count of the webhook callback registrations in hooks.json, reduced by webhooks.registered_context_count, the same owner the dashboard's listing and its id-addressed delete use, so the gauge cannot report a number no surface can confirm. The file is shared with the script-hook store, whose entries are not registrations and are not counted. A MISSING file is a genuine 0 (the steady state of an install that uses no webhooks); a file that is present and does not parse yields NO observation and increments kirocrew.inventory.probe.failures{probe=webhooks}, the same fault/gap split crons.json draws. registered_context_count reads it without logging, so a file that stays corrupt is reported at WARNING once, by the probe, not on every export interval. No hook id, session key or context summary is read -- each is agent-chosen free text, and the count rides in the value.
kirocrew.cron.fires counter kind (agent / script / command — the dispatch shape; script and command bypass the model entirely, so this is the split between a job that costs tokens and one that costs none), trigger (scheduled / manual) cron_service/service/run_execution.py::_run_job_isolated (bound as CronService._run_job_isolated), one increment per execution, emitted BEFORE the jitter sleep so a run cancelled during jitter still counts as fired. No job id, name, message or schedule string reaches the attributes.
kirocrew.artifact.created counter kind (the closed ALLOWED_KINDS set), source (the closed ALLOWED_SOURCES set), kind_auto (bool — the kind was inferred from the content rather than pinned by the caller) artifacts_writes.py::create (bound as ArtifactStore.create), emitted after the write and its change notification, so a failed create contributes nothing. kind and source are exactly the values _validate_kind / _validate_source already close over, so no slug or free-form label can enter the series.
kirocrew.workflow.runs counter authored (bool — the run writes its own script from an intent instead of executing supplied source), replay (bool — a restart-subtree reusing cached agent results rather than re-calling the model) workflows/runner.py::WorkflowRunner.run, the one place every run passes before executing anything, foreground or background (the background entry point drives this same method). Both attributes are booleans over run mode; no intent text, script or run id is emitted.
kirocrew.work.created counter kind (goal / item), depth (0 / 1 / 2+ — the creating board's own depth) dashboard/handlers/work_ledger_conductor.py::api_work_ledger_record via _count_recorded, after the write's work/recorded entry landed in the crew log, which is the point the route itself treats as the commit. kind=goal counts ONLY the goal write that opened the board (its committed header carries goal_version == 1); a reword, a round bump, a cap raise and an explicit new_goal are all later writes and count nothing, so the series reads as boards rather than as goal writes. kind=item is one minted item. Declared in metrics/work.py; no goal text, title, acceptance or item id is emitted.
kirocrew.work.item_closed counter state (accepted / rejected / abandoned — the ledger's terminal states, since close refuses open), fails (0 / 1 / 2 / 3+), duration (le_10m / le_1h / le_4h / gt_4h, each edge inclusive: 600 s / 3600 s / 14400 s from the item's created_at to its closed_at, beyond which gt_4h) Same site, on action=close, from the committed item rather than from the request. A duration is a bucketed counter rather than a histogram so the distribution survives a backend that keeps no explicit buckets; the edges live in metrics/work.py::DURATION_BUCKET_EDGE_SECS. An unreadable or missing stamp yields duration=other rather than a plausible-looking zero. Counted at the close rather than in the crew log's work fold because that fold forgets a closed item once WORK_CLOSED_WINDOW newer ones close, keeping only a total — so state, fails and the duration exist only at the moment the close commits.
kirocrew.work.dispatches counter target_kind (session / crew) Same site, on action=bind — the action that names the target — after that bind's work/recorded entry landed. Deliberately NOT on the crew/dispatch echo written just after it: that echo is best-effort and is skipped entirely for a board driven from a slot belonging to no crew, so counting on it would drop every dispatch such a board made. session is the only kind the bind builds; crew is declared because crew_log/entry_types.py::CREW_TARGET_KINDS names it, so a second target kind lands in its own series instead of in other.
kirocrew.work.reports counter status (progress / done / blocked / question / reply / failed; reply only while work_ledger.remote_dispatch is on), reason (approval / needs_human / none when the report named none) dashboard/handlers/work_ledger_worker.py::api_work_report, after the report's work/recorded entry landed, from the committed item. A refused report — an unknown or closed item, a done carrying no evidence, a status outside the worker's vocabulary — returns before this point and contributes nothing. reason is carried as the literal none rather than omitted, so "no reason given" is distinguishable from a dropped attribute. failed is declared because crew_log/entry_types.py::CREW_REPORT_STATUSES names it; the ledger's worker half can commit only the others, so this route never emits it today. No summary or artifact text is emitted.
kirocrew.crew_mode.turned_on counter via (switch / create), repeat (bool — a Crew Mode chat already existed) dashboard/chat_api/agent_switch.py::switch_slot_agent after a committed switch from a non-conductor onto a conductor (via=switch), and dashboard/chat_api/slot_create.py::api_chat_slot_create for a new chat born on one (via=create). repeat is the repeat-use signal: true when the gateway already knew another Crew Mode root. No session key, title or agent name is emitted.
kirocrew.crew_mode.new_chats counter crew_mode (bool) slot_create.py::api_chat_slot_create, one per new chat. The share of new chats in Crew Mode is sum(kirocrew.crew_mode.turned_on) / sum(kirocrew.crew_mode.new_chats) over a window.
kirocrew.crew_mode.runs_finished counter state (accepted / rejected / abandoned) dashboard/handlers/work_ledger_conductor.py::_count_crew_mode_run, after an action=close on a board a Crew Mode root owns landed in the crew log. A run is one crew member item a Crew Mode chat dispatched.
kirocrew.crew_mode.steers_per_run counter steers (0 / 1 / 2 / 3+) Same site: the closed item's own steer events, bucketed. A bucketed counter rather than a histogram so the distribution survives a backend with no explicit buckets.
kirocrew.crew_mode.steers counter on_item (bool — the steer landed as a steer event on a ledger item) dashboard/handlers/crew_workers.py::api_crew_worker_steer, one per delivered person steer. No message text is emitted.
kirocrew.crew_mode.worker_waits counter status (blocked / question), reason (approval / needs_human / none) dashboard/handlers/work_ledger_worker.py::api_work_report, after a committed report on a Crew Mode board that says the crew member is stuck or needs the person. Other statuses count nothing here. No summary text is emitted.
kirocrew.context.compactions counter success (bool — whether the compaction ATTEMPT itself completed) session_compaction.py::CompactionCoordinator._fire_compact_callback, one point per compaction that reached a verdict. Placed on this method rather than inside the callback because every compaction passes through here whether or not a surface registered a compact callback, and the early return below would otherwise drop exactly the surfaces that register none. The sum is compaction ATTEMPTS. success is deliberately NOT an effectiveness measure: the coordinator learns whether the backend finished the compaction, not how much context it actually freed, and no before/after context reading is taken, so a compaction can complete while reclaiming little. The counter's success is also not the callback's. _recycle_held fires this funnel with success=True because the SESSION now has headroom — which is what the callback needs to know — but it is reached exactly when an in-place /compact FAILED and the provider had to be replaced instead. Counting that as a successful compaction would report the failure mode as the success case, so the counter reports False there. The recycling marker is set for the whole of _recycle_held and popped only after this call, which is what separates the two populations at this point; test_a_failed_compact_that_recycles_is_not_counted_successful pins that ordering.
kirocrew.mcp.reconnects counter pool (bool — the reconnecting stub was a warm-pool bridge rather than a directly owned one) mcp_gateway/stub_link.py::_reconnect, one increment per stub reconnect to a restarted gateway daemon, emitted after the replayed handshake is delivered. Each increment is a bridge that died and was rebuilt under a live session, so a rising rate is daemon instability that sessions are absorbing silently. The daemon detail string and the per-session reconnect ordinal go to the log line only, never the attributes.
kirocrew.mcp_gateway.liveness.escalated_probes counter answered (bool — the daemon replied to the longer probe after missing the fast 2 s ping) mcp_gateway/manager.py, once per escalated liveness probe. answered=true is a loaded-but-alive daemon, the precursor to a kill cycle; false missed both probes
kirocrew.mcp_gateway.liveness.escalated_latency_ms histogram (ms) process (gatewayd) mcp_gateway/manager.py, recorded only when the escalated probe was answered (a miss has no latency), so the _LIVENESS_ESCALATED_TIMEOUT_SECS bound can be checked against field data
kirocrew.approval.decisions counter decision (allow / auto_approve / deny — the gate's own bounded action constant), security_deny (bool — separates a hard security refusal from a policy-state deny) hook_runtime/hook_types.py::ToolHookResult._count, called by the four result factories and nowhere else, so it fires exactly once per gate consultation. A surface that OVERRIDES the gate constructs a result directly (the chat runner downgrading an auto-approve to an interactive card, for instance) and is deliberately NOT counted: counting every construction would report one request as two decisions and keep a count for a verdict that was discarded. Counting from the factories rather than from __post_init__ behind a flag is what removes the flag — it had no reader but the counter, so it was state carried purely to signal. Both attributes are bounded by construction — no deny reason, tool name or command reaches the recorder.
kirocrew.adaptive.decisions counter action (decrease / increase / pause / probe / resume) adaptive/controller.py::AdaptiveController.apply; emitted only when adaptive concurrency changes a cap or pause state, not on hold/fixed ticks.
kirocrew.sel.prune_failed.count counter — heartbeat.py; one point when the daily Security Event Log retention prune raises.
kirocrew.taskq.depth histogram (1) state (a taskq.model state name — the closed model.STATES set, terminals included: the sampler reads a bare GROUP BY state) dashboard/session_health.py::SessionHealthMonitor._sample_metrics, one observation per state per health computation: the durable queue's depth as the sampler saw it. The aggregator's last/max reading per state is the gauge.
kirocrew.taskq.oldest_wait_secs histogram (s) — same sampler; age of the oldest row still waiting for dispatch (TaskStore.oldest_wait_secs).
kirocrew.taskq.completions counter outcome (the terminal state name) taskq/store.py::TaskStore.transition on every terminal write and TaskStore.cancel; the rate over it is the completion rate.
kirocrew.taskq.effective_cap histogram (1) lane_kind (subagents, spawn_gate, or a name a controller registers — code-bounded) same sampler; the cap currently ENFORCED per lane, equal to the user maximum only when nothing is degraded.
kirocrew.taskq.pressure_reason counter reason (the controller's closed reason enum) same sampler, one increment per health computation taken while a degrade reason is active.
kirocrew.crew_log.savepoint.write counter kind (session / slot / other), outcome (written / failed), bucket (lt_10ms / lt_100ms / lt_1s / ge_1s) crew_log/checkpoint.py::save (session) and ::save_slot (slot), one point per pass that actually offered a payload to the savepoint store — roughly once per 256 entries. A pass that returned before writing (no write earned yet, no witness, the unit removed, the lease contended, an unserializable state, every fold sitting at another boundary) records NOTHING: those are refusals, and counting them as failures would read as a failing disk. outcome compares what landed against what was OFFERED (written == attempted), not against the whole bundle — a session pass skips any fold whose boundary its witness does not certify, and holding the pass to those too would report a refusal as a failing disk. Whether the bundle may then CLAIM a savepoint is a stricter question answered separately in save. Both points are taken in a finally, so a raise through the store counts as failed rather than going unrecorded, and both are emitted after release_lease so the first (synchronous) recorder build cannot lengthen the window in which a removal of that unit is refused. bucket bands the store write itself from a time.monotonic() pair; both call sites are on the fold and read path, never a write thread. No unit id, slot key or fold name is an attribute.
kirocrew.crew_log.eager.queue_depth gauge (observable) — metrics/crew_log_health.py::read_eager_queue_depth → crew_log/eager.py::queue_depth, the live Queue.qsize() read when a reader collects. A gauge from live state rather than an up/down counter, so a crash cannot leave it drifting. Absent, not zero, when this process has committed no eager entry: there is no fold thread and no queue, which is a different fact from an empty queue. The reader never calls _ensure_worker, so a telemetry collection cannot be the thing that starts the fold thread.
kirocrew.crew_log.writer.buffered gauge (observable) — metrics/crew_log_health.py::read_writer_buffered, WriterStats.buffered read live at collection — crew log entries the writer is holding, the backlog number that distinguishes a stale board from a wedged disk. Same two rules as the queue gauge: live state rather than an up/down counter, and absent rather than zero when no writer was built in this process (it reads the installed instance directly rather than through the accessor, which would build one).
kirocrew.crew_log.bus.subscriber_error counter kind (fold_advanced / tree_advanced / other) crew_log/bus.py::publish and ::_call, the two branches that swallow a subscriber's exception (mutually exclusive — _call serves only the baseline replay in subscribe, so one exception is never counted twice). Failure path only, and the count is what the log line cannot give: the publisher is told nothing by design (it has already done the fold), and a WARNING per failed event carries neither a rate nor a total, which is what an operator asks of a subscriber failing on every fold. The bus has no queue — an event with no subscriber is dropped by design — so there is no backlog to measure. This is the one counter on this table a crew log WRITE thread can reach: tree_advanced is published from the session-tree projection, which a session/opened or session/spawned job advances while the write-behind is running that job. Three things bound it — a subscriber that returns normally costs nothing at all, the raising path already formats and emits a WARNING that costs more than the counter beside it, and the counter resolves this module from sys.modules rather than importing it, so a write thread can never trigger the first load of the metrics facade. fold_advanced is published only from the eager fold worker. The exception text and the subscriber's identity never become attributes; a kind the enum has not been taught maps to other.
kirocrew.host.procs_peak / kirocrew.host.fds_peak / kirocrew.host.rss_peak_mb histogram — declared for the host budget's peaks since the previous sample (emitter: the budget snapshot, wave-3 integration).
kirocrew.loop.lag_ms histogram (ms) process (gateway / gatewayd) adaptive/controller.py::AdaptiveController._sample_and_tick, one observation per controller sample: how late the sample_secs timer fired on the gateway event loop, the same number the controller feeds its loop_lag signal. The cap decreases and pauses that signal drives are logged at WARNING.
kirocrew.recovery.attempts counter layer (L1_tool_call … L5_gateway), action (retry / escalate / notify / give_up) recovery/ladder.py::RecoveryLadder.observe_failure, one per decision.
kirocrew.recovery.escalations counter from_layer, to_layer same, one per hand-up.
kirocrew.recovery.duration_secs histogram (s) layer RecoveryLadder.observe_success: first failure of the run → the success that closed it.
kirocrew.recovery.restarts counter layer RecoveryLadder.record_restart (the gatewayd watchdog after each respawn; backend/runtime rebuilds at their sites).
kirocrew.gateway.boot.duration histogram (ms) server (dashboard / api), outcome (ready) dashboard/server.py::start_dashboard / dashboard/server_runtime/api_entrypoint.py::start_api_server — boot-to-ready: wall-clock from the server's start_time until full init completes and it is about to accept traffic. Emitted via metrics/http_metrics.py::record_boot_to_ready. Best-effort; never blocks startup.
kirocrew.gateway.request.duration histogram (ms) method (fixed HTTP-verb allowlist, else OTHER), route_template (matched aiohttp canonical TEMPLATE, e.g. /api/artifacts/{slug}, else __unknown__), status_class (1xx..5xx / other) metrics/http_metrics.py::make_route_latency_middleware — outermost gateway middleware on BOTH start_dashboard and start_api_server. Times full in-gateway HTTP handling; upgraded WebSocket connections and text/event-stream SSE responses are excluded so connection/turn lifetime cannot pollute request latency. Bounded cardinality (see below).
kirocrew.crew.session.started counter agent_kind (closed: the owned agent-file stems, plus default, crewmate, other) metrics/crew_log_export.py::_otel_usage_diff, from the crew log's otel_usage fold rather than from a session-lifecycle call site. Distinct from kirocrew.session.started above, which counts a gateway session allocation split by session_source: this one counts a crew LOG opening split by the kind of agent that opened it, and only for logs this process both wrote and could prove were opened after it installed the subscriber. Never over-counts across a restart; under-counts a session opened before install.
kirocrew.crew.turn.started counter actor (the crew log's own closed ACTOR_VALUES set), agent_kind (the same closed set as kirocrew.crew.session.started, for the session the turn ran in) Same module and fold; one add per turn/started entry, derived as a delta against the last cumulative render this process exported for that session. agent_kind is read from the fold's already-closed opened_agent_kind, never from the raw agent name, so turns run inside conductor and worker sessions can be counted apart.
kirocrew.crew.export.lag histogram (ms) — Same module, on_fold_advanced; one observation per delivery an _EXPORTERS row actually handled, so the session-scoped deliveries for other folds that reach the same callback are not timed. Self-timing, so "the export is not slow" is read rather than asserted. SDK default boundaries for now — this instrument is deliberately absent from _HISTOGRAM_BUCKETS_MS until a real install's deliveries are measured.
kirocrew.crew.session.tree.depth counter depth (closed: 0, 1, 2, 3+) Same module, from the crew log's otel_session_tree fold; one add per session this process can prove is new to it, bucketed by that session's BIRTH depth as session/opened.depth recorded it (crew_log/otel_session_tree_fold.py::depth_observation / depth_bucket). A counter per bucket label rather than a histogram, so the distribution survives backends that do not keep explicit histogram buckets. A session whose depth could not be established (the tree unseeded, the reading incomplete, a creator cited with no log, a cycle) carries no depth and is left OUT of the distribution rather than defaulted to 0, roots included, so it is never censored on one side only.
kirocrew.crew.session.tree.child counter parent_kind (the creator session's agent_kind, same closed set as kirocrew.crew.session.started), nth (closed: 1, 2, 3, 4-7, 8+) Same module and fold; one add per child session a creator makes, read from the session/spawned entries on the creator's own log. nth is the child's ordinal among its creator's children (otel_session_tree_fold.py::nth_bucket_counts). Creators with at least k children = the sum of adds at nth >= k, so the width distribution is read without waiting for a creator to end. Same never-over-count rule: a creator not provably new to this process is baselined on first sight.
kirocrew.crew.session.tree.roots gauge — crew_log/otel_session_tree_roots.py::register_session_tree_gauge, registered from provider.py beside register_inventory_gauges / register_process_gauges and so collected on the reader's own thread. How many roots in the folded session tree have at least one child — the count of trees that actually branch, read by folding the in-memory tree once per collection (bounded by TREE_UNIT_CAP). A reading the tree cannot give yields a gap, never a 0, on the same condition the depth counter withholds an add.

| kirocrew.process.threads.python | gauge | — | metrics/process_gauges.py::register_process_gauges, callbacks run only at reader collection (no polling threads). threading.active_count(). | | kirocrew.process.threads.os | gauge | — | Same module; platform_compat.process_thread_count(os.getpid()) — OS-level count that catches native pools (ggml, grpc) invisible to threading. Linux-only; None elsewhere (gap, not zero). | | kirocrew.process.open_fds | gauge | — | Same module; delegates to platform_compat.count_open_fds (shared with gatewayd's zombie-diagnostic fd_count): /proc/self/fd or /dev/fd entry count minus the enumeration fd; Windows reports the kernel handle count (platform-dependent semantics). | | kirocrew.process.memory.rss_bytes / .peak_rss_bytes | gauge (By) | — | Same module; delegate to platform_compat.proc_rss_bytes (current) / proc_peak_rss_bytes (high-water mark), both cross-platform. A 0 return maps to None: gap, never a fake zero sample. | | kirocrew.process.cpu.seconds | gauge (s) | — | Same module; platform_compat.proc_cpu_seconds process-lifetime user+system CPU. An observable GAUGE, not a counter: as counters these four were the only CUMULATIVE series exported, and one cumulative series forces every consumer into whole-hour stateful aggregation. Listed in process_gauges.LIFETIME_TOTAL_METRICS, which the dashboard aggregator reads to reduce them window-relative rather than reporting a lifetime total as a current reading. A 0 return maps to None: gap, never a fake zero (which the reducer would take as a new baseline). | | kirocrew.process.gc.collections / .collected / .uncollectable | gauge | generation (0/1/2) | Same module; gc.get_stats() per generation, process-lifetime totals reported as gauges (see the CPU row). Rules GC in/out of a leak diagnosis (rising uncollectable = reference cycles; flat collected with rising RSS = native leak). | | kirocrew.process.memory.rss_sampled | histogram (By) | process | adaptive/controller.py::AdaptiveController._emit_process_histograms, one observation per controller sample, from the HostSample.rss_mb the loop already probes for its own decisions. A reading of 0.0 publishes nothing: platform_compat.proc_rss_bytes answers 0 on failure by contract rather than raising, so the probe's own except never fires and a failed read would otherwise enter a CUMULATIVE distribution as a fabricated zero that no later sample can correct — and a live process never has a true resident set of zero, so the strict predicate discards nothing real. The same quantity as the rss_bytes gauge above, kept as a SECOND instrument because merging a gauge across instances retains only min/max/mean — a fleet-wide p90 is not recoverable from it at any storage layer, while histograms sharing bucket boundaries merge element-wise. Recorded from the controller rather than from process_gauges because OTEL has no observable histogram: a histogram needs a caller on a timer, and this loop already is one. Own bucket family _RSS_BUCKETS_BYTES. | | kirocrew.process.cpu.utilization | histogram (1) | process | Same emitter and cadence; share of the whole machine as a ratio where 1.0 is every logical core saturated. Computed client-side by process_gauges.cpu_utilization from two consecutive HostSample.cpu_seconds readings over the interval between the instants those two readings were TAKEN at, divided by read_logical_cores(). The instant is HostSample.cpu_clock, read inside probe_host beside the total rather than on the event loop after the worker thread resumes: the resumption delay varies from tick to tick so it does not cancel, and at the one-second floor of controller_sample_secs a few hundred milliseconds of it is a double-digit-percent error in a published value nothing downstream can correct. Computed here rather than downstream because process.cpu.seconds is a lifetime TOTAL: a consumer would have to difference consecutive samples per process lifetime and detect restarts to recover a rate, and once that gauge is merged across instances there is nothing left to difference. Returns a gap, never a fake zero, when either reading is <= 0 (a failed probe reads 0.0), the clock did not advance, the core count is unknown, or the total went backwards (two different processes). The first tick of a process has no predecessor and so publishes nothing. Values above 1.0 are NOT clamped — nothing samples the clock and the kernel's accounting at the same instant — which is why the bucket family carries a bound above saturation. Own bucket family _CPU_RATIO_BUCKETS. | | kirocrew.inventory.crons.active | gauge | — | metrics/inventory_gauges.py::register_inventory_gauges, callbacks run only at reader collection. The enabled-count reduction is cron.enabled_count_from_disk, a module-level sibling of unhealthy_jobs_from_disk that returns (count, loadable) and is the single owner of that loop — CronService.count_enabled_from_disk calls it too and keeps only the count, so the probe cannot drift from what the scheduler considers enabled, and neither needs a running gateway. Deliberately NOT list_jobs: that re-arms the asyncio timer, which from a reader thread with no running loop raises AND cancels the live timer first, stopping every scheduled job. A missing store is a genuine 0 and no fault (a fresh install has no crons). A store that is present but unparseable yields a gap and increments probe.failures{probe="crons"} — _read_job_records calls that case a fault, and a silent gap would be indistinguishable from a host that stopped exporting, which is the confusion the counter exists to resolve. | | kirocrew.inventory.monitor_loops.active | gauge | — | Same module; autonudge.get_instance() then a materialized list(...) snapshot of the loop registry counting active. The registry is mutated only under an asyncio.Lock on the event loop, which gives a reader thread no protection and cannot be acquired from it; the snapshot is safe anyway because each dict op is atomic under the GIL and a gauge stale by one loop is fine. get_instance() returning None (spawned agent process, unit test, auto-nudge off) yields a gap — a 0 there would be indistinguishable from a running service with none armed. | | kirocrew.inventory.skills.installed | gauge | — | Same module; one module-global SkillsLoader(install_builtins=False) (install_builtins=True would make a metrics probe perform a content-hashing write sync). Listing is a recursive walk, so it is cached for _EXPENSIVE_TTL_SECS (300s) — five collection cycles on the default interval. One combined total, NOT a builtin/user split: builtins are copied onto disk into the same tree, so nothing in the listing distinguishes them and a split would be a guess. | | kirocrew.inventory.memory.migrated | gauge | — | Same module; memory.migrated as 0/1. Reports the migration bit rather than a "memory enabled" switch because no such switch exists — the embedding provider is coerced to a real value on load, so memory is structurally always on and an "enabled" gauge could only read 1. The bit flips on the first boot that initialises vector memory, fresh installs included, so 1 is the healthy reading and a 0 on a live install is a fault: vector memory never initialised, migration aborted before the flip, or the flag write was skipped on an unparseable config.json. That is what keeps it from being a rollout metric that goes quiet once a fleet converges. | | kirocrew.inventory.knowledge.documents | gauge | — | Same module; the raw source count. Deliberately NOT a constant 1 under a magnitude-band attribute: the quasi-identifying argument for banding does not hold when the only destination is the operator's own collector describing their own machines, while the band costs a series per band, hides growth inside a band (the drift the gauge exists for), and fixes the boundaries at emit time — a backend can band a raw count at query time and cannot recover a count from a band. Counts the sources table, not items (an item is one chunk, so counting items would report a chunking artifact as a document count), via a read-only sqlite3 URI guarded on exists(): constructing a KnowledgeStore would run schema init plus the writer-locked orphan migration and would CREATE the database on an install that never ingested anything. Cached 300s. Absent database yields a gap. | | kirocrew.inventory.lessons | gauge | — | Same module; the raw lesson count of the global (V1) lesson store, same shape and rationale. Private Memory V2 crew-member lessons live only in each member's database and are not counted. Reads through one module-global LessonStore, whose own mtime cache makes a repeated read on an unchanged file a single stat — reused across ticks because that cache lives on the instance, and left uncached here because a second cache would add only staleness. An absent file is a genuine 0 (an install that never saved a lesson), unlike the knowledge database. | | kirocrew.inventory.mcp.servers | gauge | class (first_party / third_party) | Same module; the roster merged from the agent config and _load_mcp_json_by_source, classified against mcp_discovery._MANAGED_SERVER_NAMES — reused rather than restated, since a second name list here would drift silently the first time a managed server is added or renamed. Server names are read to classify and then discarded: no name ever reaches an attribute. A roster is user-chosen free-form text, so publishing it would be both a cardinality bomb and a disclosure of what the user has installed. An empty merged roster yields a gap: the loaders fail soft to {}, so empty cannot be told apart from an unwritten config, and the managed servers are always present once installed. | | kirocrew.inventory.config.toggle | gauge | key (closed enum, inventory_gauges.CONFIG_TOGGLES) | Same module; each declared feature switch as 0/1. Config is read ONCE for the whole set, so the gauge costs one fingerprint-cached load per collection instead of one per key. telemetry.enabled is deliberately absent — this module only runs inside the consent gate, so it could report nothing but 1. memory.migrated is absent because it has its own gauge. A key whose field cannot be resolved is OMITTED rather than reported as 0 (a renamed config field must read as a missing series, never as a switch someone turned off); test_every_declared_toggle_resolves_against_a_real_config is what keeps that from being a silent hole. | | kirocrew.inventory.members | gauge | — | Same module; read_members, the count of crewmates the PERSON created on this install, reduced by members.count_person_created_crewmates. It walks the same rows GET /api/members builds (a dispatchable name with a resolvable slug) and counts the ones members.is_person_created_crewmate accepts, which is the roster's own dashboard_created rule (source == "kirocrew" AND a member id), so the gauge cannot drift from what the Crewmates page lists as made by the person. Mate itself (source == "builtin"), package-synced and app-made crews, legacy rows and the reserved default member are not counted: package-synced rows are the large majority of the raw roster on a busy host and would bury the number. Cost is one fingerprint-cached config load per collection, uncached like the config toggles. No name, slug or member id is published; the count rides in the value. An empty roster is a real 0. A config read that raises, or a half-read config (_base_unreadable or _overlay_unreadable), records a members probe failure and yields no observation: the loader keeps going on a file it cannot parse, so the crewmates in that file would otherwise drop out and publish a false decrease. One case it cannot separate: a file that parses with a non-object agents value, which the loader reads as empty without marking it degraded, so it reads 0 like an install with no crews. |

| kirocrew.inventory.probe.failures | gauge | probe (closed enum, inventory_gauges.ALL_PROBES) | Same module; times a probe raised, per probe. Absent on a healthy install — it publishes only once something has failed, so a healthy host adds no series, and roster assertions must exempt it (_HEALTHY_ABSENT in test_otlp_wire_e2e.py). It exists because absence has TWO readings that this family cannot afford to conflate: a missing series can mean the probe broke OR that the host stopped exporting, and distinguishing a drifted host from a dark host is the job these gauges were added to do. A broken probe therefore presents as data — the failure series is present and climbing, which a host that went dark cannot produce. The first failure per probe also logs at WARNING (a default install's journal keeps WARNING and above, so debug would be invisible on exactly the installs that need it); repeats drop to debug so a permanently broken probe cannot emit one WARNING per export interval for the process lifetime. An observable GAUGE whose reading is a process-lifetime total, declared in inventory_gauges.LIFETIME_TOTAL_METRICS (see the process CPU/GC rows): as an observable counter it was the last CUMULATIVE series exported, and one is enough to force every consumer into whole-hour stateful aggregation. |

All twenty registrations are wired in provider.py::_build_recorder (live path only) and wrapped so a gauge failure can never disable telemetry as a whole. The two families register in SEPARATE try blocks — they read entirely different subsystems, and the process gauges must not go dark because, say, the skills tree is unreadable. Each reader callback is individually guarded — a failing probe yields a gap for that cycle, never an exporter error.

Because the inventory probes read subsystems rather than process counters, cost is the binding constraint: a callback runs once per export interval per reader, so an install with an OTLP destination configured enters them on two ticker threads concurrently. Every probe is O(1) in-memory, a single small file parse, or explicitly TTL-cached (_ttl_cached, whose produce call runs OUTSIDE its lock so a second reader misses the cache rather than blocking on the first). A None reading is cached like any other answer, so an install that can never answer does not re-pay an expensive failing probe every cycle.

Scope of the kirocrew.inventory.* family, stated here because the names invite the opposite reading: it is OPERATOR-fleet telemetry, not project-wide adoption analytics. Collection is off by default, egress is a second opt-in, and OTLP needs the kirocrew[otlp] extra, so any aggregate describes one operator's opted-in machines. Adoption data structurally cannot ride this trunk — see "Why it is NOT part of the OTEL trunk" below, whose four reasons apply to these instruments exactly as they do to the beacon's payload. Do not later "promote" these to answer install-population questions.

One publisher per install (mark_install_reporter). _build_recorder runs in every process that touches telemetry, and an install runs several at once — the gateway, gatewayd, spawned agents and apps (which is why the local exporter shards per PID). That is correct for process-scoped instruments and WRONG for install-scoped ones: each process would publish the same "this install has 14 crons" under its own resource, so an aggregate over installs counts one install once per running process, and the bucketed gauges (whose shape is "publish 1, sum by bucket") would make one install look like N. Deduplicating downstream would work off the resource's install-scoped service.instance.id, but only as query-time discipline every consumer has to remember (see below, where that trade is what keeps this mechanism). So exactly one process publishes: the gateway calls inventory_gauges.mark_install_reporter() during startup, and every callback checks it at COLLECT time (not at registration), so it does not matter whether a recorder was built before or after the claim. A process that never claims it registers the same instruments and publishes nothing. The election is an EXPLICIT call rather than inferred from, say, autonudge.get_instance() being non-None (true only in the gateway today) — an inference stops holding the moment that service's startup moves or becomes conditional, and the failure mode is inventory silently disappearing.

That failure mode applies to the explicit call too, which is why the call SITE is part of the contract: it sits in GatewayOrchestrator.run() at a point no branch guards. A feature-flagged host does the same damage an inference would — the claim first shipped inside _init_autonudge(), which returns early under KIROCREW_AUTONUDGE=0, so one env var left no process publishing inventory at all. Nothing reported it, either: the callbacks return before probing, so probe.failures stays empty and the whole subsystem reads like a host that stopped exporting. test_reporter_claim_is_unconditional parses the gateway and asserts the call is reached from run() without passing through a conditional (try is allowed — telemetry must never block boot).

The resource's persisted install-scoped id is the precondition for ever removing this mechanism, and it is not by itself a replacement. An install CAN be deduplicated downstream (max by (install) over its processes) — but that is query-time discipline: the exported data would still contain one copy per process, and a naively written fleet sum would read N times the truth. The election makes the data correct at the point of emit, which no consumer has to remember. So retiring it is a deliberate trade (fewer moving parts here, a rule every dashboard must follow there), not something the identity settles on its own.

Scope of the guarantee: it elects one GATEWAY PROCESS, which is one install wherever the product runs one gateway per data home. Two gateways sharing a data home on different ports would both claim, because the dashboard's singleton is per port, not per data home — a pre-existing property of that configuration that duplicates every install-scoped fact, and one no flag can detect (a second gateway double-counts whether or not it runs crons). A pod is not this case: it boots with its own KIROCREW_HOME and copies no crons, sessions, or databases, so its inventory belongs to a different install and publishing it is correct.

The counters beside stats.py, and why they are not the same numbers

stats.py's in-memory singleton already counts adjacent ground: sessions_created, subagents_spawned and its tool-approval trio, all consumed by one dashboard system handler. Those tallies are NOT merged into the counters above, and the counters are not derived from them. The singleton is a process-local count that resets on restart and answers "what has this gateway done since it came up"; the instruments above are time-series carrying attributes, survive a restart, and answer "what is the rate, broken down by shape". Merging them would mean giving the singleton attributes and durability it has no consumer for.

The hazard the split creates is that the names are near-identical while the POPULATIONS differ, so the two numbers can legitimately disagree and be read as a bug. kirocrew.approval.decisions is the sharpest case: it counts gate VERDICTS, every consultation including the auto-approves and denies that never surface to a person, whereas the singleton's approval trio accrues on the surface where a human acted. kirocrew.subagent.spawned fires on the confirmed-start funnel after admission, so it excludes rejected spawns a request-time tally would include. Each row above therefore states which population its counter measures; a comparison across the two systems is only meaningful once those definitions are read.

Histogram bucket boundaries: per instrument, not shared

Boundaries live in two metric-name → boundaries maps in metrics/provider.py — _HISTOGRAM_BUCKETS_MS for millisecond instruments and _HISTOGRAM_BUCKETS_BY_UNIT for everything else — merged by histogram_bounds(), from which one View(instrument_name=…) is built per instrument. Ten families, each sized to its instrument's measured range:

Family Range Unit Instruments
_FAST_BUCKETS_MS 0.5ms – 60s ms gateway.request, db.query, mcp.backend.acquire, skill.lazy_load, context.section, telegram.api, embed.queue_wait, embed.inference, mcp_gateway.liveness.escalated_latency_ms
_STARTUP_BUCKETS_MS 1ms – 60s ms session.startup, chat.first_token, mcp.lazy_load, gateway.boot
_TURN_BUCKETS_MS 1s – 1h ms turn.duration
_WATCHDOG_IDLE_BUCKETS_MS 1s – 4h ms watchdog.idle
_TOOL_CALL_BUCKETS_MS 0.5ms – 1h ms tool.call.duration
_SESSION_BUCKETS_MS 1s – 7d ms session.duration
_CREDIT_BUCKETS 0.025 – 2500 credit turn.credits
_USD_BUCKETS 0.001 – 100 usd turn.cost_usd
_RSS_BUCKETS_BYTES 32MiB – 16GiB By process.memory.rss_sampled
_CPU_RATIO_BUCKETS 0.001 – 1.25 1 process.cpu.utilization

_TOOL_CALL_BUCKETS_MS spans a sub-millisecond builtin read to a multi-minute build in one instrument, so it keeps both ends: a _FAST ceiling would floor every long shell command into the overflow bucket, and a _TURN floor would collapse the entire builtin population onto boundary one. _SESSION_BUCKETS_MS is the only family measured in hours and days — a speculative session removed before its first turn lives seconds, while a dashboard tab left open across a working week is ordinary — so it is densest from a minute to a few hours where interactive sessions land, keeps sub-minute bounds because the short-lived teardown paths (unclaimed, destroyed) are a real population, and carries a 7-day ceiling so a long-lived tab is not floored into overflow.

Why the unit map is separate. The dashboard's generic histogram aggregation reports a statistic under *_ms keys unless the emitting module declares a non-millisecond unit for that instrument, and the frontend formats those keys with a millisecond suffix, so _HISTOGRAM_BUCKETS_MS membership is a claim that the instrument really is milliseconds. A credit or a dollar amount registered there would make both the claim and the rendered value wrong. The two billing instruments are claimed BY NAME by _aggregate ahead of the generic branch and reported inside the turn block via _amount_stats, under unit-neutral keys (p50, p90, total, …) each carrying its own unit — claimed by name because the turn block adds a per-model attribution split the generic surface has no shape for, not for the unit.

The two sampled process histograms have no dedicated block to be claimed into, so the generic branch handles their unit itself: it reads events.NON_MS_HISTOGRAM_UNITS through _non_ms_histogram_units() and routes a named instrument through _amount_stats instead of stats(). That mapping lives with the emitter for the same reason LIFETIME_TOTAL_METRICS does — the module that declares an instrument is the one that knows what its reading means — so no unit is re-spelled by a reader. test_provider_bucket_views.py fails when an entry in _HISTOGRAM_BUCKETS_BY_UNIT is missing from that mapping, which is what keeps a new non-ms histogram from silently reaching the *_ms surface. _amount_stats also rounds to six decimals rather than one, because a sub-cent per-turn cost rounds to 0.0 at the duration family's precision.

Each amount also carries by_model: the same total split by the emitted model attribute, as {model, count, total, mean} rows sorted by total descending. Without it the attribute would be recorded by the emitters and then discarded at the one place that reports the amount, leaving "which model spent this" unanswerable from the API. Percentiles are deliberately NOT repeated per model — that would add a bucket array per row to answer a distribution question nobody asks of a single model — and the split is never truncated, the same rule (and reason) as the cost_breakdown.by_model block below. A turn whose model was empty is counted under unknown rather than dropped, so the rows always sum to the pooled total beside them.

How the completeness guard finds instruments. An instrument missing from both maps silently inherits OTEL's default 0–10000 boundaries. test/metrics/test_provider_bucket_views.py does not detect histograms by a .duration name suffix, which is a convention rather than a property: it would miss kirocrew.embed.queue_wait / kirocrew.embed.inference (millisecond histograms without the suffix) and by construction any non-duration one. The guard walks the source by AST for histogram(...) calls, reading each instrument's name and its unit= keyword, and asserts: every emitted histogram is registered in exactly one map, every registered entry has an emit site, the ms map holds only unit="ms" instruments, the unit map holds none, and no single name is emitted with two different units.

Exponential histograms are not usable here yet. They would remove the hand-tuned arrays, but the local JSONL shard is serialized with the SDK's to_json(), and an exponential data point carries scale/zero_count/ positive instead of bucket_counts/explicit_bounds. The dashboard aggregator classifies a point as a histogram by the presence of bucket_counts and reads percentiles from explicit_bounds, so an exponential instrument would be dropped from the payload entirely — a silent absence, which is worse than a coarse bucket. Adopting them requires teaching the reader that shape first.

Why not one shared array. A single 1ms–60s array serving every histogram through a catch-all View(instrument_type=Histogram) has a ceiling sized for session startup, so a kirocrew.turn.duration sample (227589ms — an agent turn is a whole agent loop including tool round-trips and any wait on an interactive approval) lands in the +Inf overflow bucket. Since _pct_from_buckets can only report an overflow bucket's LOWER bound, the aggregator returns p50 == p90 == 60000 — a ceiling artifact presented as a real latency, while mean/max (exact, from sum/max) stay correct beside it. p50 == p90 is the signature of this failure. Instruments in this system span six orders of magnitude (sub-ms pooled acquires to multi-minute agent turns), so one array must sacrifice either fast-end resolution or slow-end truth.

Why there is no catch-all fallback. The OTEL SDK applies EVERY matching View, not the first. A per-instrument View plus a catch-all therefore publishes the same metric name twice with different bounds. The SDK offers no negation in View matching, so the catch-all had to be removed rather than narrowed.

Boundary generations never merge. Bounds are baked into each data point at record time, so any boundary change makes the 14-day scan window straddle two incompatible generations of one metric. handlers/telemetry.py::_Hist therefore groups data points by their EXACT explicit_bounds and reports statistics from a single group — the one holding the newest data point. Selection is by recency, not volume: majority selection would let a stale generation keep winning while it out-counted the new one, so right after a boundary change the old bounds would be reported for up to the whole 14-day window — for turn.duration that means continuing to serve the ceiling-pinned percentiles this grouping exists to remove, while omitting the new samples. Recency makes the change take effect on the first post-change sample; the reported population is then small but truthful, and count says so. (count remains a tie-break for data points carrying no time_unix_nano.) Outcome tallies are grouped too — accumulated inside _Hist rather than alongside it, because scoping only the buckets and count would leave the outcome breakdown summing across generations: the page would show N turns beside an outcome bar totalling more than N, and a fault_rate computed over a different population than the latency next to it. other_generations (0 = a clean window; >0 = the window straddles a boundary change and only the dominant generation is reported) and total_count (samples across EVERY generation) are therefore returned by _Hist.stats() itself, so they travel with every set of numbers they qualify — the startup blocks, the turn block, each other histogram, and each per-attribute split.

The dashboard renders the PAIR, not the generation count: it shows "showing 1,134 of 2,926 samples" beside the affected figure. A generation count is an internal unit a reader cannot convert into missing data, so "1 older generation" left a truncated n=1134 unreconcilable against the 2837 hit counter next to it, while the shown/total pair is directly comparable to both. other_generations remains the structural fact (how many incompatible groups the window holds) and stays in the response for diagnostics.

They are deliberately NOT pasted on by the response builder per block. That is how it shipped, and the generic other instruments were never given the field: the MCP acquire card reported one generation's n (1,154 of 2,926 real samples) beside a full-window counter, with nothing anywhere saying a generation had been dropped. A statistic and the caveat that makes it readable are one value, not two.

This is load-bearing, not defensive: the historical shared array and _TURN_BUCKETS_MS have the SAME bucket-count length, so a length-only check would have merged them positionally — adding a pre-change sample from the old +Inf bucket into the new +Inf bucket and reporting p90 = 3,600,000ms (one hour), while a 5s sample landed in a 5-minute bucket. Grouping also keeps count/sum/min/max consistent with the percentiles; accumulating those across generations while only one generation's buckets survived would describe a mean over one population and percentiles over another.

Completeness is therefore load-bearing. With no catch-all, a histogram missing from the map silently falls back to OTEL's default 10s-ceiling boundaries — reintroducing the same class of bug. test/metrics/ test_provider_bucket_views.py scans the source for kirocrew.*.duration metric names and fails when one has no map entry (and when a map entry has no emitting call site). It also pins the no-duplicate-streams property and asserts the 227589ms regression sample does not overflow. When adding a duration histogram, add it to _HISTOGRAM_BUCKETS_MS.

Bounded cardinality of kirocrew.gateway.request.duration (rec #1)

The per-route latency label route_template is never the concrete request path, query, id, or body — it is the aiohttp route TEMPLATE (/api/items/{item_id}), whose {…} placeholders are constants baked into the route table. The bounding is structural: collect_route_templates(app) snapshots the finite set of registered templates once (lazily, on first request, after all routes — including edition-contributed and post-middleware routes — are present), and route_template() returns a value ONLY if it is a member of that frozen set; anything else (an unmatched 404 aiohttp SystemRoute, or a template not in the snapshot) collapses to the single sentinel __unknown__. Therefore the distinct route_template label values are bounded by len(known_templates) + 1, a constant fixed at startup that cannot grow with traffic. Combined with the fixed method allowlist (≤ 8 values) and the fixed status_class domain (6 values), total series are bounded by (len(known_templates) + 1) × 8 × 6. The test test/metrics/test_gateway_http_metrics.py::test_bounded_cardinality_under_many_distinct_ids proves this against real OTEL data points: 100 distinct ids yield exactly ONE route_template value. Privacy: the only request-derived labels are method / route_template / status_class — no prompt, content, token, path, query, user id, or secret is ever recorded, and every string label still passes the recorder's redact() guardrail.

Dashboard handler

dashboard/handlers/telemetry.py — GET /api/telemetry/startup scans the JSONL shards over ONE reporting window shared by every block of the payload. The window is ?days=N (rolling, ending now) or ?since=&until= (a fixed range in epoch seconds; the panel sends local midnights, so until is the midnight after the last picked day), resolved by usage.resolve_window. A request naming no window (the panel's Default choice, where it sits until the reader picks) answers exactly as before the range control existed: the OTEL and context blocks cover the last 14 days (_WINDOW_DAYS) and the spend block the last 7 (SPEND_WINDOW_DAYS); the payload says so with window_default: true and cost_window_days. A named window drives every block, and is CLAMPED, never refused: a rolling days to at least one day, the start to usage.MAX_WINDOW_DAYS (90) before now, the end to now rounded up to the minute (so a range running to "today" keeps one until between 5s polls), an inverted range swapped, and a collapsed one widened to the day before its end. The panel's custom picker offers start days back to MAX_WINDOW_DAYS - 1 days before today, so a picked range holds at most 90 whole dates, all inside the clamp. The effective window is echoed as window_days, window_start/window_end (UTC ISO-8601) and window_rolling, beside metrics_retention_days (telemetry.retention_days, 0 disables age pruning; telemetry.max_total_mb can still drop the oldest shards by size), which the panel uses to say when OTEL figures cannot reach as far back as the window and to title those cards with the span they do cover. The turn drill-down (/api/usage/turns) is fetched over the echoed bounds, not the request, so it lists exactly the turns its row summed. Shard FILES are picked by their UTC day. With a named window each data point is then kept only when its own time_unix_nano falls inside it (a point with no timestamp keeps its shard's day); with no window every point in the picked shards counts, exactly as on main. The cache keys on the window, that mode, plus the shard fingerprint; a rolling window keys on its length, so the moving start costs no cache miss and the 30s TTL bounds its drift. Aggregation is offloaded via asyncio.to_thread; it aggregates the startup histogram into p50/p90 split by cold/warm (spawned attr) + outcome + daily series, the turn histogram into a turn block (stats + outcome counts + fault_rate), and generically surfaces every other kirocrew.* metric (other list) so new emit call-sites appear without a handler change. Scalar (non-histogram) metrics in other are classified by the SDK's own JSON markers — a Sum's data block carries aggregation_temporality/is_monotonic, a Gauge's carries neither. DELTA sums keep summing across cycles/PIDs; CUMULATIVE sums and the lifetime-total gauges named in process_gauges.LIFETIME_TOTAL_METRICS (process CPU/GC, whose newest sample is a running total rather than a state) buffer samples per (PID, process-identity, attrs) stream and reduce them time-ordered after the scan (shard iteration order is not chronological). Routing both shapes through one structure is also what keeps a lifetime total continuous across the shards written while those four were still observable counters. The identity half of the key is the resource-level kirocrew.process.start_time token the exporter stamps: a changed token for the same PID is a deterministic process boundary, so a reused PID starts a fresh stream even when the new process's first snapshot already exceeds the old maximum — the one shape value-based detection cannot see — while an unchanged token across provider rebuilds stitches the rebuild segments into one stream. Within an identity-keyed stream a value below the running maximum is treated as shard garbage (one identity is one OS process, whose counters are monotonic), never banked as a reset. Only a STRING token counts as an identity: a corrupt shard carrying any other type there reads as identity-less rather than minting a stream that mutes the heuristic. Identity-less streams (legacy shards, or platforms without a start-time read) reduce under the counter-RESET value heuristic — a snapshot below the stream's own maximum marks a process boundary and banks the finished segment, while re-emitted snapshots at/above the maximum are no-ops — so provider rebuilds stay idempotent and pre-change shards keep their exact totals. Either way the stream's first in-window sample is subtracted as a baseline so a process older than the window reports only in-window activity, never its lifetime total (stream total = banked + live segment - baseline; add across streams). Non-finite scalars (json's Infinity/NaN literals) are rejected per point via one shared coercion helper; gauges emit kind: "gauge" with latest (the newest sample, never a sum). Gauge samples are keyed per exporting shard PID so concurrent processes (gateway + MCP daemons) never collapse into one series: a single-PID window keeps the plain shape, a multi-PID window reports the newest process's reading as latest and a pid=-keyed by_attr breakdown. Malformed shard records degrade per-point (a garbage value skips that point, a garbage timestamp sorts oldest) rather than failing the endpoint. Percentiles are interpolated from bucket counts (made meaningful by the DELTA temporality + explicit-bucket View). Security: the user-configurable telemetry.local_dir and each shard pass validate_file_path (sensitive-path check) before any read. Cross-process: metrics are emitted by the ACP/gateway processes, so reading the durable shards is the only correct path (an in-memory reservoir in the dashboard process would never see them).

GET /api/telemetry/collection (api_collection_status) is the small companion route behind the recording switch. It reports the effective enabled state, the metrics directory (metrics_dir), whether an env var (env_pinned / env_var) or a config.local.json overlay (overlay_override) pins the setting, and whether an OTLP endpoint is configured (otlp_configured). It is deliberately separate from /api/telemetry/startup, which parses every metric shard in the window to aggregate percentiles — far too much work for a panel that only needs to know whether a switch is on. otlp_configured reports only THAT an endpoint exists: the endpoint string never leaves _telemetry_cfg(), because it can carry credentials in userinfo or query parameters. It is what lets the panel disable the enable direction rather than offering a write that comes back 409, while leaving disable available — which is where an opt-out matters most. Nothing on this route is an egress control, so unlike the beacon there is no governance ceiling to report.

Both routes report the EFFECTIVE state, not the stored config flag. _telemetry_cfg() resolves consent the way the collector does — env_pin() from metrics/provider.py overrides telemetry.enabled — so enabled on /api/telemetry/startup and /api/telemetry/collection cannot read "off" while metrics are being written, or "on" while nothing is. The pin comes from the provider rather than a second read of the env var here, because two resolutions are two things to keep in sync and a control that disagrees with the collector about what "on" means is worse than no control. env_pinned is what lets the panel's switch disable itself instead of offering a write the collector ignores.

other histogram splits (_OTHER_SPLIT_ATTRS). An other histogram also carries a splits map ("attr=value" -> the same stats shape) for a NAMED set of low-cardinality attributes — currently warm only. This exists so one side of a split can be reported alone: the dashboard's cold-spawn figure is acquire.splits["warm=false"]. Splitting on every attribute present was rejected because gateway.request.duration carries method+route, which would grow one sub-histogram per endpoint and force an arbitrary truncation cap on the payload; a named boolean keeps the split two entries wide with no cap.

Note that kirocrew.mcp.lazy_load.* is NOT the cold-spawn signal even though its name suggests it. It is emitted only from the legacy pre-ensure_backend spawn path, which modern stubs never take, so it records nothing on a current deployment (0 data points across 47 shards / 12 days observed) while real cold spawns are recorded on the acquire histogram under warm=false. The instrument stays because that legacy path can still execute for an old stub; the dashboard does not read it.

Startup phase gating. Only the end-to-end startup point (phase absent, as on the claude path, or phase=total from the kiro path) feeds the startup totals — count, cold/warm split, outcome, daily series, and the bucket distribution. Per-phase points are aggregated separately into startup.phases[] ({name, count, p50_ms, p90_ms, …}). Counting them as startups multiplies the startup count by the number of phases and sums several unrelated latency distributions into one set of buckets, which renders as a spurious multi-modal "distribution".

Failures are reported as counts, not as success rates. Both outcome instruments are fail-closed — AcpClient.ensure_ready and AcpProvider._start_kiro_runtime (providers/acp_runtime_start.py) each default their outcome to error and overwrite it with ready only on the success path, and _turn_outcome maps a real stop_reason — so a zero here is a measurement rather than an instrument that cannot report bad news. What a rate over these populations cannot do is survive rounding: a window of ~1400 startups makes one failed startup 99.93%, which renders as a flawless 100%. A rate had no reachable value between "perfect" and a problem big enough to clear half a percent, and it saturated in the direction that hides failure. A count has no such ceiling, so the page shows the absolute number of non-ready startups and of non-ok turns. Where a rate is still shown (turn fault rate, which answers a different question — faults per unit of work), a non-zero value below the rounding threshold renders as <1% and never in the success colour.

Both figures count everything outside the success value rather than naming the failure values. Enumerating them (error + timeout) silently excluded any third outcome — auth_required, and the unknown that shards predating the attribute aggregate under — which put the displayed count and the rate beside it on two different populations.

context block. The response also carries per-turn context-window occupancy — {turns, p50_pct, p90_pct, max_pct, sessions[]} — sourced from the per-turn token row store below, NOT from the OTEL shards: occupancy is a per-session ratio and slot keys are unbounded-cardinality, which must not become a metric label. sessions[] is EVERY session in the window ordered by peak occupancy (bounded only by a payload backstop of 500), each reporting peak plus the LATEST turn's identity (agent/model/surface) and absolute used/window. Rows whose window is missing or zero are skipped rather than defaulted. The block is null when no row carries the fields, and is served independently of the telemetry.enabled switch because those rows are always written (the panel therefore renders it even with OTEL export off).

The panel's source. website/src/pages/TelemetryPanel.tsx is the panel: the window control, the tab choice, the off, empty and error states, and the OTEL-shard figures (the health strip, the Latency and Startup tabs), which format through its fmtMs. Beside it, telemetryPanelModel.ts holds the payload types, the measured window (PanelWindowCtx) and usePersistedChoice; TelemetryPanelBlocks.tsx the sortable DataTable, Sums and Histogram; and TelemetryPanelConversations.tsx the Spend and Context tabs with the formatters and colour thresholds only they read. Every other module imports the panel, never an owner.

Per-turn token usage row store

Separate from the OTEL histogram sink above (~/.kiro/crew/metrics/, DELTA histograms for trends/alerting), the gateway also keeps a per-turn row store for cost and context analytics: one JSON object per model-spending turn appended to <data home>/usage/tokens/YYYY-MM-DD.jsonl (shards partitioned by the user's local date). It is always on (not gated by telemetry.enabled) and never egresses the host. Written by dashboard/handlers/usage.py::persist_token_record (sync) / persist_token_record_async (the chat hot path — builds the record on-loop, offloads the append via asyncio.to_thread); both are best-effort (exceptions swallowed, no fsync). Aggregated for the dashboard by _parse_token_history (30-day window, shard-fingerprint + 120s-TTL cache).

Each row (_build_token_record) carries:

field type meaning
_type str always "tokens" (record discriminator)
ts str ISO-8601 local timestamp of the turn
slot str chat slot / session key
provider str LLM backend (acp / claude_code / bedrock / …), "" if unknown
model str resolved model id for the turn; "auto" when the completed turn only exposes the Auto request, or the backend reported auto as the resolved id (a session on the default model, whose _model stays ""); "" if no model source is available. A billed row persisted with a blank model logs a WARNING (_warn_unattributed_billing) so the path that lost the model can be found; the provider-wrapper walk that resolves it is bounded at _WRAPPER_CHAIN_MAX_NODES (64)
input / output int prompt / completion tokens (structurally 0 on the ACP backend — kiro-cli bills credits only)
cache_create / cache_read int cache-write / cache-read tokens
cost float provider-reported USD cost (0.0 on ACP)
credits float kiro-cli per-turn credit spend (float-coerced)
turns int model calls the turn made (TurnUsage.num_turns); on kiro-cli counted from its per-call _kiro.dev/metadata frames (see acp-client "Per-call context readings"), 0 when the provider reports none
duration_ms int provider-reported turn duration (0 on ACP)
surface str canonical dispatch/session source. Dashboard-backed turns derive the source from the effective session key through telemetry_channel_of (so linked Slack/Telegram sessions retain their transport); Task Runner writes taskrunner; other writers use their dispatch origin (cron, subagent, monitor, heartbeat, webhook, workflow, …). "" if unset
agent str agent id resolved for the turn; "" if unset
context_used int context-window tokens occupied after the turn (int-coerced)
context_window int served context-window size in tokens (int-coerced)
context_sum int the context-window occupancy after each of the turn's model calls, added up (TurnUsage.context_sum_tokens): the volume of context the turn's calls carried, cached or not. Measured per call on kiro-cli; 0 when the window is unknown or the provider reports no per-call reading. Not a billed token count
stop_reason str the turn's terminal stop reason read off the EVENT_COMPLETE event, or a synthetic reason on a dashboard turn that ended without one (_persist_abnormal_turn_usage: the caller's reason, else cancelled); "" when the producer has none, e.g. a bare TurnUsage from provider_last_turn_usage. Free-form is fine HERE (the row store has no cardinality limit, unlike OTel attrs) — this is where per-agent stall analysis happens: joining stop_reason (error: tool stall / stale_recover) against the row's agent field attributes watchdog outcomes to free-form agent names retroactively

The surface / agent / context_used / context_window / context_sum fields and stop_reason are all additive — every field defaults ("" / 0) so existing callers stay valid and shards predating a field (which lack its key) remain parseable; readers must tolerate their absence.

The row carries no injection breakdown. context_trace, the one reader of ctx_blocks and phase, is served from the crew log's usage projection; the same composition is recorded by crew_log.emit .on_context_composed as context/composed, with the same block labels and the same phase. Shards written earlier still carry both keys and stay parseable — nothing reads them. context_used / context_window are read from the provider at the persist call site via usage.read_context_tokens(source), which calls the provider's public context_used_tokens() / context_window_tokens() accessors (providers/base.py, implemented for ACP in providers/acp.py + acp/session_provider.py) behind getattr guards and returns (0, 0) on any missing accessor or exception — so non-ACP providers and test doubles record zeros and the analytics helper never breaks the turn it measures. surface lets channel-backed turns and background dispatch surfaces (cron/subagent/ monitor/heartbeat/webhook/taskrunner/workflow) retain their canonical source; context_occupancy normalizes the historical task_runner spelling to taskrunner when rows are read. Zero-token surfaces (cron script=/command= modes, heartbeat maintenance ticks) never call a model and must not write a row.

Per-turn injection breakdown (context/composed). Recorded in the crew log rather than on this row (see above). Its sources are produced by context_blocks.split_blocks(prompt, user_chars=…), which attributes the FINAL assembled prompt to the blocks that produced it by matching the bracket markers the assembly emits ([CRITICAL RULES, [Memory, [Skills:], [USER PROFILE], [UI LANGUAGE], [RESPONSE PREFERENCES, [CURRENT USER REQUEST, [REPLY FORMAT RULES], …) rather than counting at each of the ~30 append sites. Reading the OUTPUT means the attribution cannot drift from what was actually sent. _MARKERS is deliberately kept in sync with EVERY opener the assembly can emit — including the identity/session banners ([USER PROFILE], [UI LANGUAGE], [CHANNEL], [INCOGNITO SESSION], [TEMPORARY SESSION], the cancelled-turn preamble), generated request-prefix openers assembled inside build_message before the current request ([THEME PERSONA], triggered [Skill: …] bodies, and [REPLY FORMAT RULES]), and openers prepended AFTER build_message returns (the re-injected [Previous chat history for this tab …], [Hook context] — in both emitted spellings, with and without the colon — and the [System: …] regenerate line): a marker absent from _MARKERS does NOT surface as its own bucket, it folds into the PRECEDING recognised block and mislabels those bytes, so the set must stay complete.

Budget cuts beside the breakdown (context/composed.omitted). Block sizes say how big each block was, not whether a budget dropped anything to reach that size, and the omission notice inside the prompt is read by the model alone. Each budget decision that writes such a notice also calls context_omissions.record with the numbers the notice states: the lesson rule and findings tiers (render_lesson_tier, and render_withheld_tier for findings withheld as off-topic), the pref.* preference facts, the preferences file past the protected-content ceiling, the background admission pool (admit_background), a member briefing at its character cap, and folder steering (a collection ceiling records that the section is incomplete; the section's character cap and the member essentials envelope record the documents not loaded whole). _turn_build_prompt (chat_turn/prompt_assembly.py) opens context_omissions.collect() around build_message, and _turn_dispatch_gates writes the collected record as omitted on that turn's context/composed. The collector is a ContextVar, so two sessions composing against one store never see each other's numbers, and the composer's worker thread inherits it through copy_context. Outside a collection record is a no-op. The record is bounded by construction: record admits one entry per kind in context_omissions.KINDS, each a reason from REASONS and at most four integers or the closed set of background section names, which is exactly what the entry declares; the crew log refuses an undeclared member, and that refusal would also lose the block sizes. Omitted text, keys and rules are never recorded.

Which lessons a turn carried (usage/injections/). The lesson tiers are ranked against the request, trimmed to a budget and filtered by topic, so the set a turn received cannot be rebuilt from the store afterwards. render_lesson_tier (and render_withheld_tier) also calls injection_receipts.record with the memory keys of the rows it KEPT and the omitted count; _turn_build_prompt opens injection_receipts.collect() beside context_omissions.collect(), and _turn_dispatch_gates, right after context/composed, appends one JSON line to <data home>/usage/injections/YYYY-MM-DD.jsonl (local day, owner-only): ts, session_key, turn_index (the crew log ordinal), user_msg_id (the meta.mid of the row that opened the turn, frozen on entry; empty on a synthetic turn), model, agent, surface, phase, memory_store, lessons, lessons_omitted, findings, findings_omitted, and turn_lessons: the keys turn_lessons_block adds to a follow-up under memory.inject_lessons_per_turn. Only an admitted turn whose composition rendered a lesson block writes a line, so a per-turn build that added none writes nothing: the model still holds what the session's earlier lines recorded. Incognito and temporary sessions write nothing. The write runs on a worker thread (no suspension point in the gate) and never fails the turn. Legacy JSONL lesson rows have no key: a kept one is not named, and a cut one still counts in *_omitted. Lesson text is never written. Purpose: scoring which lessons change behaviour (#20446).

Current-request recency boundary. Injected blocks are not the only proof that a turn is contextual. A keyed warm provider session (is_new_session=False) carries native conversation history even when this turn emits no Kiro Crew context block, and a cold session/load resume carries restored native history when resumed=True. Both paths mint [REPLY FORMAT RULES] (when interactive) and the [CURRENT USER REQUEST …] header before the current turn so the user's text owns EOF. Only a standalone call with no session key, no restored/native history, and no injected blocks preserves the legacy raw-user-text-first shape with guidance trailing; it has no older provider topic from which to regress.

A block owns the span from its opener up to the earlier of the next opener and its OWN closer (_CLOSERS, keyed by the same labels — only the closers the assembly actually emits are listed, so extending it is a data change rather than a scanner edit). A label with an opener in _MARKERS but no _CLOSERS entry keeps the absorbing behaviour this table exists to remove, so the two are kept in step by grepping the assembly for [End / [END rather than by adding a closer only when one is noticed. One opener can have MORE than one closer spelling — a hook context is closed [End of hook context] by context_assembly/turn_message.py and [End hook context] by dashboard/chat_turn/turn_dispatch_gates.py — and an entry covering one spelling silently leaves the other block absorbing what follows it, so such a label carries an alternation rather than whichever spelling was found first. The closer taken is the LAST match before the next opener, not the first: a block's content can quote its own closer — a custom agent prompt that documents the envelope it is injected into embeds [END AGENT SYSTEM PROMPT] verbatim — and first-match would end the block at that quotation and book the rest of its real body as unclassified. Last-match is right by construction, because nothing of the block follows its real closer, so any earlier occurrence in range is content. The closer search is bounded by the next opener, which is what keeps a WRAPPER correct: [SESSION CONTEXT closes long after the memory family opens inside it, never finds its own closer in range, and ends where it always did — unbounded, it would swallow every block nested within. The blank line a block emits right after its closer stays with that block, so a closed block does not leave a two-character crumb behind it. Escaping matters in one place worth naming: [End of skill] is a PREFIX of [End of skills], so an unanchored closer would let one loaded skill claim the whole skills index that follows it. Characters that fall between a block's closer and the next opener therefore surface as unclassified, as do leading bytes before the first marker. This is the honest reading and it replaces a silent one: without closer awareness a span ran all the way to the next opener, which turned unattributed bytes into MIS-attributed ones with no way for a reader to tell a genuinely large block from a small one that had absorbed its neighbours. The trigger is ordinary rather than exotic — the blocks between two openers are conditional (workspace identity and the docs pointer are skipped for a custom agent, the memory family for a session sealed from the user's memory), so their absence is exactly what lets an earlier block absorb everything downstream of it. Measured on one real session, a ~470-character [USER PROFILE] block was reported as 8,116. The returned sizes sum EXACTLY to len(prompt) (closure); the user's own text is carved into the your_message label using the exact (start, end) span that build_message reports through its user_span_out out-parameter. That span is authoritative because build_message is the only code that sees every transform applied to the turn: a HOOK_MODIFY transform hook can REWRITE it wholesale, marker neutralization rewrites forgeable boundary markers (changing the length of anything ahead of the user's text), and the return path folds _MULTIBYTE_TABLE punctuation over the whole message (— → --, … → ...). A caller measuring the pre-transform message cannot know the post-transform offsets, so it passes the bounds of the user's text within the text it hands over and build_message maps them forward. When a transform hook has replaced the turn, the caller's bounds describe text that no longer exists, so the hook's output is attributed to your_message in full — it IS the user's turn at that point. Steps that PREPEND to the finished prompt after build_message returns (the incognito/temporary notice, re-injected history, hook context, the regenerate system line) slide that span, so the caller re-derives it once from the length delta and verifies the shifted slice still holds the same text; if it does not, the span is dropped and the reconstruction below is used rather than persisting a wrong attribution. The legacy user_chars / user_offset reconstruction stays as the fallback for callers that do not assemble through build_message. Sizes are characters, not tokens — characters are exact, free and tokenizer-independent, whereas the only tokenizer available here is OpenAI's BPE, which would add a systematic unknown error against a Claude backend, and comparing one turn against another (the whole point of the breakdown) wants an exact unit rather than an approximate one. phase separates the one-off session-start injection from the much smaller per-turn one so a reader never pools the two populations. Non-finite floats are rejected at the single write chokepoint. _write_token_record dumps with allow_nan=False and, only on the resulting ValueError, rewrites non-finite floats to 0.0 — so the common path pays no per-turn scan. The guard lives there rather than per field because provider floats (cost_usd, credits) reach the record unvalidated and a float is exactly what a bad one looks like, so a type check cannot catch it. Without it, json.dumps writes the bare tokens NaN / Infinity, which are not valid JSON while json.loads still accepts them: one such row travels silently into a web.json_response body no browser can parse, taking down every panel that reads the store instead of losing one turn's numbers. cost_breakdown additionally skips non-finite rows on read, because shards written before this guard can already contain them.

Subagent turns are excluded from both readers. usage.is_session_slot(slot) drops any row whose telemetry_channel_of is subagent, at the point the row is READ rather than where it is grouped — so the window totals, the prior-period deltas, by_model, by_channel, by_category, the context bands, the priciest turn and the occupancy percentiles are all computed over one population and cannot contradict each other. A subagent is a fragment of another session's turn rather than a session with its own lifecycle, and its usage row carries no field pointing back at the session that spawned it, so it can be neither listed nor attributed. The consequence is deliberate and worth stating: these totals are the totals of the sessions the panel lists, NOT of the account.

Read side. usage.context_occupancy(days) aggregates these rows into per-turn occupancy percentiles plus a per-session peak ranking (own shard-fingerprint + 30s-TTL cache, same contract as _parse_token_history), and handlers/telemetry.py serves it as the context block of GET /api/telemetry/startup (a plain module-scope import — handlers.usage imports nothing from dashboard.handlers beyond its own usage_* owners, which import nothing back, so there is no cycle to dodge). usage.context_trace(slot, days) is the per-session drill-down, and it reads the slot's usage PROJECTION rather than the shards. The fold's context part is one fixed-size state that does not grow with the session (projection_usage_measured): it never reads the log back, an entry that does not concern it returns the same state object, and its worst case is bounded whatever the turn count: 31 full rows (the ring plus the start row), about 894 KB in memory, under the slot-fold cache's 995,342-byte member ceiling. What it renders, and so pushes, is slim: the newest turn whole and the chart's numbers for every other row, about 6 KB on a real session and at most about 20 KB. A past turn's whole panel is a separate read (below). The trace returns what that state renders:

  • latest -- the newest turn's one breakdown: Crew's five running sums (ours: message, memory, rules, skills, other) since the engine's context last restarted (a new unit or an engine compaction restarts them, a compaction from the compacting turn's own prompt; a compaction/applied the gateway ran in place, which the engine did not measure, keeps only what was composed after the newest turn that closed with an occupancy reading, so a deferred verdict keeps the prompt composed before it arrived), the engine's own categories when it measured the turn (engine, from the newest context/measured, last-wins, with its percent, counts, splits, window and media), its time to first output (ttft_ms), and the occupancy pair context_used / context_window from the newest turn/completed reading. A new unit resets that reading, because the previous session's occupancy says nothing about this one. When the engine did not measure the turn, loop_chars is the occupancy reading at chars_per_token less Crew's sums, with loop_estimated set.
  • trend -- the newest 30 turns (TREND_TURNS, the panel's MAX_CHART_TURNS), oldest first, each as the chart's numbers only: its own turn ordinal, Crew's five sums, the agent loop's part (loop, measured turns only: the chart shows no estimate), the turn's own occupancy reading in tokens (used, 0 with none) and the row's version. The state keeps, per row, what that turn's own panel needs: its Kiro Crew blocks by name (at most BLOCKS_PER_TURN, 64, the composer's own most; a name under TEXT_LIMIT; the rest counted in blocks_omitted), a copy of its engine measurement (all 14 categories, the history split, list counts, the toolIO, workspace and memory splits, window, media and time to first output), and its own occupancy reading. GET /api/telemetry/context-trace?slot=<key>&turn=<ordinal> (usage.context_turn_detail, projection_usage_measured.context_turn) serves one kept row as a whole-turn view in latest's shape (turn_detail), read from the in-memory fold state through read_slot_state, never from the log: 400 turn_invalid unless turn is a positive ASCII integer, 404 turn_not_kept once the turn has left the ring, and the same dashboard-only gate and app-caller 404 as the trace. The panel reads it when a past turn is picked and caches it by slot, ordinal and that row's version in the trace (rowVersion), which the fold stamps from a write counter (row_writes) every time it writes the row, so a turn sent again is re-read even when its numbers are unchanged and a new turn elsewhere re-reads nothing; while a changed row is re-read, the previous read of the same ordinal stays on screen, disclosures open. A pick belongs to its session: the tab is remounted per slot, so switching sessions opens the new one on its newest turn. Each ring row stores the ordinal it was closed at, so a slot with several session starts labels every row correctly. A turn sent again (a retry, a regenerate, an edit-resend of an earlier message) keeps its ordinal, and every row from that ordinal on (ring and start row) is dropped because the conversation was cut back to it, so no two rows carry one number and turn_count is the real count. The ordinal is read from the ordinal memo (turn_memo: the current unit's turn numbers in ordinal order, bounded at CONTEXT_ORDINAL_MEMO_LIMIT, 10,000, the slot's live message cap, so it reaches any turn a rewind can), never computed from turn numbers, which count durable transcript rows; a rewind past the ring keeps its ordinal and cuts every ring row after it, and a refused turn (turn/refused) takes an ordinal too, since it holds its number in the conversation; its previous attempt's measurement, time to first output and engine turn id are dropped until the new attempt reports its own.
  • start -- the newest session-start turn's row, kept out of the ring because the panel lists it above the chart, with figure, the size its own panel heads with (the engine's characters when measured, else its occupancy reading in tokens).
  • revision -- the fold revision the page was read at.
  • earlier -- exactly how many turns precede the ring's first row, and turn_count, the one turn counter.

turns, totals, injected_chars and user_chars are built from that state for the Session Breakdown tree, which reads them per sub-agent; each of turns is what that kept turn sent, by block name, read from the state (context_turn_blocks), not from the pushed value: {phase, blocks, total_chars, ordinal, first_start}. The state keeps no per-turn time, model or reading for those rows, so they carry none, and the page has no engine_unknown map. The occupancy pair peak_context_used / context_window is the provider's own reading and the window it was measured against, both from the single turn/completed closed in the last days that is the FULLEST: the highest used / window ratio, which the tree turns into a fill ratio (context_peak). Fullest is the ratio and not the largest used, because a model switch moves the window: 90k of a 100k window is fuller than a later 200k of a 1M one. The comparison is integer-exact by cross-multiplication, >= keeps the newest of equally full turns, and a reading with no stated window is not a candidate. The state keeps the newest PEAK_READINGS (200) dated readings for it (peaks), across units, the reach the per-turn rows had; with none inside the span the pair is 0 and the newest non-zero window request/configured stated in this unit (conf_window). days selects nothing else and is echoed as window_days. Block sizes are characters and the occupancy pair is tokens. A session recorded with the crew log switched off reads empty -- this surface depends on that switch, which is the one thing it is not independent of. Billing is not on this payload: slot_turn_usage (below) is the per-turn reader for credits / duration_ms.

The engine measures a turn only when asked: every KAS session/prompt carries contextBreakdown: "summary" and ttft: true in _meta.kiro (kas_profile.PROMPT_META). Turns KAS starts on its own (its workflows) are not measured: Kiro Crew does not enable KAS workflows today, and the session-level declaration (automaticTurnProfiling) will be added when it does. Cron, sub-agent and task turns open no crew-log turn, so a measurement written for one attaches nowhere and never overwrites a parent's breakdown.

How the panel draws it. The chat Activity panel's Context Breakdown tab (website/src/pages/ContextBreakdownPanel.tsx, ContextEngineBreakdown.tsx) shows the picked turn first (the newest by default), as one breakdown in the style of a window: a headline, a bar, and a flat list, zero rows hidden.

  • Measured turn (latest.engine present): the headline is the request's characters and, when the engine reported its window, about how many tokens that is by the window reading. The bar is the request at full width, each part at the engine's own percent. Row 1 is always Kiro Crew (rules, memory, skills, session details: every Crew block outside the other three, such as the date, workspace, user, folder, replayed history and reply format), a group that can be folded and opens by default; then Your messages; then, largest first, Messages (the engine's history less Crew's blocks: replies, reasoning, summary and "Other text sent as you") and every other category the engine counted, each with its own split where it sent one. Engine-side rows are named apart from Crew's own: "Engine system prompt" (not Crew's rules) and "Skill list (engine)" (not Crew's skills). A line under the bar gives the window share.
  • Turn not split (no engine): every row is in tokens at chars_per_token, marked ~. Kiro Crew and Your messages are Crew's own sums; "Rest of the context" is the occupancy reading less that estimate, never below 0 (a warning says so when the estimate passes the total). The bar is drawn against the whole window. An info tip on "Rest of the context" names the session's harness (the backend of GET /api/chat/slots/{slot}/selection-capabilities, the route the model picker reads) and the place to get a split: Settings, the Agent Harness tab, the Kiro Agent choice, and says that new sessions then use Kiro Agent while this session keeps its current agent. On Kiro Agent the tip says the row is estimated because the engine has not measured the turn yet. A short tag beside the tip names which tip it is before any hover: "How to split it" on another harness, "Not measured yet" on the newest Kiro Agent turn, "Not measured" on a past one. When that route fails, its error is shown beside the trace's and the tip and its tag are left off, since the harness is not known. With no occupancy reading, the rows stay in characters and nothing else is claimed.

Below it, the trend: the start row ("Turn N · session start", its characters), listed above the chart because its one-off size would pin a linear axis, then a stacked-area chart of the trend rows (x = turn ordinals as the fold stored them, y = characters; estimated tokens at chars_per_token instead when the panel is in tokens, that is the newest turn is not split, has an occupancy reading, and no row has a measured loop, so the chart and the rows above it share one unit), with a "Turns 1-N earlier" line from earlier that says when the start row is one of them. Every column is a real button (arrow keys move the pick, aria-pressed marks it; below 12px a column, one plot-wide surface maps a click to the nearest turn), and the start row is a button too: picking one shows that turn's detail exactly as the newest turn is shown, the engine's own rows when the engine measured it and the estimated view otherwise (on Kiro Agent, "Estimated. The engine didn't measure this turn."), plus a disclosure listing the Kiro Crew blocks that turn sent, by name. The newest column shows latest itself. While a past turn is picked, a "Show latest turn (N)" button goes back to latest, and so do Escape and Home on a column: the newest turn may still be running and so not be a column yet, and without it nothing could undo a pick. Its bands are Crew's five categories and "Rest of the context" (the measured loop; on a token chart, the row's own used less Crew's estimated tokens, so each column reaches that turn's harness total; a turn with no reading gets no such band and its column is drawn hatched, not as a drop, which the caption then says), and the legend lists only the bands that are non-zero. When the newest turn is not a column yet, a line under the chart says it is still running. When the headline is in tokens but the chart counts characters (an unmeasured Kiro Agent turn), a note under the headline says the unit changes. The blocks disclosure's heading counts every block the turn sent and carries a chevron (right when closed, down when open); the positive entries before the every-turn members are grouped into one list row, plus blocks_omitted, whose line is styled as plain text. X-axis labels are strided from the measured plot width (axisLabelIndices). Fills come from the --ctx-cat-* and --ctx-eng-* theme tokens (website/src/styles/design-tokens.css). Grey (--ctx-eng-loop) is the rest of the context in both the list and the chart, and the engine's own Other has a pink of its own (--ctx-eng-other) wherever it is drawn, so the two never read as one thing. An empty trace reads "not recorded while the crew log is off" when the payload's recording is false, naming the .env the gateway reads from the payload's env_file (sent only then), and the "nothing recorded yet" copy otherwise. handlers/telemetry.py::api_context_trace serves it as GET /api/telemetry/context-trace?slot=<session key> (400 when slot is missing or blank). It is independent of the telemetry.enabled switch but dependent on the crew log it reads (empty while KIROCREW_CREW_LOG is off). The endpoint is dashboard-only because, unlike /api/usage/turns, this reader has no row-ownership model: an app caller is refused outright with the standard indistinguishable 404 (code: not_found) and the refusal is SEL-audited (deny-by-default, App Kit §5.2). An app that needs its own turns' billing has /api/usage/turns.

Row-timestamp parsing has one owner. usage._parse_row_dt (in usage_store.py) is the single spelling for reading a stored row timestamp (Z rewritten to +00:00 for py3.10's fromisoformat; a naive stamp left naive so a caller's .timestamp() reads it in local time); _parse_row_ts derives the epoch form, and every shard/row reader in the usage handlers (slot_spend, context_occupancy, cost_breakdown, slot_turn_usage, the token-history and transcript-day readers) resolves timestamps through them. Two readers of the same rows must not disagree about which rows a window contains. _usage_number is likewise the one guard for copying a numeric field out of a row (bool is not a count; ints are accepted directly because math.isfinite would overflow on an oversized int; a non-finite float is dropped).

Per-turn usage rows for one session. usage.slot_turn_usage(slot, days) is the per-turn drill-down under slot_spend's aggregate: one row per turn with ts, model, and the numeric fields named by usage.TURN_USAGE_FIELDS (tokens in/out, cache create/read, credits, cost, duration_ms, and the context meter pair). A non-numeric or non-finite field is dropped from its row, never the row itself. handlers/telemetry.py::api_usage_turns serves it as GET /api/usage/turns?slot=<session key>[&days=N | &since=&until=] (400 on a missing slot; the window is read by the same resolve_window as the panel, so the Spend table's drill-down lists the turns of the range its row summed, defaults to SPEND_WINDOW_DAYS, and clamps to MAX_WINDOW_DAYS rather than refusing; the response is {slot, days, turns}, with days the clamped length and no start/end echo, because the drill-down already sends the bounds it measured). This is the endpoint an app is granted through its manifest's permissions.api to account for what its own agent slots cost — apps otherwise have no path to credits, and the shard files' location and row shape stay this module's private contract. App isolation is ROW-level (App Kit §5.2, deny-by-default): each row is stamped with the owning app at write time (_build_token_record's app field, threaded from the turn's slot), and an app caller receives only rows stamped with its own app — however the slot is named, and whether or not it is still live. A live-slot ownership check was deliberately rejected: it leaks on slot-name reuse (a recreated slot vouches for the previous owner's retained rows) and denies an app its own completed sessions, which are exactly what an audit reads. A foreign slot key answers 200 with no rows — indistinguishable from a slot that never ran — and rows predating the stamp are invisible to app callers. A disabled app is refused outright (is_app_enabled, deny-by-default, the same gate the opt-in builtin routes wrap every handler in): disable revokes read access, not only future writes. Every app-caller decision is SEL-logged — including a malformed request's refusal — and SEL plus the enablement probe run off-loop. The window is enforced per row, not only per shard file (the oldest shard in a window covers a whole day); a row whose timestamp cannot be parsed is excluded — accounting excludes what it cannot date. Dashboard users (empty request app) read any slot; the Telemetry panel's Spend table is the dashboard consumer — each session row expands into its per-turn rows through this endpoint, which is where a mid-session model switch or a single runaway turn becomes visible (an average hides both). The Spend tab's first screen is a summary: KPI figures for the period (credits, the change against the previous period, turns, per-turn cost, the priciest turn), then bars for credits by origin, credits by model and the top sessions. The per-session table sits under a Full table disclosure that starts closed and remembers its state in telemetry:spend-table-open. Stamping boundary: rows are stamped at the two write sites that can run app-owned work — the dashboard chat runner (the slot's _app) and the subagent completion path (info.app, an app-dispatched subagent's spend). The task-runner, workflow, Slack and background-one-liner writers do not stamp because those surfaces are not app-owned — an empty stamp there is the correct value, not a gap. Webhook-session rows are currently unstamped and therefore invisible to app callers; if webhook sessions gain app ownership, that write site must stamp too. Same independence from the telemetry.enabled switch as the context trace.

Where it renders. The breakdown is a per-session side-panel tab (ViewKind context, opened from the panel's + menu directly under Logs), not a section of the global Telemetry page. It is a developer surface, listed in the + menu's diagnostics group with Logs. The tab is scoped to the chat slot it was opened from, which is why it carries no session picker: a per-turn drill-down that first asks the reader to choose a session cannot say anything until they do, and Logs — the other "what actually happened in THIS session" view — already sets the precedent. It never polls: the tab reads once on mount (refetchOnMount: 'always'), then re-reads once per pushed usage session_projection frame (coalesced per slot, website/src/hooks/websocket/contextTraceRefresh.ts) and once on a socket reconnect, because the trace gains a row per turn and a tab left open would otherwise go stale. The Telemetry page keeps the aggregate context occupancy block only.

Both readers stay OUT of the OTEL metric pipeline for the same reason: occupancy and the per-turn breakdown are per-session, per-turn detail, and slot keys are unbounded-cardinality labels that must never become metric labels (the context_occupancy docstring states the same rule). The bounded half — block label → size, aggregated — is what could belong on a metric; this per-session half is the drill-down. Without these readers the fields were write-only: recorded on every turn, read by nothing.

usage.cost_breakdown(days) is the second reader, same cache contract, serving the cost block of the same endpoint. It answers "where did the credits go" from fields the row store already carries — no new instrumentation:

sub-block derivation
totals credits summed over the window, plus the preceding window of equal length and the delta between them
by_model per-model credits, share, credits-per-turn, and per-model delta vs the prior window. Every model, never truncated — a top-N cut hides exactly the cheap-model-creep this block exists to show
by_channel same shape, keyed by telemetry_channel_of(slot)
context_bands mean credits per turn bucketed by absolute context_used, which is what makes the cost/context relationship legible (a turn at 900k costs ~4.7x one at 100k)
conversations EVERY session in the window (not a top-N), each with its category, the unollapsed channel beneath it, peak occupancy, span, per-turn growth rate, and a projected turns-to-compaction. Named conversations for payload compatibility; the entity is a session
by_category same shape as by_model, keyed by usage.session_category(slot) — the taxonomy the panel groups by: bg for an unattended session (cron, heartbeat, task runner), otherwise the transport (dashboard, telegram, slack, …)

Channel comes from the slot key, not from surface. New writes now derive surface from the effective session identity, but historical rows may contain a wrong, non-empty value (for example a Telegram turn stamped dashboard). The row format has no schema version or writer/trust marker that distinguishes those legacy values from corrected writes. Cost/category attribution therefore keeps the slot key authoritative; preferring a non-empty surface would silently misattribute existing history. The stored slot/session key remains stable across the write-side correction and is still the compatibility boundary for this reader.

The dedicated _consolidate session retains the consolidation telemetry channel and is grouped under bg in Spend, alongside _bg background work.

Two origin columns, deliberately, and they are labelled apart. The page shows the session-origin dimension twice, derived two ways, because the two answer different questions. Spend's Origin column (session_category, from the slot key) answers WHICH SURFACE OWNS THE SESSION; the Context tab's Turn surface column (the row's own surface field) answers WHICH CODE PATH RAN ONE TURN. So the same unattended session reading all background on Spend and heartbeat on Context is correct, not a disagreement, and the two column headers each carry a tip naming their own derivation. The ruling that fixes this shape: a trusted surface may NOT introduce Spend attribution values that do not exist today, and monitor spend stays booked to the conversation it nudged. A monitor nudge exists only to serve one conversation, so moving its credits into a separate monitor bucket would make that conversation under-report what it actually cost — the one number the Spend tab exists to give. The consequence is that the two vocabularies are permanently different sets: monitor (slack/gateway_runtime/nudge_channel_fire.py) and webhook (handlers/hooks.py) are written as row surfaces and are NOT members of TELEMETRY_CHANNELS, and nothing in the read path compares one against the other. Unifying them is therefore a product decision that has been made and declined, not a cleanup: test/metrics/test_telemetry_column_taxonomy.py pins the two sets apart with those two values as the known non-members, so a later unification has to confront this ruling rather than discover it.

A conversation's title is attached by the endpoint, from two sources in order. cost_breakdown names nothing — slot keys are all the row store holds. handlers/telemetry._with_conversation_titles resolves the live slot's display_title first, so a rename shows before it has been flushed; a session with no live slot falls back to the title on its transcript's metadata line, read through ConversationLog.get_metadata off the event loop and keyed by slot_transcript_key(slot), which is what folds a channel-born slot onto the channel's own transcript. Only an explicit metadata title counts: list_sessions would answer with the first user message and then with the session key, which turns a ranking label into prompt text and leaves no way to tell a named conversation from an unnamed one. A row with neither source reports an absent title rather than its key. Both sources pass the same two scanners on the way out, so where a title came from cannot change what leaves the endpoint.

The growth slope is fitted per segment, never across the window. Occupancy is a sawtooth: a compaction drops it back toward empty, and 7 of the 8 top-spending conversations measured carry one to four such drops. A secant from the first turn to the last therefore crosses discontinuities and is dragged down by every reset, understating the live rate by ~47x on real data — one 104-turn conversation projected 795 turns of headroom while its current segment climbed at 4%/turn, i.e. ~17 turns from compaction. Erring toward "plenty of room" is the damaging direction for a figure whose only purpose is a warning, so the slope is fitted on the stretch since the last fall larger than _COMPACTION_DROP_PCT (sub-threshold drift is ordinary jitter in what counts toward context_used, not a compaction). Growth and the projection are withheld unless that CURRENT segment holds _COST_MIN_GROWTH_TURNS points — a long conversation freshly past a compaction knows nothing about its new trajectory, and withholding is the honest answer rather than extrapolating from two or three points.

Tests: test/test_usage.py (TestReadContextTokens, TestBuildTokenRecordContextFields, TestPersistTokenRecord*), test/metrics/test_context_occupancy.py (aggregation, skips, latest-turn wins, historical Task Runner spelling), test/metrics/test_cost_breakdown.py (channel attribution incl. the bare dashboard slot form, no-truncation, prior-window deltas, band bucketing, post-compaction slope, growth withholding, non-finite rejection), test/test_dashboard_chat_token_usage.py (effective session source at the dashboard write site), test/test_turn_duration_task_runner.py (canonical Task Runner writes), and test/metrics/test_telemetry_titles.py (title redaction + cache purity, and the closed-conversation fallback: the canonical transcript key, live-wins-over-persisted, one read per shared transcript, and no title where the metadata line names none), plus the context-trace endpoint.

Circular-import rule

metrics/provider.py imports config.loader at module top; call sites reached from inside config.loader's import chain (e.g. acp/client_start.py) MUST import get_recorder lazily (inside the function) so the provider is never loaded during that chain.

Anonymous outbound telemetry (beacon.py, apps/install_receipt.py)

There are now five independent telemetry paths. Keep them straight:

Path Purpose Data Egress Switch
OTEL metrics (metrics/) Ops observability DELTA histograms / counters Never (local JSONL; OTLP only if the operator sets an endpoint) telemetry.enabled (off)
Token row store (usage/tokens/) Cost + context analytics One row per model-spending turn Never always on
Injection receipts (usage/injections/) Lesson precision scoring One line per admitted turn that rendered a lesson tier: memory keys and counts, no text Never always on; skipped for incognito and temporary sessions
Beacon (beacon.py) Product analytics One anonymous ping per install per day Yes — to the Kiro Crew endpoint telemetry.beacon_enabled (on)
Install receipt (apps/install_receipt.py) Official app adoption One anonymous receipt after a successful official-catalog install/update Yes — to the same endpoint telemetry.beacon_enabled (on)

src/kiro_crew/beacon.py, tests test/test_beacon.py. Fired from slack/gateway_runtime/process_entry.py::run_gateway on a detached daemon thread (never awaited — a 5s blocking urllib call must not delay boot or pin interpreter exit), and skipped entirely under --test-mode so the offline E2E gate cannot egress.

Why it is NOT part of the OTEL trunk

Four independently disqualifying reasons — do not "consolidate" them later:

  1. The privacy guardrail eats the payload. MetricsRecorder._guard() runs every attribute through schema.redact(). A 64-char sha256 install id becomes "[REDACTED]" (40+-hex rule), so DAU would silently compute as 1. Ids that merely survive redaction are no better: schema.py mandates low-cardinality enum-like values and the instrument cache never evicts, so a per-machine id is precisely the "cardinality bomb" that contract prevents.
  2. OTLP egress is an extra. opentelemetry-exporter-otlp-proto-http ships in kirocrew[otlp], not the default dependency set, so a beacon riding it would measure only users who installed an optional extra.
  3. telemetry.enabled is a published no-egress promise (config help, this spec, the dashboard panel all say "nothing leaves this machine"). Hanging an outbound heartbeat off it would retroactively change what that switch means.
  4. Shape mismatch. OTEL carries pre-aggregated data points; DAU needs one row per install per day deduped at query time. Aggregating away the id makes DAU uncomputable.

The one thing it does share is the atomic create-once file pattern from handlers_system.py::_get_telemetry_salt (os.link, owner-only mode, in-memory fallback).

What DailyActiveInstances means (and does not)

The metric is DailyActiveInstances — deliberately not "DAU" and not "Installations". The distinction matters because it is easy to misread the number:

  • It measures activity, not installs. The install_id is a persistent identity generated once at first run, not a per-install event counter. A copy that runs on ten days contributes to ten daily buckets with the same id; COUNT(DISTINCT install_id) per day therefore answers "how many copies ran today". Install volume is a separate metric (NewInstallations, from the first_seen bit, plus model-CDN first-fetch).
  • It is not DAU, because the denominator is not a person and cannot be. There is no account system — only a random per-data-home id — so one operator on three machines counts as 3, and three people sharing one machine count as 1. Calling it DAU would invite the reader to treat "14" as 14 people.
  • The over-count is larger here than for a typical CLI: Kiro Crew supports pods, worktree previews, and KIROCREW_HOME overrides, so one person on one machine easily has several data homes. Dev homes and CI are suppressed, but a user's own pods are not.

Instance is the honest middle: it names a running thing (not the act of installing) without claiming to have resolved it to a human.

The five fields (a fixed allowlist)

payload() is the only producer; there is no caller-supplied field, so the shape cannot be widened from a call site.

Field Example Why
id 32-char hex Random UUID4. Dedup key for COUNT(DISTINCT).
v 0.1.2 Version adoption. Release only — every build stamp is stripped by release().
py 3.12 Minor only. Answers "when can the floor move off 3.10".
dist dmg Which install path users take. Clamped to a fixed set. Baked at build time (see below).
first_seen 1/0 One bit → new-install and "launched once, never again" rate.

Four fields were REMOVED — do not re-add them

chan (release channel), os, arch and gov (governance posture) were part of this payload and are gone. Each was individually low-cardinality and individually defensible; the problem was collective. The install id is stable, so every attribute on the request is correlated with every other, and channel + OS + architecture + governance posture together partition the population far more finely than any one of them suggests — a nightly, ARM, governed-and-verified install is a small crowd to hide in even though no single field is identifying.

So the rule for this payload is not "is this value low-cardinality?" but "how much smaller does this make the crowd this install hides in?" For all four the answer was "too much for what it bought":

  • chan — a nightly install is a small population by definition, which is exactly what makes it identifying when paired with a stable id. This one is a real capability loss, not a relocation: release() strips the prerelease label, so a nightly 0.1.2-nightly.<stamp> and a stable 0.1.2 both send v=0.1.2 and are indistinguishable on the wire. Channel-split adoption is no longer answerable from the beacon. It remains observable from the release feeds / CDN fetch counts (feed/<channel>/latest-cli.json), which are per-artifact and carry no install id — a better source for that question anyway.
  • os / arch — the platform mix is answerable from download/CDN telemetry, which is per-artifact and carries no install id at all: a strictly better source for the same question.
  • gov — governance adoption is an enterprise-fleet question, and a fleet operator knows their own posture. Correlating it with a stable id bought a number nobody was blocked on.

The client is authoritative: beacon._fields() returns exactly four keys (plus id) and test/test_beacon.py asserts the key set, so a re-addition fails a test rather than shipping quietly. website/src/test/PrivacyPanel.test.tsx asserts the four names are absent from the user-facing disclosure for the same reason.

Historical rows keep the old params. Log lines already in S3 carry chan/os/arch/gov forever, and un-upgraded clients keep sending them until they update. That is harmless — the aggregator simply no longer reads them, so the fields age out of the analysis without a migration.

Why dist is baked into the artifact

dist answers "which install path do users actually take", so it has to describe the artifact, not the environment the artifact happens to run in.

Resolution order is baked module → env var → "source":

  1. kiro_crew/_build_info.py, generated by scripts/stamp-distribution.sh and written by each packaging path. Authoritative because it ships inside the artifact and a running install cannot change it. Imported once at module import into beacon._BAKED_DISTRIBUTION (an optional-dependency try/except ImportError, since the module exists only in a packaged artifact); that binding is also the seam tests patch, because writing a real file into the installed package is process-wide shared state that races under the default -n auto.
  2. KIROCREW_DISTRIBUTION, kept as a build/test override.
  3. DEFAULT_DISTRIBUTION ("source"), the correct answer for a git checkout, where the module is absent (and gitignored, so it is never committed).

The env var is deliberately the weaker source. It is inherited by every child process and settable by anyone with a shell, so a stray export in a profile would relabel that host's daily count. An unknown baked value falls through to the env var rather than going on the wire, so a bad stamp can never emit an unclamped value.

Path Where it stamps Value
Wheel .github/workflows/build-wheel.yml, before python -m build wheel
macOS desktop packaging/build-desktop.sh → build_backend dmg
Linux desktop packaging/build-desktop.sh → build_backend (re-stamped per LINUX_TARGET_DISTS target) appimage, deb, rpm
Windows desktop packaging/build-desktop.sh → build_backend nsis
Container docker/Dockerfile, after the wheel install docker
Git checkout nothing source

Three non-obvious constraints:

  • The desktop stamp writes into the installed tree inside each backend bundle, not the repo. A universal macOS build produces two backends and both must carry it, and stamping $ROOT would leave the module in a developer's checkout.
  • The container re-stamps after installing the wheel. The wheel it installs is already stamped wheel, so without the overwrite every container reports the wheel channel. It overwrites the module rather than setting the env var, because the baked value outranks the env var by design.
  • The value is derived from the electron-builder target, not the host OS (mac.target: dmg, the Linux AppImage/deb/rpm targets, win.target: nsis). A Linux host also builds wheels, so branching on the host would mislabel them. The stamp is read by more than the beacon: platform/update_capability.py keys the externally-managed classes on it, so a desktop artifact that carries the wrong value is offered the CLI release feed and the wheel installer command.

The three tests that execute stamp-distribution.sh skip when bash cannot run it, probed by actually invoking the script rather than by shutil.which("bash"): Windows resolves that name to a WSL launcher stub which then fails and cannot see a Windows path. test_stamp_script_exists is asserted unconditionally so a deleted script still fails on a host without bash.

test/test_beacon.py::TestDistributionStamp pins the precedence, the clamp, the gitignore entry, and that the script accepts every KNOWN_DISTRIBUTIONS value, so adding a channel to the frozenset without teaching the script fails a test instead of failing at release time.

Why v is clamped

__version__ is not low-cardinality in the field. Dev and nightly builds carry a per-build timestamp (0.1.2-nightly.20260731t065756, 0.1.2.dev20260731065756), so sending it raw both fingerprints and misleads:

  • It fragmented the one number the field exists to produce. A real 57-install 0.1.2 population reported as 35 plus a fringe of one-install series — which reads as adoption decay when nothing changed.
  • It was silently lossy. The aggregator caps each breakdown at BREAKDOWN_LIMIT (25) per day. Past that, real low-install releases fall below the cut and vanish from both CloudWatch and the permanent rollup — and a long-tail release is exactly the one you most need to know is still running. The limit now queries LIMIT+1 and reports truncation (log line plus a truncated key in the invocation result) instead of dropping quietly.
  • It was a fingerprint. A near-unique per-build value combined with a stable install id picks out individual machines, which is precisely the correlated- attribute hazard the rest of the allowlist is designed to avoid.

The channel is discarded, not moved to a field of its own (see "Four fields were REMOVED" above). beacon.channel() and its markers are deleted along with it, so there is no dormant helper for a future call site to re-wire.

Normalized on both sides. release() clamps at the client, but a client only stops sending raw stamps once it upgrades, and rows already in S3 keep them forever. So the aggregator derives v from the raw v param in SQL (_VERSION_EXPR) as well. That makes the metric correct for pre-clamp clients and means re-aggregating a historical day yields the clean value. An unparseable stamp becomes the single bounded bucket unknown, never a new metric.

Anti-fingerprinting is a design constraint, not a side effect. Every value is a low-cardinality constant or coarse bucket, because the id is stable and attributes on the same request are therefore correlated: enough precise attributes (exact patch level, CPU count, RAM, timezone, uptime) would combine into a quasi-unique identifier even though each is individually "anonymous". Do not add precise numeric or high-cardinality fields.

Never sent: prompts, model output, file contents, paths, repo names, credentials, hostname, username, IP. Notably it does not reuse handlers_system._get_owner_hash() — that is HMAC(salt, hostname + ":" + username); it stays local-only, and sending it would change its character.

Fail-open is the load-bearing contract

No beacon failure may ever block, delay, or surface an error in a user action. A telemetry path that can fail a turn is worse than no telemetry, so this is enforced structurally, not by care:

  • Boot is never delayed. run_gateway starts beacon.send on a detached daemon thread and never joins or awaits it. Measured cost to the boot path with a beacon hanging 30s: 0.08 ms (the thread spawn). daemon=True also means it cannot pin interpreter exit. TestFailOpen pins both, plus a source assertion against a refactor to await asyncio.to_thread(...), which would silently reintroduce up to HTTP_TIMEOUT_SECS of boot delay.
  • Every failure returns False silently. Verified across ten real modes: DNS failure, connection refused, TLS handshake failure, HTTP 500, HTTP 403 (corporate block), socket timeout, captive-portal garbage bytes, malformed URL, network-unreachable, and disk-full on the stamp write.
  • Both filesystem probes are inside the guard. should_send() and payload() touch the data home, so they run inside send()'s try — an unwritable home raises PermissionError from config_dir(), and a container whose UID has no passwd entry makes Path.home() raise RuntimeError (not OSError). Both are caught; the documented in-memory-id fallback is what engages.
  • http.client.HTTPException is named explicitly in the except tuple: it subclasses Exception directly, so it is neither OSError nor ValueError. Without it, http.client.InvalidURL / BadStatusLine escaped into the daemon thread and threading.excepthook printed a traceback to gateway stderr on every boot.
  • Even an unexpected exception class is contained — it dies inside the daemon thread and the main thread is unaffected.
  • status() never tracebacks. A diagnostic has to be readable precisely when something is broken, so its probes are guarded too.

The one thing deliberately not swallowed is a stamp-write failure's consequence: _mark_sent() runs only after a delivered request, so a failed send retries later rather than silently losing the day.

Default-ON with six suppressions

telemetry.beacon_enabled defaults true and gates the repo's only default-on egress family: the heartbeat and official-app install receipts. telemetry_permitted() suppresses both when KIROCREW_TELEMETRY_DISABLED is truthy, an enterprise governance ceiling pins capabilities.telemetry off, the config toggle is false, the process looks like CI, KIROCREW_HOME is non-default (dev home / pod / worktree preview), or this install has never sent and dashboard.privacy_acked is still false (the first-egress gate below). beacon.should_send() adds the heartbeat-only daily throttle; receipts are event-based and do not use it.

Each suppression carries a stable Verdict.code from beacon.REASONS alongside the English reason, so the dashboard can translate the outcome instead of printing an operator diagnostic.

Enterprise opt-out: the capabilities.telemetry governance scope

The Settings toggle, the CLI and the env var are all operator controls: a user on the machine can flip any of them, and so can the agent for the first two. A managed fleet often may not egress to a vendor endpoint at all, which needs a control the running app cannot undo. That is the capabilities.telemetry SCOPE_CATALOG capability row (capability_default=True, so policy-absence keeps the documented default-on behavior for standalone users).

It is read from the trust-root security_policy.json, which lives in security._SENSITIVE_HOME_DIRS (defined in security/paths_lists.py). The agent cannot rewrite its own ceiling: the sandbox mounts the file read-only, and it stays readable on purpose because boot_platform() resolves the ceiling inside the sandbox; only an enforced harness's child has it withheld (see security). That is what makes this enterprise-pinnable where a config.json field would only be a suggestion. Authoring is a normal capability block:

{"version": 1, "boot": {"fail_closed": true},
 "capabilities": {"telemetry": {"enabled": false}}}

Enforced at four chokepoints — one send gate plus every write path to telemetry.beacon_enabled, because any one alone is a half-control:

Chokepoint Behavior when pinned
beacon.should_send() Refuses the send, reason disabled by governance policy (capabilities.telemetry) — ranked above the config flag so a managed host reports the policy, not the local value
PATCH /api/config/kirocrew 403 on telemetry.beacon_enabled=true; writing false is always allowed (tightest-wins — a narrower local choice composes with the ceiling)
kirocrew telemetry enable Exits 1 without writing config.json; disable still works
kirocrew config set [--local] telemetry.beacon_enabled true Exits 1 without writing. Easy to miss: the generic setter reaches the same key, and --local writes config.local.json, which takes precedence over the base file — so leaving it ungated would make it the one remaining way to store true on a pinned host

Adding a fifth write path means adding a fifth gate. The rule is that no path may leave a pinned host storing true; test_beacon.py::TestGenericConfigSetterIsGated and test_config_patch.py pin the existing ones.

Without the write refusals a pinned host could sit storing beacon_enabled: true behind a toggle that does nothing — the same false-promise-on-a-privacy-control failure the overlay check already guards against.

Level-1 POLICY only. The probe requires layer == "policy", so a Level-2 profile pinning capabilities.telemetry does not suppress the beacon: the probe runs from the boot thread with no session, and a bare not-permitted test would read a transient deny-all-profile race as an administrator pin on an ungoverned host. Full rationale in governance.md → "Anonymous telemetry".

beacon.is_governance_pinned_off() is the public probe, surfaced as governance_override on GET /api/telemetry/beacon and as the strongest of the three "pinned" notes in the Privacy panel (it outranks the env-var and overlay notes, which would otherwise offer remedies the ceiling makes pointless).

It fails CLOSED (fail_closed=True), like capabilities.theme_install and capabilities.publish. The two dispositions look symmetric and are not: a wrong DENY loses one heartbeat, but a wrong PERMIT egresses from a fleet that explicitly forbade egress — breaking the exact promise the administrator was given, on a payload that leaves the machine. fail_closed also makes governance_permits audit the degrade as a critical SEL event, which is precisely the condition under which a silent degrade-to-permit would be indefensible.

The probe distinguishes two failure sources, because conflating them produces a different bug in each direction:

Source Identified by Disposition
Evaluator could not answer (degrade, or the call/import itself died) reason starts with GOVERNANCE_ERROR_REASON pinned — fail closed
Evaluator answered, but from Level 2 layer == "profile" not a pin (see the policy-only note above)

The second row is load-bearing: a transient deny-all-profile race on a host with no policy at all arrives as an ordinary Decision, so no except can catch it, and reading it as a pin would blame an administrator who does not exist. TestGovernancePin pins all three outcomes.

The enforcement decision is SEL-audited; the probe is not. should_send passes audit_tool="beacon_send", routing through vet_and_audit so a suppressed heartbeat lands a governance_decision record. status() passes audit=False because it backs GET /api/telemetry/beacon, which the Privacy panel refetches — auditing an inspection would flood the trail.

is_default_home() compares against ~/.kiro/crew directly, never against config_dir() — config_dir() honors KIROCREW_HOME, so comparing the two always matches and the suppression would never fire (a real bug caught by TestDefaultHomeDetection).

kirocrew telemetry status | disable | enable — status prints the exact payload and never materializes an id (install_id(create=False)). Its numbered, choose-one opt-out list leads with kirocrew telemetry disable because that choice persists to config.json and survives a new shell. Each method is a separate visual block: the environment override groups separately labelled macOS/Linux, PowerShell, and Command Prompt syntax, followed by the equivalent config key.

beacon.is_env_opted_out() is the public probe for "the env var pins this off". It exists so the dashboard can distinguish off because the stored flag is false (a toggle can flip it) from off because the environment says so (a config write would be accepted and then have no effect).

In-product opt-out (Settings → Privacy toggle)

The GUI twin of kirocrew telemetry disable. It writes the same key — telemetry.beacon_enabled via PATCH /api/config/kirocrew — so the two controls cannot disagree and the choice survives restarts and upgrades.

  • Only the boolean is dashboard-editable. telemetry.beacon_endpoint is deliberately absent from _EDITABLE_CONFIG: exposing it would let a dashboard caller redirect the heartbeat to an arbitrary host.
  • GET /api/telemetry/beacon (handlers/telemetry.py::api_beacon_status) feeds the control. It returns the stored flag (enabled) and the effective verdict (would_send + reason), because the env var, a CI host, a non-default data home, or a config.local.json overlay each suppress sending independently of the flag. A privacy control that reads "on" while something else silences the beacon — or "off" while it still sends — is a false promise, so the panel states which one is in force.
  • env_override reports specifically whether KIROCREW_TELEMETRY_DISABLED pins the state; when it does, the toggle is disabled rather than offering a write that cannot take effect.
  • overlay_override does the same for a config.local.json entry. config.local.json deep-merges over config.json — the file the toggle writes — so an entry there would let the switch snap back to the overlay's value after a successful save. The endpoint reports it and the panel disables the toggle and names the file (the same case kirocrew telemetry disable detects and reports; see cli_commands._telemetry). The probe is best-effort: a missing, unreadable, or malformed overlay reports "not pinned", since enabled already carries the authoritative effective value.
  • The handler routes KiroCrewConfig.load() and the overlay probe through asyncio.to_thread — both stat/read files, and this runs on the aiohttp event loop, where a synchronous read stalls every other request behind it.
  • The endpoint is read-only, never materializes an install id (status(create=False)), does not return the id itself, and fails toward off on an unreadable config — a diagnostic must not 500, and must never claim telemetry is on when that cannot be proven.

In-product privacy disclosure

The dashboard discloses the default-on beacon without turning disclosure into a consent flow:

  • The first-run Privacy chapter is the disclosure surface. There is no passive banner above the routed content: a dismissible strip competed with the chapter that already discloses the same thing, and its "Privacy details" link was the only thing it added.
  • Settings → Privacy is the durable surface. It explains the default-on, at-most-daily beacon and its exact five fields: random installation id, app version, Python minor version, installation channel, and first-run flag. PrivacyPanel.test.tsx asserts both that all five are named and that the four removed fields are absent, so this copy cannot drift from beacon.payload() in either direction. It also states the excluded data, carries the opt-out toggle (above), and keeps the status/disable commands beneath it — the CLI remains documented because it is the only way to override a config.local.json overlay and the only control on a headless host.
  • The mandatory first-run Privacy chapter (components/PrivacyChapter.tsx, chapter 2 of 3) shows the same disclosure and the same toggle in the OnboardingChapterShell layout, so a new user sees what is sent and can opt out before reaching the product. Every path out of chapter 1 routes through it, including "Skip all", and it offers no skip and no Escape. It is still not a consent gate: Continue is always enabled and never requires a choice, because the default is a decision the user can change here, in Settings → Privacy, or from the CLI.
  • The first heartbeat is withheld until that chapter is acknowledged. The gateway starts the beacon thread at boot, before the dashboard has rendered, so an ungated fresh install would ping before the user could decline: an opt-out offered only after the fact. Continue persists dashboard.privacy_acked (and kirocrew telemetry enable|disable sets it too, for headless hosts), which beacon.telemetry_permitted reads. The gate is first-egress only (beacon.is_first_send()): an install that has already sent is past the disclosure, and keying every heartbeat on the flag would silence it permanently rather than once. dashboard.privacy_acked falls back to dashboard.onboarded, so a user who finished first run before the chapter existed is never re-gated. It gates the install receipt too, via the shared consent ladder.
  • The disclosure copy and the control are single-sourced in components/PrivacyDisclosure.tsx (PrivacyDisclosureSections, TelemetryToggle, PrivacyCommandList) and consumed by both surfaces, so the first-run explanation and the durable panel cannot drift.
  • The durable surface distinguishes local usage/context records from optional performance metrics. Performance metrics are off by default and remain local when enabled, with one explicit exception: they egress only when the operator configures an OTLP endpoint. It also carries the recording switch over telemetry.enabled — a second, independent control from the beacon toggle above — fed by GET /api/telemetry/collection and disabled when an env var, a config.local.json overlay, or a configured OTLP endpoint means the write cannot take effect (or would start egress).
  • The panel renders a stable reason_code (beacon.REASONS), never the sibling reason string. reason is untranslated operator prose (already sent today (2026-08-04)) kept for logs and bug reports; interpolating it put a developer diagnostic on screen in all 10 languages. An unrecognized code falls back to a generic translated note rather than rendering a raw key.
  • These surfaces add no tracking. They delay exactly one thing, the first heartbeat, until the disclosure has been shown, and otherwise only explain behavior and controls that already exist.

Official-app install receipt (apps/install_receipt.py)

A successful install_from_registry call emits one best-effort receipt only when the selected row came from the bundled or edition-contributed official catalog. The row's _registry marker is the authoritative negative discriminator: a user-configured external registry row is refused before dispatch. Local-path installs and self-registration never call the sender. This provenance gate keeps private/corporate app names on the host.

The sender mirrors the beacon's posture: the same endpoint, the factored shared consent ladder (telemetry.beacon_enabled, KIROCREW_TELEMETRY_DISABLED, the capabilities.telemetry ceiling, CI/test suppression, and non-default-home suppression), a detached daemon thread, a 5-second timeout, and silent failure. It intentionally does not share the beacon's daily throttle because each successful app install is a separate event.

The exact GET route is:

/b/1/install/<app-slug>?t=<token>&k=<fresh|update>&v=<release>
Location/field Contract
<app-slug> Public official-catalog identifier in the path. No custom-source or local app slug is eligible.
t First 32 hex characters of HMAC-SHA256(key=receipt_secret, msg=b"app-install:" + app_slug), where receipt_secret is a 64-hex random secret generated on first use, stored owner-only as app_receipt_secret under the data home, and never transmitted anywhere — deliberately independent of the beacon install id, which the collector already holds from every heartbeat and could otherwise use to recompute tokens for public slugs and link one installation's receipts across apps. Deterministic for one installation and app, different across apps, and joinable neither into an installed-app profile nor to any heartbeat row. If the secret cannot be read or created, no receipt is sent.
k Exactly fresh or update, derived from whether the app was installed before the successful call. Updates must not inflate adoption rank.
v Kiro Crew release normalized by beacon.release(); build stamps are removed.

test/test_install_receipt.py mocks the network and pins every suppression, token, URL, provenance, success-only, and fresh/update property. Beacon tests remain the regression contract for the shared gate's original behavior.

Server-side AppInstallations rollup (infrastructure follow-up)

The telemetry-account owner must extend the existing CloudFront → S3 → Athena → Lambda aggregation outside this repository. The product-side contract is:

  1. Select only /b/1/install/<slug> rows with an official catalog slug, a 32-character lowercase-hex t, k in {fresh, update}, and a normalized release v; retain the existing no-client-IP/no-User-Agent log projection.
  2. For k=fresh, publish AppInstallations as the count of distinct (app_slug, t) pairs per UTC rollup window (equivalently COUNT(DISTINCT t) grouped by app slug). Duplicate delivery or reinstall from the same data home must not inflate the app's count.
  3. Keep k=update in a separate update series; never add it to install rank. v may provide a bounded release breakdown, but is not part of uniqueness.
  4. Feed only aggregate per-app results to catalog-ranking publication. Never expose or join raw tokens, and never join receipt rows to heartbeat ids.

The Athena query and aggregator Lambda live in the telemetry AWS account, so that implementation and deployment are an explicit infra-account task outside this repo and outside this PR.

Cross-machine identity hazard

A snapshot restored onto a second machine must not clone the id — two hosts sharing one would collapse into a single Daily Active Instance, and the copied stamp would suppress the new host's sends.

This is guaranteed by non-selection, NOT by a basename filter, and the distinction matters: portability.EXPORT_EXCLUDE and snapshot_components.NEVER_SNAPSHOT_FILES are matched by BASENAME over the staged workspace/, plan_memory/ and skills/ trees, so registering a beacon filename there would silently drop any user file that happens to share the name — real data loss on restore, in exchange for nothing. Root-level export copies a hard-coded allowlist (config.json, hooks.json, crons.json, notifications.jsonl, project_dir, workspace_dir) and snapshot staging copies an explicit per-component list (CORE_FILES); neither names a beacon file, so the root paths are never selected in the first place.

TestSnapshotAndPortabilityRegistration pins the whole property in both directions: the names are absent from both filters, absent from the export allowlist and CORE_FILES, and a workspace file sharing either name survives.

Bounded, symlink-safe state reads

All beacon state goes through _read_state(), never Path.read_text(), with three guards:

  1. Regular files only (lstat + S_ISREG). read_text FOLLOWS symlinks, so a link at beacon_install_id pointing at /dev/zero made the read allocate unboundedly until OOM — inside the gateway's beacon thread. The check also rejects FIFOs (whose open() blocks forever) and device nodes, without ever opening them.
  2. Bounded length (_MAX_STATE_BYTES, 4 KiB). Even a regular file can be enormous if a log is rotated onto the name.
  3. Lenient decode (errors="replace"). A strict decode raises UnicodeDecodeError, which is a ValueError and not an OSError, so it escaped the callers' handlers and killed kirocrew telemetry status.

Anything unreadable returns "", which every caller already treats as absent/corrupt — so the id regenerates rather than merely not crashing. The /dev/zero and FIFO tests use a real thread timeout, because the failure mode is "never returns", which a plain assertion cannot catch.

Server side (telemetry account, us-west-2)

Zero application code — the access log is the data product:

client ─GET /b/1/<id>?v&py&dist&first_seen─> CloudFront E1YM983XX3ASBM
                                     │ CloudFront Function returns 204 at the edge
                                     ▼
        standard logging v2 → s3://kirocrew-beacon-logs (PERMANENT, tiered)
                                     ▼
              Athena kirocrew_analytics.beacon_logs (partition projection)
                                     ▼
        kirocrew-beacon-aggregator Lambda (daily 00:20 UTC) writes BOTH:
                    ├── CloudWatch KiroCrew/Product  → dashboard (~15-month view)
                    └── kirocrew_analytics.beacon_daily → PERMANENT record

Metrics published. DailyActiveInstances, BeaconPings, NewInstallations, ActiveByVersion, ActiveByPython, ActiveByDistribution, and ActiveByCountry. ActiveByChannel / ActiveByOS / ActiveByArch were removed with their source fields; their historical CloudWatch points and rollup rows are left in place (the data is real for the days it covers — deleting it would be rewriting history to match a current schema).

country is retained, and is the one field the client does not send. It is derived at the CloudFront edge and is coarse (a 2-letter code); the IP it comes from is never written to storage, since the log delivery does not select c-ip at all. Dropping it would mean removing c-country from the delivery's recordFields, which shifts every column in the TSV — the Glue table's six columns are positional, so new log lines would silently misparse (the status = '204' filter would match nothing) until the table was migrated. Keeping the least identifying field in the set was the better trade than a schema migration on a live pipeline.

Retention: S3/Athena is permanent, CloudWatch is a 15-month view

CloudWatch cannot be the durable store. Metric data is retained for at most 15 months and expires on a rolling basis (1-min → 15 days, 5-min → 63 days, 1-hour → 455 days), and that ceiling is not configurable. So the dashboard is inherently a ~15-month window, by AWS design rather than by our choice.

The permanent record is therefore two things in S3:

  • Raw logs — s3://kirocrew-beacon-logs. The lifecycle policy has no Expiration on any rule; objects only transition (Standard → Standard-IA at 90d → Glacier Instant Retrieval at 365d) to cut cost. Glacier Flexible Retrieval / Deep Archive are deliberately avoided — they require an async restore before a read, which would silently break the long-range Athena queries this design exists to support. Versioning is on; only noncurrent versions are pruned (365d).
  • Daily rollup — kirocrew_analytics.beacon_daily (Parquet, stays in Standard forever). One small row per (day, metric, dimension, value). This is what makes "permanent" useful: raw logs grow linearly forever, so a multi-year dashboard query would scan every line ever written, while the rollup keeps such queries fast and cheap.

_persist_rollup() is idempotent — it deletes the target day's rows before inserting, so a backfill or a retry after a partial failure cannot double-count. It runs last in the handler, after the CloudWatch puts, because CloudWatch is best-effort presentation while the rollup is the durable store: a rollup failure must surface as a Lambda invocation error (visible on the dashboard's error widget) rather than being masked by an otherwise-successful metric write.

Destructive-rewrite guard (do not remove). That same idempotent delete makes an empty query result destructive: it rewrites a day to nothing. A if not facts check is not sufficient, because DailyActiveInstances, BeaconPings and NewInstallations are appended unconditionally — as zeros — so an empty day still reaches the delete with three all-zero rows.

A run that queries a partition whose logs do not exist reports SUCCEEDED with no error and publishes zeros, so without a guard it would wipe the rollup to zero rows — the durable record destroyed by a "successful" invocation. The guard skips the rewrite unless at least one fact is non-zero (an all-zero day carries no information, so skipping is lossless), and an empty partition logs an explicit WARNING naming the partition, because a silent success is what makes the failure hard to see.

Object Lock is deliberately NOT enabled. It would make the logs literally undeletable, which sounds like "permanent" but is the wrong trade for a privacy-sensitive dataset: it would also remove our own ability to purge after an operator mistake, a schema error, or a future deletion obligation. The goal is "retained indefinitely by policy", not "physically impossible to delete".

The aggregator cannot delete the permanent record. Its IAM policy grants s3:PutObject/s3:DeleteObject on kirocrew-beacon-logs/rollup/* only — raw-log access is read-only, so the component that consumes the history has no permission to destroy it.

No client IP is ever stored. The log delivery's recordFields selects only date, time, cs-uri-stem, cs-uri-query, c-country, sc-status — c-ip, x-forwarded-for, User-Agent and Cookie are simply not delivered. Verified against a real delivered log file. This is why the design uses a Lambda-free CDN log path rather than CDN logging with default fields, and it is what makes the "no IP" claim structural rather than a promise.

Aggregator timestamp rule (load-bearing): metrics are stamped at the end of the target day, clamped to now - 1min. CloudWatch accepts timestamps up to two weeks old but points 24h+ old can take 48 hours to become queryable, while <3h old are near-immediate. Stamping midnight-of- yesterday from an 02:30 run put every point in the 48-hour bucket and the dashboard rendered empty despite the data being accepted; the clamp is needed because a same-day backfill's 23:59 is in the future and PutMetricData rejects >2h ahead. Both failure modes were hit in development.

Model CDN. embeddings_settings.py::_DEFAULT_MODEL_URL points at kirocrew-models (distribution E2UX23B48LKM6V, OAC-only bucket access). The _GGUF_SHA256 pin remains the sole integrity gate, so a tampered CDN object can only fail verification. Because _ensure_downloaded() returns early when model_ready(), a CDN request is a first-install signal, not a DAU signal — the dashboard shows it as "downloads", deliberately separate from DAI.

Crew assembly extents

context_blocks.measure_prompt reuses the existing marker classifier and an authoritative user span to report exact characters and UTF-8 bytes at the crew_assembly boundary. ContextBuilder.build_message emits both through the existing recorder as kirocrew.context.block.chars and .bytes, with bounded section, domain, boundary and lifecycle attributes. Lifecycles are fresh, warm, resume, reinjection and minimal; they describe composition, not proof of provider receipt or retention. Domains distinguish request, contract, replay, following-interaction guidance and background. Unknown marker gaps remain visible. No body, source path, user identity, token conversion or cost is recorded.

The reading explicitly reports native=UNKNOWN and external_mcp=UNKNOWN. Provider-owned history, resources, native prompts and external MCP serialization cannot be measured from this string. Dashboard prefixes added after this boundary remain covered by the existing final-prompt character classifier, not by a claim that the assembly reading measured them. These extents are not full model input. Synthetic Unicode fixtures assert both closure sums and request attribution.

Crew assembly measurement gate

ContextBuilder.build_message skips measure_prompt when both its metrics recorder and DEBUG logging are disabled. Attribution depends on the caller's accurate user_text_range: Slack passes the actual user's slice after adding a cancelled-turn restore prefix, so that prefix remains context rather than request text. Character and UTF-8 byte totals describe Crew assembly only; native provider inputs, external MCP schemas and actual model calling behavior remain unmeasured.

Usage metrics: who started the work, and which features were used

These instruments answer what Kiro Crew is used for. Every attribute value is a closed list defined in code and pinned by a test; an unknown value is other. None carries an id, a name the operator chose, a path, a URL, a model id or text.

Shared attributes (metrics/usage_attrs.py):

Attribute Values Source
trigger_source user, cron, heartbeat, monitor, subagent, workflow, webhook, app, agent, system, other stated by the code that starts the work; the dashboard maps its turn actor, and user needs a person's send that is not a restored queue row. A caller that states nothing records other, never user
origin the 11 channel namespaces, dashboard, cli, webhook, api, internal, other a rollup of telemetry_channel_of; a webhook names itself
session_kind chat, channel, cron, subagent, workflow_step, task_runner, heartbeat, background, other a rollup of session_source
Metric Type Attributes Recorded at
kirocrew.turn.completed counter outcome, trigger_source, origin, harness inside emit_turn_duration, so it counts the same turns as kirocrew.turn.duration with the same outcome (metrics/turn_completed.py)
kirocrew.turn.completed.duration histogram (ms, turn buckets) outcome, trigger_source, harness same; only for a positive duration. Not in PRODUCT_METRICS yet: the allowlist takes counters and gauges only, and kirocrew.turn.duration carries open model and provider attributes that a destination cannot narrow away, since it filters by name
kirocrew.session.started counter adds session_kind, trigger_source record_session_started; a key whose kind fixes the opener uses it, else the caller's or the running turn's trigger
kirocrew.subagent.completed counter adds result: text / empty / none the terminal report, past the finalize claim
kirocrew.autonudge.loops.stopped counter adds ended_by: agent_stop / user_stop / budget / subject_done / stalled / error / other the store's commit point; a fixed grouping of reason
kirocrew.memory.recalled counter path: tool / context; hit an explicit recall after its receipt commits; each admitted turn's composition
kirocrew.lesson.injected counter hit each admitted turn composed with a lesson tier
kirocrew.skill.lazy_load.count counter adds skill: a shipped skill id (nested ones by their path), custom or other each on-demand skill body load; the duration histogram keeps hit alone
kirocrew.app.task.completed counter app: a shipped built-in app id (BUILTIN_NAMES, its apps/builtins/ directory name) or private; outcome: ok / error / other a finished Papyrus compile (record_app_task)
kirocrew.channel.message.received counter channel, admitted, automation run_channel_turn, once admission decides
kirocrew.ui.control.clicked counter control (8 values) POST /api/telemetry/ui-control, owner only, from the dashboard's option chips, question card and follow-up card

Recording is an in-memory add after the measured action commits, and a recording failure never changes that action's result.