Local-first metrics telemetry built on the OpenTelemetry SDK (Apache-2.0 / CNCF).
The trunk is designed so future work is purely adding instrument calls at call
sites — never changing this plumbing. Default OFF (telemetry.enabled: false): all metric call sites are cheap no-ops and nothing is written or
exported, byte-identical to no telemetry (mirrors the mcp_gateway.enabled /
skills.lazy_load opt-in convention). Any instrument that needs durable state to
survive a process death — currently only the session-lifetime breadcrumb — must
gate that state on the same consent, and fail closed when consent cannot be
read, so this sentence stays literally true.
Source: the src/kiro_crew/metrics/ package. The tables below cover the modules
that carry contract meaning; the package holds more (per-domain instrument modules
for sessions, turns, tool calls, DB, process and inventory gauges, plus
temporality.py and events.py), so read the directory rather than treating any
list here as the inventory. Tests: test/metrics/.
| File | Purpose |
|---|---|
schema.py |
Namespace constants (NS_CORE = "kirocrew.", NS_GENAI = "gen_ai.", NS_APP_PREFIX = "app.") + validate_name / validate_attrs / redact guardrails. Documents the low-cardinality contract. |
recorder.py |
MetricsRecorder — facade over the OTEL Meter. Every metric passes namespace + privacy guardrails BEFORE reaching an instrument. Instrument-cache creation is lock-guarded (atomic check-then-create). Best-effort: a telemetry failure never propagates to the caller. meter=None = no-op recorder. |
provider.py |
Consent gate + process-global recorder (get_recorder()) + graceful shutdown() / reset_for_testing(). get_recorder() serves a memoized recorder and re-resolves the telemetry.enabled consent value every _CONSENT_RECHECK_SECS (30s), rebuilding when it moved — see "Recorder lifecycle & threading" below. Public consent surface: env_pin() / TELEMETRY_ENV_VAR. When enabled, wires a PeriodicExportingMetricReader to the local JSONL exporter. Installs one View per instrument from histogram_bounds() (_HISTOGRAM_BUCKETS_MS for millisecond instruments plus _HISTOGRAM_BUCKETS_BY_UNIT for the non-ms ones), each with its own ExplicitBucketHistogramAggregation boundaries (see below) — deliberately NOT a catch-all instrument_type=Histogram View. |
local_exporter.py |
JsonlMetricExporter — appends one JSON line per export cycle to <dir>/metrics-YYYY-MM-DD-<pid>.jsonl (default dir ~/.kiro/crew/metrics). Per-PID single-writer shards keep append + rotation lock-free, so concurrent exporters do not lose DELTA cycles. A private .metrics.lock serializes only retention sweeps; pruning skips canonical shards owned by live PIDs or modified within the safety window. Bounded retention (rec #14): shards rotate before an append exceeds max_total_mb; closed/expired shards are pruned directly by age and oldest-first size. Pruning is throttled to at most once per 300s and fully best-effort. Dir mode is 0o700, file mode 0o600, and nothing egresses the host. Declares DELTA preferred_temporality for Counter/UpDownCounter/Histogram so daily aggregation is an element-wise sum across cycles/PIDs; the map itself lives in metrics/temporality.py, which owns it for the OTLP leg too (declaring it inline here is what let that leg ship CUMULATIVE for the same instruments with nothing failing). Observable counters are deliberately NOT mapped and would export CUMULATIVE: the delta baseline lives in the provider, which is rebuilt in-process on a telemetry consent change, so DELTA would re-emit the process-lifetime total once per rebuild. Nothing registers an observable counter today — the process CPU/GC readings and the inventory probe-failure count are lifetime-total GAUGES, which is what removed the last cumulative series — so the entry guards the next one; the aggregator still reduces cumulative streams window-relative (deterministic identity boundary + time-ordered legacy reset detection + first-in-window baseline) for the shards written before that switch. Process identity: each record is stamped once at resource level with kirocrew.process.start_time (schema.RESOURCE_ATTR_PROCESS_START_TIME) — the writing process's OS start-time token from platform_compat.own_process_start_time(), module-cached so provider rebuilds inside one process stamp the SAME value, and reboot-unique (Linux start ticks + boot UUID; macOS microsecond proc_pidinfo instant; Windows creation FILETIME). A read that cannot honor one-token-one-process (unreadable boot UUID, no libproc, 1s-only sources) emits NO token rather than an aliasable coarse one — a degraded token would merge lifetimes AND mute the reset heuristic that catches merges. The shard-filename PID plus this token identify a process beyond PID reuse, making the aggregator's cumulative reset detection deterministic. The stamp lands on the serialized JSONL line, never on the SDK Resource — that Resource also feeds the opt-in OTLP reader, and this host-local token must not egress. Fail-soft: when the platform read is unavailable the field is absent and the aggregator's legacy value heuristic applies. Resource level, not a metric attribute, so it never multiplies series cardinality. |
temporality.py |
Single owner of the instrument-kind -> temporality map, for the local sink AND the opt-in OTLP leg. delta_preference() returns a fresh DELTA map for Counter/UpDownCounter/Histogram (ObservableCounter deliberately absent, gauges never applicable); otlp_preference() returns it, or None to stand aside when OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE is set. Exists because the map lived inline in local_exporter.__init__ while _build_otlp_reader passed nothing, so the two destinations reported the same instruments DELTA on disk and CUMULATIVE on the wire with nothing failing. |
http_metrics.py |
Gateway HTTP observability (rec #1): record_boot_to_ready() (boot-to-ready histogram) + make_route_latency_middleware() (per-route latency, wired as the outermost middleware on both start_dashboard/start_api_server). Bounds route_template cardinality via collect_route_templates() (build-time snapshot) + route_template() (__unknown__ fallback); clamps method to a fixed allowlist and status_class to 1xx..5xx/other. Upgraded WebSocket connections and text/event-stream SSE responses are excluded because their handler elapsed time is connection/turn lifetime, not HTTP request latency. Best-effort — a telemetry failure never alters a response. |
get_recorder() returns a memoized recorder on a fast path guarded by a
monotonic clock (_consent_recheck_due): once a recorder exists it is handed back
directly until the recheck window elapses. Every _CONSENT_RECHECK_SECS (30s) the
call hands a consent check to a worker (_schedule_consent_check_locked ->
_consent_worker), which re-reads telemetry.enabled and rebuilds when it moved.
That is what makes kirocrew config set telemetry.enabled true — a write from a
SEPARATE process — take effect without a gateway restart. Inside the gateway the
wait is skipped by the config watcher: watch_config(), registered at gateway boot,
subscribes _on_config_change to the telemetry section, and any telemetry.*
write awaits asyncio.to_thread(shutdown) so the next metric rebuilds from the new
values. A config that cannot be READ
yields "no change" rather than False, so a transient read error never tears down a
working recorder; the worker stamps the recheck clock either way, so an unreadable
config cannot turn every metric call into a fresh file read.
get_recorder() itself never reads config and never builds anything, apart
from the very first build of the process, which has nothing to serve in the
meantime. KiroCrewConfig.load() is a fingerprint-cache hit in the steady state
(~0.3ms) but a full read plus schema validation when the file actually changed
(~14ms), and the rebuild costs ~57ms of SDK import — neither belongs on the event
loop, which the route-latency middleware drives on every HTTP request. The
consequence to know: the recheck is eventual in BOTH directions. A change is
noticed at the window boundary and lands a thread hop later, which is immaterial
against the window it already sits behind. _check_in_flight keeps a busy window
from spawning one worker per request, and is cleared in the worker's finally so a
crash costs one window rather than stranding the check.
Consent resolution is env-first: env_pin() reads TELEMETRY_ENV_VAR
(KIROCREW_TELEMETRY) and, when set, decides the effective state regardless of the
config flag; _consent_enabled() falls back to telemetry.enabled.
_lock is never held across a provider or reader shutdown.
_take_provider_locked() clears the globals under the lock and RETURNS the
provider; the flush happens after release. A provider shutdown joins each reader's
export thread (30s deadline) and, with telemetry.otlp_endpoint set, ends in a
synchronous network POST — holding the lock across it would stall get_recorder()
on lock ACQUISITION, and the route-latency middleware calls get_recorder() on the
event loop for every HTTP request. This applies to every path that reaches a
teardown, including the one that discards a superseded build.
Which thread flushes depends on who dropped the recorder. The consent worker
flushes on its own thread, after releasing the lock. shutdown() flushes on the
CALLER's thread, so process teardown and the config-watcher applier (which calls it
via asyncio.to_thread) both observe completion — still not under the lock.
Partially-constructed readers from a failed init are reaped off-thread the same way
(_reap_readers_detached), because a reader shutdown performs a final export.
_build_recorder() writes no module state. It returns a _Build tuple
(recorder, provider, resolved consent) that a caller installs under the lock via
_install_locked(), so only the install is a critical section. Consent is recorded
alongside the recorder even on a host that can never record (OTel absent), because
leaving it unset would make every recheck window see a difference and rebuild a
no-op recorder in a loop.
A _build_generation counter makes a mid-build disable stick. Each rebuild
captures the counter and rechecks it before installing; _take_provider_locked()
bumps it, so a disable that lands while a build is in flight is not undone by the
build finishing afterwards — the superseded build's provider is flushed (outside the
lock) and discarded instead. Every consent change starts its own worker rather than
skipping when one is already running: skipping would leave the recorded consent
updated while the recorder stayed a no-op, and the next recheck would then find no
difference and never retry.
Only the FIRST build of a process is synchronous (_ever_built is still
False), because there is no recorder to serve in the meantime and that path is not
a steady-state request path. Every later build is a REbuild and goes off-thread,
including one after shutdown() cleared the state — otherwise the config route's
own write would put the SDK import back on the event loop. reset_for_testing()
clears _ever_built, so a test's next build is synchronous and assertable without
polling. shutdown() bumps the generation but does not stop an already-running
consent worker, so reset_for_testing() also waits (bounded, _RESET_WAIT_BOUND_SECS)
for _check_in_flight to clear before returning — raising if the bound expires —
so a worker left running by an earlier test can never mutate module state
underneath the next one.
One consequence of the detached flush: while it is still in its join, a re-enable can put a second exporter on the same per-PID shard, which the local exporter's single-writer assumption does not cover. Both sides swallow their IO errors, so the worst case is one dropped export cycle rather than a corrupt shard.
- Namespace: core callers must use
kirocrew.*orgen_ai.*; app callers must useapp.<app_id>.*and cannot spoof the core/gen_ai namespaces (validate_nameraisesValueError, the recorder swallows it, nothing is recorded). - Privacy: string attribute values pass
redact()— AKIA/ASIA keys,SecretAccessKey=, private-key headers, 40+ char hex, JWT shapes,password=/token=patterns, base64-encoded credential variants, and a Shannon-entropy heuristic all yield"[REDACTED]". The first-partykiro_crew.securityscrubbers (redact_credentials,redact_exfiltration_urls— both return(cleaned, warnings)tuples) are also consulted. Long non-suspicious strings are truncated toMAX_ATTR_VALUE_LEN(128). - Cardinality: metric names + attribute values must be low-cardinality
constants; attribute count is capped at
MAX_ATTR_COUNT(32). Instrument caches are keyed by name and never evicted.
TelemetryConfig in config/sections_telemetry.py (parsed by config/loader_builders.py; section telemetry in
~/.kiro/crew/config.json):
| Field | Default | Meaning |
|---|---|---|
enabled |
false |
Main switch. Off = no-op recorder, nothing written. Editable from the dashboard (Settings → Privacy) as well as the config file, kirocrew config set, and the env var; re-resolved live, so a change takes effect without a restart. |
local_dir |
"" |
JSONL shard dir; empty = ~/.kiro/crew/metrics. ~ expansion supported. |
export_interval_seconds |
60 |
Flush interval (floored to 1). |
retention_days |
0 |
Age pruning is disabled by default to preserve pre-existing history on upgrade. Set a positive day window to opt in (rec #14). |
max_total_mb |
0 |
Size pruning is disabled by default to preserve pre-existing history on upgrade. Set a positive opportunistic directory budget to opt in; protected active writers can temporarily exceed it (rec #14). |
otlp_endpoint |
"" |
Opt-in OTLP/HTTP metrics endpoint (e.g. http://localhost:4318/v1/metrics). Empty = no network egress (default). When set, aggregated metrics are ALSO pushed to this collector in addition to the local JSONL sink; requires pip install "opentelemetry-exporter-otlp-proto-http==1.44.0" (rec #1). Metrics signal only: no spans or log records exist, so _build_otlp_reader logs a WARNING (destination name only, never the URL) when the endpoint path ends in /v1/traces or /v1/logs, and still builds the reader. |
Field validation (TelemetryConfig.__post_init__): export_interval_seconds
below 1 is floored to 1; negative retention_days / max_total_mb are clamped
to 0 (cap disabled) rather than being interpreted as "prune everything".
Default posture — nothing collected, nothing leaves the host.
telemetry.enabled defaults false, so every metric call site is a cheap no-op
and no file is written — including the session-lifetime breadcrumb, which is
gated on the same consent and fails closed. Even once local collection is
enabled, otlp_endpoint defaults empty and otlp_endpoints defaults to an
empty list, so no data ever leaves the machine unless the operator explicitly
sets an OTLP endpoint (or the build ships a collector of its own).
Easy opt-in (four equivalent ways):
- Config flag: set
"telemetry": {"enabled": true}in~/.kiro/crew/config.json. - CLI:
kirocrew config set telemetry.enabled true. - Dashboard: the recording switch in Settings → Privacy, which writes the same
key through
PATCH /api/config/kirocrew(telemetry.enabledis in_EDITABLE_CONFIG). That route refusestruewith HTTP 409 when any OTLP destination resolves (telemetry.otlp_endpoint, atelemetry.otlp_endpointsentry, or one the build ships):_build_recorderattaches an OTLP reader whenever the active telemetry provider supplies a destination — which the public default does for any non-empty endpoint — so enabling from a switch offered as local-only would start network egress. The endpoint is chosen in the config file, so that is where enabling on such a host is done. Disabling is always allowed — a narrower local choice always composes. An unreadable config also fails closed with a 409. A successful write reaches the recorder through the config-watcher applier (watch_config->_on_config_change->asyncio.to_thread(shutdown)), not through the route itself, so the value applies on the very next metric rather than at the next recheck. - Env var: export
KIROCREW_TELEMETRY=1(also acceptstrue/yes/on;0/false/no/offforce-disables). The env var overrides the config flag and is handy for CI / containers / one-off debugging. It gates local collection only — it never enables network egress. Resolved by the publicprovider.env_pin()and consumed byprovider._consent_enabled().
None of these requires a gateway restart: get_recorder() re-resolves consent
every _CONSENT_RECHECK_SECS (30s) and rebuilds when it moved (see "Recorder
lifecycle & threading"). The gateway process is where the session/turn/HTTP
metrics are recorded; other kirocrew processes pick the value up on their own
recheck or at their next start.
Any telemetry.* change rebuilds the recorder, not just enabled.
provider.watch_config() registers one applier on the telemetry section, and its
callback calls shutdown() — the next get_recorder() then builds a fresh
recorder from the new config. That covers the settings the 30s consent recheck
cannot see: the export cadence and every other value _build_recorder reads
once. The recheck itself compares three things against the live recorder:
enabled, the agent pair the harness label comes from (_agent_key), and a
sha256 of telemetry.otlp_endpoint plus telemetry.otlp_endpoints
(_endpoints_key). So a process with no config watcher (the MCP gateway
daemon, through recheck_consent) still drops the reader of a list entry that
was turned off or removed, within one window. Registration is
idempotent per process, so a re-entered boot path cannot stack appliers that each
rebuild. The rebuild is lazy by design — shutdown() only drops the memo, so the
~57ms of SDK import is paid on the next metric call rather than on the config
watcher's own thread.
Beacon fields are the exception: they are read once for the boot heartbeat, so a change to them applies at the next start.
External OTLP egress (opt-in, off by default): egress destinations are
pluggable. _build_recorder asks the active TelemetryProvider for them via
otlp_destinations(cfg) and attaches one PeriodicExportingMetricReader per
returned destination that names the "metrics" signal (provider._otlp_destinations
resolves, provider._build_otlp_reader builds one reader). Each destination's
host must also pass the security policy's capabilities.telemetry otlp_hosts
list (otlp_governance.governed_destinations before the reader is built and otlp_governance.gate_exporter before each export; see governance.md), and a
policy that pins capabilities.telemetry off drops every destination. The public
DefaultTelemetryProvider turns a non-empty otlp_endpoint into exactly one
destination and returns none when it is empty, so a standalone build behaves as it
did when this module constructed that exporter itself. Readers are appended AFTER
the local JSONL sink, so an edition can add destinations but can never remove or
replace the sink the dashboard reads, and it supplies no cadence — the export
interval stays telemetry.export_interval_seconds. Nor does it supply a
TEMPORALITY: _build_otlp_reader passes the same DELTA map the local sink uses
(metrics.temporality.delta_preference, the single owner for both sinks), so one
MeterProvider's two destinations cannot describe the same instrument two
different ways. It passes NOTHING when
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE is set, leaving that variable
in full control: the exporter applies an explicit dict ON TOP of whichever base
the variable chose (OTLPMetricExporterMixin._get_temporality in the
otlp-proto-common package — the reader lives there, not in the SDK tree or the
http exporter tree), so passing the map unconditionally would silently override an
operator who asked for CUMULATIVE.
A destination carries name (a non-secret label used in logs, because the
endpoint value never is), endpoint, signals, and optionally an
authenticated session. The session is why the seam exists: requests
re-evaluates Session.auth per request, so a credential that rotates during
process lifetime (an OIDC/SSO id_token, STS credentials) is re-read on every
export, where OTEL_EXPORTER_OTLP_HEADERS — the only injection point before this
seam — freezes into the exporter's session at construction and starts returning
401 once it rotates. A destination built in code carries static headers on
Session.headers; one declared as data names them in headers_env (below).
The method must be cheap and side-effect-free per call: it is read once per
recorder build AND on every egress-posture read (the Privacy panel's status, and
each telemetry.enabled config write), so an edition must build its transport
once rather than acquiring a credential inside it.
Filtering is deny-by-default: a destination with an empty endpoint, or one aimed
at a signal this core does not emit, is dropped rather than trusted. A provider
that raises contributes nothing and is reported at WARNING, and telemetry keeps
working local-only; the same is true when the platform context cannot be composed,
because for an egress seam "no destinations" is the closed state.
The MCP gateway daemon resolves its provider without booting. gatewayd never
runs boot_platform (it is not the composition process; see app_call.py), so on
a non-standalone profile current_context() raises there, and a recorder that
resolves destinations through it builds local-only (its log says
platform context unavailable; OTLP egress disabled (local-only)).
gatewayd._prepare_telemetry (from main()) builds the recorder before the event
loop starts, composing nothing: a standalone host resolves the lazy default context,
so its telemetry.otlp_endpoint is live from the first build, and the pool gauges
exist from startup. A consenting host then composes ONLY the edition's telemetry
provider on a background thread through platform.bootstrap.compose_telemetry_provider
(the same profile resolution, admission gate, contract check and security floor as
boot, shared with bootstrap_context through _checked_companion) and hands it to
provider.use_telemetry_provider. NO context is installed, so every other
current_context() reader in the daemon keeps the behavior it had (why the daemon
composes rather than receiving destinations from its parent is recorded in
platform-context.md, under mcp_gateway/gatewayd.py::_compose_and_apply). Composition builds
the edition's whole context and keeps only .telemetry; because it is off the boot path
and the event loop, a slow edition never delays the daemon. Only when it adds a
destination the startup build lacked is the recorder dropped and rebuilt (on the
provider's worker thread); the pool gauges, read live, lose nothing.
Accepted limit: the startup window is local-only. Counters and histograms the
daemon records before that rebuild (kirocrew.mcp.backend.acquire.duration,
kirocrew.mcp.lazy_load.*, kirocrew.mcp.warm_pool.acquire) stay in the local sink
and are never exported, because the swap replaces the MeterProvider and the OTel SDK
cannot hand its cumulative state to a new one. On a non-standalone host that window
is the composition time (about 1 s measured on an edition install) plus the rebuild,
so a backend acquired in the first seconds after a daemon start is under-counted in
the exported series. Making the first acquire wait on composition would put the
edition's start-up cost on a session's first tool call, and replaying recorded points
would need a second store; neither is worth a one-second gap per daemon start.
An off-consent daemon composes nothing
and exports nothing; a later opt-in composes lazily in the off-loop rebuild, and a
factory that raises costs egress only. Consent is re-read only inside get_recorder(),
which the daemon otherwise calls only when a backend is acquired, so
daemon/sweepers.py::_telemetry_consent_sweeper forces a re-read
(provider.recheck_consent) every gatewayd._CONSENT_SWEEP_SECS (half of
_CONSENT_RECHECK_SECS, so 15 s): an idle daemon stops exporting within one tick plus
the config read of a withdrawal, inside the 30 s window. The tick forces the read
rather than calling get_recorder(), because that one rechecks only once a full
window has passed since the worker's last stamp, which lands after the tick that
started the read; a tick per window would re-read only every other tick (about 60 s).
The resource is the one every process builds (_resource_attributes), with the
daemon's own process.pid; its kirocrew.process.* gauges therefore start arriving
as one more process per install, which is the per-process design described under
"One publisher per install". Its resource carries kirocrew.process.role=gatewayd
(absent on every other process), so a query separates the daemon's series, for
example its own RSS as kirocrew.process.memory.rss_bytes{kirocrew.process.role="gatewayd"},
from the main process's without joining on process.pid.
Install support with pip install "opentelemetry-exporter-otlp-proto-http==1.44.0". If a destination is supplied
but the package extra is
not installed, telemetry
degrades to local-only with a warning instead of crashing. The OTLP exporter
sees the same data points as the local sink: the MetricsRecorder facade
sanitises attributes before they reach ANY reader, and call sites are required
to pass low-cardinality constants rather than prompts, content, tokens, paths or
user ids. That sanitisation is defence in depth over the requirement, not a
substitute for it, so egress is only as safe as the call sites feeding it.
Bounded local retention (rec #14, explicit opt-in): both destructive caps
default to 0, so upgrading cannot delete existing telemetry history. Operators
can opt in independently to age and/or size bounds:
- Age cap — set
retention_daysto a positive window (for example7); shards whose mtime is older than that window are eligible for deletion. - Total-size cap — set
max_total_mbto a positive budget (for example128); before an append would exceed the live-shard budget, the exporter rotates that shard and opens a fresh canonical writer. Closed shards are then deleted oldest-first until the combined size is under budget. The active writer is retained; with multiple process-local writers, enforcement remains opportunistic rather than a strict instantaneous directory-wide byte ceiling. In the worst case, live protected shards can temporarily approach the number of active writers timesmax_total_mbbefore those writers rotate and closed shards become eligible for oldest-first deletion. - Both caps are independently opt-in and can be disabled again by setting
the value to
0. - After an operator enables a cap, before the first destructive plan in each
exporter process, retention emits
one fixed, path-free warning and defers deletion for a full 300-second prune
interval. Operators can set either cap to
0during that window. The notice is process-local; there is no persistent migration marker or format to carry into future releases. - Per-PID append + rotation are lock-free. A private cross-process
.metrics.lockserializes only prune sweeps; contention skips pruning but never discards the DELTA payload already appended. Canonical shards owned by live PIDs or modified within the 300-second safety window are not deleted. - Pruning is throttled (≤ once per 300s), runs only AFTER a successful append,
and considers only regular files matching the exact generated grammar
metrics-YYYY-MM-DD-PID[-ROTATION_NS].jsonl; broad-prefix lookalikes, invalid dates, symlinks, and the lock sidecar are excluded. It is fully best-effort — a rotation/prune failure is logged and swallowed, never breaking export.
Must never be recorded: prompts, message/tool content, token counts,
filesystem paths, user ids, and secrets. telemetry.otlp_endpoint is
schema-sensitive so credential-bearing collector URLs are masked by config
API/UI consumers as well as omitted from logs.
That is a contract on call sites — they emit only low-cardinality enum-like
attribute values — backed at the MetricsRecorder facade by the schema.py
guardrails (see below), which redact a string matching a known credential shape,
or clearing the entropy backstop, to "[REDACTED]" before it reaches an
instrument. Namespace validation is exhaustive; attribute redaction is defence
in depth with a bounded reach (schema.py documents where the entropy backstop
can and cannot fire), so it narrows the blast radius of a call site that breaks
the contract rather than making one impossible.
Tests: test/metrics/test_local_exporter.py (retention: direct age cap,
oldest-first size cap, live-writer protection, live-shard rotation, non-blocking
prune lock, append survives prune contention, both-disabled,
broad-prefix/malformed shard lookalikes ignored, export-then-prune never raises),
test/metrics/test_provider.py (default-off, env-var opt-in/opt-out,
OTLP None by default = no egress, OTLP reader built when endpoint set, degrade
when extra missing, plus TestConsentRecheck: out-of-band enable/disable, the
rebuild and the flush both off the calling thread, a mid-rebuild disable not
undone by the build, the hot path not reading config every call, an unreadable
config keeping the live recorder, shutdown() applying a change without waiting
out the window and without holding the lock across the flush, and the env pin
still winning after a config edit), test/test_collection_status_endpoint.py
(GET /api/telemetry/collection: effective state, env/overlay pins, endpoint
presence without the endpoint string),
test/metrics/test_schema.py (redaction / namespace).
The singleton _consolidate session maps to the bounded consolidation source
label through messaging.link's exact-key table. Similarly prefixed keys such as
_consolidateXYZ and _consolidate:123 remain other; this key is not a namespace.
The SDK Resource is the payload contract for both sinks: it labels every
series on the local JSONL line AND, when otlp_endpoint is set, everything
pushed to that collector. _resource_attributes() in provider.py is the
single place it is built, and every value there is either a closed set or an
explicit clamp, because a resource attribute is a label on EVERY series this
process exports — one unbounded value there multiplies every instrument.
| Attribute | Value | Bound |
|---|---|---|
service.name |
kirocrew |
constant |
service.instance.id |
the persisted anonymous install id (beacon.install_id) |
one per install |
process.pid |
os.getpid() |
one per live process |
kirocrew.process.role |
gatewayd in the MCP gateway daemon (provider.mark_gatewayd_process, called by gatewayd._prepare_telemetry before its first build); ABSENT in every other process, so their resources are unchanged |
one constant value |
kirocrew.harness |
the install's CONFIGURED default harness: global agent.acp_backend in its config spelling (any id in ACP_BACKENDS_KNOWN, for example kiro, kas, claude, codex), or claude_code when agent.provider is claude_code (metrics/harness.py::set_configured, read from the same config load as the telemetry section) |
CLOSED: ACP_BACKENDS_KNOWN + claude_code / unknown / other; one value per process |
service.version |
build version, release-clamped via beacon.release |
major.minor.patch, no build stamp |
os.type |
linux / darwin / windows |
CLOSED, else other |
host.arch |
amd64 / arm64 / x86 |
CLOSED, else other |
process.runtime.name |
cpython / pypy / jython / ironpython |
CLOSED, else other |
process.runtime.version |
beacon.python_minor() |
major.minor, never the patch |
host.cpu.logical_count |
os.cpu_count() |
small integer |
Two identities, deliberately separate. service.instance.id counts DEVICES: a
random UUID persisted on disk, never derived from hostname or username (which
on a corporate desktop routinely embed the employee's alias), and stable across
restarts so "this install over time" survives them. process.pid counts
PROCESSES: one install runs several telemetry-enabled processes at once (the
gateway plus spawned agents/apps each build their own recorder), and with an
install-scoped resource alone their gauges and cumulative counters would
interleave into one corrupted series at a backend. Collapsing the two would
report one machine's 6-8 processes as 6-8 machines.
Why the install id may egress while kirocrew.process.start_time may not.
The start-time token is a host-local process fingerprint that exists only to
make the aggregator's cumulative-reset detection deterministic; it has no
fleet-level question to answer, so it is stamped on the JSONL line and kept off
the Resource (see local_exporter.py above). The install id is the opposite:
it is the only thing that can answer "how many distinct devices", it is random
rather than derived, and it is the SAME id the beacon already sends. Note that
omitting the key does not yield an unlabelled resource — the SDK substitutes
its own per-process uuid4, i.e. a fresh series set per restart — so the choice
is between a stable anonymous id and per-restart churn, not between an id and
no id.
The id is MINTED in _build_recorder()'s live branch, immediately before the
resource is assembled, so the first export already carries it and no startup
window can leak the SDK's substitute. That branch runs only under consent True,
so an install with telemetry off creates nothing. Consequence to know: enabling
metrics is what creates the beacon's install-id file, even on a host that has
the beacon itself disabled.
Deliberately ABSENT: the distribution channel (the beacon's data-minimization pass removed its channel field, because channel sharply narrows the crowd a stable id hides in — stamping it on every metric payload would quietly undo that decision) and an install type (no reliable detection today; a guessed label would be confidently wrong). Both are guarded by a regression test.
Tests: test/metrics/test_resource_attrs.py.
Which agent harness ran the work, in one closed vocabulary owned by
metrics/harness.py. The values are DERIVED from
agent_sdk/backends.py::ACP_BACKENDS_KNOWN in their config spelling
(backend_to_stored, so kiro-cli is kiro), plus claude_code (the
agent.provider == "claude_code" seam, which wins over the backend id),
unknown (nothing to resolve at the emit point) and other (an id outside the
known set, for example one an edition registered). A harness added to
ACP_BACKENDS_KNOWN joins the set with no edit here. Two readings:
| Reading | Where | Value | Answers |
|---|---|---|---|
kirocrew.harness |
RESOURCE attribute, every series | the install's configured default (global agent.acp_backend) |
how many installs chose which harness |
harness |
POINT attribute on kirocrew.session.started and kirocrew.turn.duration only |
the session's EFFECTIVE harness, per-session and per-agent overrides included | sessions and turns per harness |
Refresh. A resource is fixed for the life of a MeterProvider, so the
resource value is read in _build_recorder() from the same config load as the
telemetry section. watch_config() subscribes the rebuild to
agent.acp_backend and agent.provider as well as telemetry, so a change to
either drops the recorder and the next metric rebuilds with the new label (the
gateway; a short-lived process keeps the value it booted with). A process that
registers no watcher, such as the MCP gateway daemon, gets the same rebuild from
the consent recheck: the build records the raw agent.acp_backend /
agent.provider pair it read (_built_agent_key), and _consent_worker
rebuilds a live recorder when the pair on disk differs. Downstream that reads as
a new series set from the change on.
Series bound. The resource attribute adds no series inside a process: it
has one value per process, so it only partitions installs. The point attribute
multiplies kirocrew.session.started by at most the size of the set
(len(HARNESS_VALUES): the known backends plus three), and in practice by
the number of harnesses one install actually runs, usually 1 or 2 (the default plus a member override). On
kirocrew.turn.duration it is close to free: each harness serves its own model
ids (model_id_namespace), so model already mostly determines harness, and
the real growth is bounded by the (model, harness) pairs an install runs, not
by the product of both sets. It is deliberately NOT on
kirocrew.session.duration or the kirocrew.turn.tokens / credits /
cost_usd family yet.
Process-level metrics (gateway boot, HTTP requests, MCP pool, crew log health, inventory gauges) have no session, so they carry only the resource attribute.
Tests: test/metrics/test_harness_label.py.
The table above is the resource EVERY sink gets, and the roster below is the instrument set every sink gets. That is the right answer for the local JSONL sink and for an operator's own collector: both are the operator's own machine talking to the operator's own backend. It is the wrong answer for a destination that may receive only part of it -- a product destination reporting how Kiro Crew is used, whose exported set is a reviewed list rather than everything the process happens to record.
So OtlpDestination carries three optional NARROWING fields. Each can only
reduce or label what THAT destination receives; none of them can widen anything,
reach the local sink, or touch another destination. All three default to today's
behaviour, so an existing destination's payload is byte-identical.
| Field | Default | What it does |
|---|---|---|
allow: frozenset[str] | None |
None |
A metric whose name is not in the set is dropped before encoding. None sends every metric collected. Full instrument names, kirocrew. prefix included. |
point_attributes: Mapping[str, str] |
empty | Fixed pairs added to every exported data point of this destination. |
resource_allow: frozenset[str] | None |
None |
This destination's batches carry a Resource reduced to these keys. A key the shared resource does not set is simply absent. |
metrics/destination_filter.py enforces all three as an exporter WRAPPER that
_build_otlp_reader puts around that destination's own exporter. The descriptor
declares the narrowing; the core performs it. Four properties follow from where
the wrapper sits:
- Containment is structural. A wrapper only ever sees batches handed to its own exporter. The local sink and every other destination read their own, and a destination that narrows nothing is handed the unwrapped exporter object -- there is no wrapper in its path to differ by.
- Nothing shared is mutated. A collected batch arrives as SDK dataclasses
whose nested objects a sibling exporter may also hold, so every narrowed level
is rebuilt with
dataclasses.replacerather than assigned in place. That also keeps it correct as the SDK adds fields (NumberDataPoint.exemplarsarrived that way). - It fails closed. If the narrowing itself raises, the batch is DROPPED and the first occurrence logs one WARNING (the rest are debug, because a per-minute reader would otherwise write a line per interval forever). Forwarding the unnarrowed batch is the one outcome that would turn a bug here into an egress of data the destination was never allowed.
- Temporality stays a core decision. The wrapper mirrors the wrapped
exporter's
_preferred_temporalityand_preferred_aggregation, whichPeriodicExportingMetricReaderreads at construction -- answeringNonethere would silently put that destination back on the SDK's CUMULATIVE default while the local sink stayed on DELTA.
A point-attribute key is refused, not dropped. Keys are checked at
construction against destination_filter.ALLOWED_POINT_ATTRIBUTE_KEYS
(ClientType only today), and an unreviewed key raises. _build_otlp_reader
turns that into "this destination gets no reader", so the cost is one
destination's egress and local collection continues. Dropping the label instead
would publish a series the receiving side cannot route, with only a debug line to
say why. Keys are a reviewed non-secret set and are named in that log; attribute
VALUES are never logged, least of all beside an endpoint that may carry a
credential. On a key collision the destination's fixed value WINS over one
already on the point: the value is a constant fixed at construction and a
receiving side routes on it, so a call site must not be able to steer it.
PRODUCT_METRICS in metrics/product_metrics.py is the reviewed set a product
destination exports under. Names are spelled as literals there because
metrics.turns imports metrics.provider at module scope and the recorder reads
the set, so importing the declaring modules would close a cycle;
test/metrics/test_product_metrics.py resolves every name against its declaring
constant instead, and fails on a name with no instrument or on one that is a
histogram or an up/down counter. Counters and gauges only: a fixed-bucket
histogram's boundaries may not survive a receiving backend, so a distribution is
exported as a bucket-labelled counter.
Being a real counter is not sufficient. Every attribute value on an exported
series has to come from a closed enum, which the kind check cannot see, so a
metric carrying an open value is held out by name with its reason:
kirocrew.turn.tokens is wanted and its instrument exists, but its model value
is the configured model id, drawn from no set this repo controls -- on a non-Kiro
backend it can be a provider resource id, so its range is whatever that provider
names and whatever the person pointed the backend at. provider is configured
the same way. It waits for a bucketed or enumerated stand-in for those two. A negative test asserts each holdout stays
absent and that its name resolves to a real instrument, so the guard cannot pass
on a typo and putting a name back means deleting a line that states why it was
out.
Tests: test/metrics/test_destination_filter.py (one provider with the real local
sink plus narrowed and un-narrowed destinations attached at once: the allowed
metric only, the fixed attribute on every point, the reduced resource, both other
sinks untouched; then the nearby inputs -- an empty allowlist, an allowlist naming
an unknown metric, observable gauges, existing attributes preserved, a collision
on the fixed key, two destinations with different allowlists, a resource key the
build does not set, a refused key, and a raise inside the narrowing),
test/metrics/test_product_metrics.py (allowlist membership, instrument kind, the
open-attribute holdouts, and selftests that drive each rejection those checks rely
on).
Every OTLP destination built from config is an entry, parsed by
metrics/otlp_endpoints.py into one OtlpDestination each, in this order:
telemetry.otlp_endpoint, when non-empty: the first entry, namedtelemetry.otlp_endpoint, not validated and carrying no narrowing, so the descriptor is byte-identical to the one the default provider returned before the list existed.telemetry.otlp_endpoints: the operator's list. The loader keeps only the shape (a list of objects); each entry is checked when destinations resolve.
DefaultTelemetryProvider.otlp_destinations returns
otlp_endpoints.config_destinations(cfg). A build that ships its own collector
returns it from its provider's otlp_destinations, after
config_destinations(cfg) so the operator's entries stay. That is the whole lock
mechanism: a destination code returns is not config, so no config value removes
or disables it, and no new seam is needed. otlp_endpoints.label_sources marks
each resolved destination otlp_endpoint, otlp_endpoints (equal to the
descriptor an operator entry produces, each entry matched once) or provider
(anything else, reported locked). Matching is by the whole descriptor, not the
name, so an operator entry that reuses a build collector's name cannot relabel it.
Entry keys: name, endpoint (required, http(s):// with a host, at most
MAX_ENDPOINT_LEN characters), enabled (a boolean; any other value refuses the
entry, so a mistyped opt-out never exports), headers_env (at most
MAX_HEADERS pairs), temporality ("" / cumulative / delta), allow
(a list of names, or a preset name from otlp_endpoints.ALLOW_PRESETS:
PRODUCT_METRICS), point_attributes, resource_allow. A malformed entry,
including a shape no check anticipated, is skipped with one WARNING per process
naming the entry and the field, never a value; it never stops collection. The
warn-once memory keeps a 16-byte digest per warning, at most MAX_WARNED (256),
and clears when full, so config revisions that are later discarded do not
accumulate in a long-lived gateway. An
unknown allow preset refuses the entry: an allowlist must not fail open into
"send everything". At most MAX_ENTRIES (16) are read. These bounds are
arbitrary safety bounds on threads and retained memory, not tuned values.
There is no key for a literal header value: an entry carrying headers is
refused with a warning that points at headers_env, so no credential has to be
written into config.json, where the dashboard and the agent's settings
diagnosis can read the config.
Three fields on OtlpDestination carry what an entry needs at reader build:
headers_env-- header name -> environment variable name, read by_build_otlp_readerwhen it builds the exporter. An unset or blank variable refuses that destination (no reader, one warning naming the variable), so a secret never sits on the descriptor or inconfig.json.temporality--""keepstemporality.otlp_preference();delta(otel_delta_preset) andcumulative(cumulative_preset) pin every instrument kind for that reader only, so the OTel variable cannot move it.process_env_credentials--True(the default, and the legacyotlp_endpointentry) leaves the exporter's fallbacks in place: with no headers of its own it sendsOTEL_EXPORTER_OTLP_METRICS_HEADERS/OTEL_EXPORTER_OTLP_HEADERS, and with nosessionit may load the OTel credential-provider session. Everyotlp_endpointsentry isFalse:_build_otlp_readerpasses a header map that stays truthy when empty (_OwnHeadersOnly, because the exporter readsheaders or <env>) and a plainrequests.Session, so the entry sends only itsheaders_envheaders. Those variables were set for one collector; without this, adding a second entry with no headers would send the first collector's credential to the second host.
Gates are unchanged and apply to every entry because every entry is a
destination: the consent gate in _build_recorder runs before any is resolved,
otlp_governance.governed_destinations applies the otlp_hosts check, and
gate_exporter re-checks each export. An enterprise build blanks both local keys
in _edition_locked_view (CONFIG_PATH_LOCKS names both).
The config view masks the whole telemetry.otlp_endpoints value
(core_config_view._masked_config_dict masks every string under a container
marked sensitive), because entries can carry endpoint userinfo. The agent's
settings diagnosis (change_cards_settings.non_default_settings, behind
diagnose_settings) reports any path the schema marks sensitive, or that sits
inside a sensitive field, only as {set: bool} (_schema_sensitive, through
config.validation._is_sensitive_path), whatever its key names look like.
Disclosure: otlp_egress_listing(cfg) returns the labelled set before the policy
check, and otlp_egress_active is bool() of it. A configured entry is listed
even when its reader would be refused for a reason that can lift with no change by
the person (an unset headers_env variable, a host the policy refuses today).
GET /api/telemetry/collection returns it as otlp_destinations: [{name, source, locked}] -- names only, never an endpoint or host -- and Settings -> Privacy lists
those names under its egress note.
Tests: test/metrics/test_otlp_endpoint_list.py (the legacy string unchanged,
parsing, refusals and bounds, a build-shipped destination next to operator
entries, the gates on it, reader headers and temporality, the disclosure listing,
and loopback wire tests: two destinations where the second is narrowed and
authenticated from the environment, and the OTel header variable reaching the
legacy entry but not a list entry), test/test_change_cards_settings.py
(the diagnosis withholds schema-sensitive fields), test/test_config_patch_otlp_endpoints.py
(a list entry and a build-shipped destination each refuse the enable; the list is
not PATCH-writable; the config view masks it).
| Metric | Type | Attrs | Site |
|---|---|---|---|
kirocrew.acp.child_permission.denied |
counter | surface (runtime / session_handle / subagent), reason (closed denial reason) |
acp/runtime_host_requests.py::_audit_denied_off_loop, acp/session_handle_permissions.py::_audit_handle_reject, and the subagent's permission narrator (subagent_manager/run.py::_SubagentNarrator); one point per backend-child permission request rejected before it can become a silent hang. |
kirocrew.acp.child_permission.routed |
counter | surface (runtime) |
acp/runtime_reader_loop.py when a child permission request reaches the mode-parity pipeline. Together with .denied, this accounts for handled child permission requests. |
kirocrew.acp.dropped_frames |
counter | method_class (permission / update / other) |
acp/runtime_frame_accounting.py::_note_dropped_frame; counts unroutable ACP frames without exposing backend method names. A non-zero permission series is the hang-regression signal. |
kirocrew.acp.skill_view.fallback |
counter | outcome (loaded_after_retry / refused_unloaded / refused_unprepared / refused_superseded) |
acp/runtime_mode_activation.py::_activate_mode_bracketed; one point per session/set_mode that did not land on the freshly prepared skill-view alias first time: loaded after a forced reload, or the session start refused because the host never loaded it, no view could be prepared, or a concurrent start adopted a newer view of the agent (or began one that did not complete) while this one's set_mode was pending. More than a trickle of refused_unloaded or refused_unprepared means the host is not loading published aliases; refused_superseded means session starts overlapped an edit to the same agent's spec, or a newer preparation that did not complete. Starting again uses the new view when it offers the agent; when the new view refuses the agent, starts keep failing until the spec is fixed. |
kirocrew.turn.timeout.cause |
counter | path (provider_timeout / dashboard_ceiling), awaiting_permission (bool), children_announced (bool) |
_on_complete in dashboard/chat_turn/turn_stats.py and dashboard/turn_dispatch.py; attributes the two timeout exits to a pending permission prompt and announced child work. |
kirocrew.session.idle_expired |
counter | turn_active (bool), orphaned (bool) |
session_cleanup.py; emitted after an idle/orphan sweep successfully resets a session. |
kirocrew.session.startup.duration |
histogram (ms) | outcome (ready / auth_required / error), spawned (bool), backend (kiro) + phase (total / spawn_init / session_new / session_load / set_model), channel (conversation source), resumed (bool) on the kiro path |
Two sites. claude: acp/client_start.py::AcpClient.ensure_ready() — times cold-start (spawn + session init) and emits in a finally so every exit path is measured, with no phase attr. kiro (default): providers/acp_runtime_start.py::_emit_kiro_startup_metric — one phase=total point PLUS one point per internal phase; spawned is unconditionally True because _start_kiro_runtime_impl always spawns a fresh runtime (the warm fast-path returns before reaching either site and is NOT measured). outcome defaults to "error" so an unexpected exception is never mislabeled "ready". Consumers MUST treat only the end-to-end point (phase absent or total) as a startup — the phase points are components of one startup. channel comes from messaging.link::telemetry_channel_of, a closed label set (an unrecognised key classifies as other, never the key itself) answering WHICH surface paid the cost; resumed separates the session/load path from session/new. session_load is recorded only when a resume was attempted, and session_new only when create_session actually ran, so a resumed startup never reports a near-zero session_new. |
kirocrew.chat.first_token.duration |
histogram (ms) | channel, first_turn (bool), resumed (bool) |
dashboard/chat_turn/turn_stats.py::_emit_ttft_metric; user-message-to-first-visible-token latency for top-level user prompts. Own bucket family _STARTUP_BUCKETS_MS. |
kirocrew.session.pool.decision |
counter | outcome (hit / miss_empty / bypass_member_capabilities / bypass_resume / bypass_stateless / bypass_member / bypass_app_launch / bypass_restricted_context / bypass_member_context / bypass_cwd / bypass_env / bypass_shared_scratch / bypass_backend_override / bypass_effort / bypass_model_at_launch / disabled / other), channel |
session_parts/runtimes.py::SessionManager._record_pool_decision, one point per get_or_create warm-pool decision. Exactly one reason is reported per decision — the disqualifiers form a disjunction, so branch order picks the reported reason, not the outcome. Deliberately a counter rather than an attribute on the startup histogram: "was the pool used" and "how long did startup take" are separate questions, and crossing them would multiply every phase series. bypass_resume quantifies how often a resume_sid disqualifies a session from the pool, and bypass_shared_scratch how often a dedicated subagent cold-starts to mount its parent's scratch. bypass_model_at_launch counts model-pinned chats on a harness that takes its model only at launch (ACP_BACKENDS_MODEL_AT_LAUNCH): the pooled process cannot switch, so it is discarded and the chat cold-starts on its pin. Values are pinned by session.POOL_DECISIONS, which lists every reason get_or_create assigns (a test reads the assignments), so no reason folds into other. |
kirocrew.session.resume.outcome |
counter | outcome (loaded / fallback_replay / no_session_file), channel |
providers/acp_runtime_start.py::_emit_kiro_startup_metric, emitted only when a resume was attempted. Distinguishes a lossless native resume from the degraded fallback (fresh session/new plus history replay on the Kiro Crew side, taken when session/load exhausts _RESUME_MAX_ATTEMPTS against a stale lock) and from a resume skipped because the session file was gone. |
kirocrew.session.duration |
histogram (ms) | end_reason (reset / removed / unclaimed / destroyed / discarded / shutdown / retired / recycled / evicted / crashed), session_source (via messaging.link::telemetry_channel_of) |
metrics/sessions.py::record_session_ended (or record_sessions_ended for a path that drains many keys at once) for every clean end; metrics/sessions.py::backfill_crashed_sessions for crashed. Why the reason is passed in, never derived. There is no single teardown funnel: SessionLifecycleService (its bodies in session_lifecycle_parts/teardown.py and session_lifecycle_parts/fleet.py) holds six end paths that each pop the registry themselves (reset / remove / remove_if_unclaimed / destroy / discard_conversation / close_all), and four more pop it outside all of those — retire_kiro_identity_sessions reports retired for the sessions it tears down on an identity change, reload_provider_factory reports retired too for the registry it clears on a provider switch (both are "this process is no longer the right host for these sessions", which is why they share a label rather than minting a second one), session_compaction.py::CompactionCoordinator._recycle_held reports recycled (context overflow replacing the provider in place), session_background.py::recycle_heartbeat also reports recycled (the background runtime replacing its own session at cycle end), and the two dead-provider evictions in session_allocation.py report evicted. So each site states its own end_reason, the same contract metrics/turns.py holds its callers to. Every path calls this while it still holds the session registry lock, in the same event-loop tick as its _sessions.pop: the pop and the histogram record are both in-memory and both happen before the call's single suspension point, and a later call lets a racing cold start register a SUCCESSOR under the reused key whose record the predecessor's teardown then consumes. reset is a PATH label rather than a cause — the idle sweep and a slot reset both reach teardown through it, and the finer causes are already counted by kirocrew.session.idle_expired and kirocrew.watchdog.recovery.outcome; the other four are separate precisely because they never take the reset route. EVERY registry removal must record an end, and that is a correctness requirement rather than a completeness one. An unrecorded removal does not merely lose a sample: the crumb below survives it, so the next boot reports the session as crashed — a failure that never happened, injected into the one population this histogram exists to measure. Four such paths shipped unrecorded and were caught in review, which is why the gate for this is test_every_registry_removal_records_an_end, a fail-closed AST walk over every module that mutates the registry (recognising pop, del and clear) rather than a hand-kept list of method names that a new path silently escapes. The crumb is what makes a crash countable. A start time held only in memory dies with the process, which is exactly the population most worth measuring, because a crash runs no teardown path at all. So record_session_started drops one small JSON file under <data home>/metrics/open-sessions/, named by SHA-256 digest of the session key PLUS the writing pid and the start time, with the key stored inside (keys carry channel ids and slot names and are not filesystem-safe; session_map.json already holds every key in the same data home, so this is not new exposure). The name identifies the writer and the generation, not just the key, and that is what makes every unlink here safe. A session key is not unique across processes — BACKGROUND_KEY is a fixed constant, so a kirocrew run and the gateway hold the same key at the same time with a live session each — so while the name was the digest alone they shared one file, and whichever ended first unlinked the other's record; the locks in this module are THREAD locks and never spanned processes, so nothing here could have prevented it. With the writer and the generation in the name, a crumb can only ever be removed by the session that wrote it. Three consequences follow. A clean end unlinks the ONE generation it can name, and an end with no live table entry unlinks NOTHING — the files sharing that digest belong to other processes or earlier runs, and removing them is precisely the bug the name now prevents. A start that displaces a live generation of the same key, and an eviction under _MAX_LIVE_SESSIONS, must each reap the generation they orphan, because once the table forgets a generation nothing can ever name its file again; both ride the same awaited worker hop as the start's own write, so those unlinks stay off the event loop without opening a second cancellation window. And the backfill legitimately finds SEVERAL files for one key, each a distinct session judged on its own recorded owner. Whatever survives to the next boot is a session that never ended cleanly, which backfill_crashed_sessions(started_before) emits as crashed and unlinks; it is also the only reader of these files, so an unparseable crumb is reaped there rather than by an end unlinking blind. A crumb that exists but cannot be READ (an OSError other than FileNotFoundError) is kept with one DEBUG line naming the error class: its owner cannot be judged, so it waits for a later boot (test_session_crumb_unreadable.py). started_before is the calling process's own start time, so a crumb at or after it is left alone. The crumb also records WHO owns it — the writing process's pid plus a per-process start identifier — and the backfill skips any crumb whose owner is still running. The cutoff alone was not enough: it only ever protected THIS process's own crumbs, and cli.py and eval/runner.py each build their own session manager against the same data home, so a kirocrew run session started before the gateway booted was read as a crash. That both invented a sample and deleted the live crumb, losing the real crash that session might later suffer. The start identifier is what makes pid reuse safe: a recycled pid belongs to a different process and must not look like the owner, and that includes reuse of the CURRENT process's own pid — a container hands the gateway PID 1 on every restart, so a crashed predecessor's crumb arrives carrying this process's pid, and skipping it on the strength of the pid alone meant its crash was never emitted and its file never cleaned on any later boot either. Liveness comes from platform_compat.pid_exists; the start identifier only refines it. get_process_start_id returns None for any process the caller may not introspect, and its own contract says a None must NOT be read as a mismatch — using it as the liveness test would judge dead every owner it cannot introspect and reap live sibling sessions. The ownership check fails CLOSED — a live pid whose identity cannot be read counts as running, so an ambiguous crumb waits for a later boot rather than becoming a fabricated crash; losing a real crash sample costs one data point, inventing one corrupts the population this instrument reports. A crumb with no ownership recorded (written before this field existed) falls back to the cutoff alone. A registration that never became a session is discarded, not ended: discard_session_start pops the table entry and unlinks that generation's crumb WITHOUT emitting, for the cancellation window between inserting a session into the registry and finishing its start record — a session that never lived has no lifetime to report, and it must not be left as a crumb the next boot calls a crash. It is deliberately not an eleventh end_reason, because those describe how a live session ENDED. Exactly-once accounting. A crumb is consumed at most once, so the teardown paths — which are NOT mutually exclusive, the idle sweep calls reset — cannot double-count between them, and a backfilled session cannot be counted again on the boot after. The clean path deliberately reads neither the crumb nor the popped entry's created_at: consuming a DIFFERENT record than the crash path would let a clean end's leftover crumb be re-emitted as crashed, and created_at is reset when a provider is recycled in place, so it measures provider age rather than registry residency. A crashed session's END time is its transcript mtime (<data home>/sessions/<stem>.jsonl, appended as the conversation runs, so it is the last moment the session was observably alive and the honest post-hoc maximum). A session with no transcript — a subagent leaves only a replay log — yields no end time and is unlinked WITHOUT an emit rather than recorded as a 0ms lifetime: absence reads as "no data", a recorded zero renders as a real zero. Two locks, split by what they may be held across. _live_lock guards the in-memory start table ONLY and is never held across I/O, because every end takes it on the event loop; _crumb_io_lock orders the two worker-thread paths that touch crumb FILES — the writer, and the backfill's read-decide-unlink — and nothing on the loop may take it. Coupling them was a real stall: the backfill held one lock across a crumb read plus an unlink on a worker while a teardown waited for that same lock on the loop. The end path needs no crumb lock at all, because ownership already partitions the work — the backfill skips every crumb whose owner is live, so the crumbs a session ends in THIS process and the crumbs the backfill reaps are disjoint sets. It is also worth stating that these are THREAD locks and so never protected anything against a sibling PROCESS; that is exactly what the ownership field is for. The write itself runs on a worker thread (asyncio.to_thread) and is AWAITED while the caller still holds the session registry lock, which is what satisfies two constraints that look opposed: the syscalls never run on the event loop, where a slow or network-homed data home would stall every gateway task behind one session insertion, and nothing can interleave, because starts, ends and successors all serialise on that registry lock. Three earlier shapes each satisfied one and broke the other — a fire-and-forget pool write let a late writer delete a successor's crumb (an ended-key set, then a generation token compared before and after the write, both failed, because every check still acted on a path the successor SHARED), and writing inline fixed the ordering by putting filesystem I/O on the loop. Naming the file after its writer and generation is what finally removed that shared path, so the post-write check now removes only this writer's own file. The END path's unlink runs on a worker thread too, and is AWAITED there for the same reason the write is. Inline on the event loop it would be one syscall against the data home on the paths a person is waiting on -- closing a tab, resetting a slot, shutting the gateway down -- so on a slow or network-backed home that single unlink would park every gateway task behind whichever session happened to be closing, felt as the whole gateway freezing rather than as a metrics problem. Queued fire-and-forget it would be worse than slow: shutdown_maintenance_executor drains with cancel_futures=True, so at close_all -- when teardown is heaviest -- the unlink is dropped outright and a cleanly ended session's crumb survives to be back-filled as crashed, inflating the exact population this instrument measures. Awaiting a worker hop is the only shape that is both off the loop and never dropped. The same-tick rule the inline version protected is unaffected, on three counts: the registry pop and the histogram sample both happen BEFORE the function's only suspension point, so a teardown path still accounts for its session immediately after its _sessions.pop; every caller awaits while it still holds the session registry lock, which is the lock a racing cold start must take to register and map a SUCCESSOR; and the hop is awaited so nothing is fired blind. Submitted is not started, so the hop is a BARE executor future awaited through asyncio.shield. On a saturated executor the work item is still queued, and cancelling the await reaches the underlying concurrent.futures.Future through futures._chain_future's _call_check_cancel, which cancels a work item that has not begun -- the same dropped-unlink failure that ruled out the maintenance pool, reached by a different queue. Shielding leaves the inner future alone, so nothing cancels the queued work and it runs when a thread frees up. Two details make abandoning it safe on the cancellation path, and both are why this is not the fire-and-forget shape the start path forbids. It is a Future and NOT a Task, so runners._cancel_all_tasks at loop teardown -- which iterates all_tasks() -- cannot reach it; shielding a to_thread COROUTINE would be wrong, because shield wraps a coroutine in a Task that teardown cancels, dropping a still-queued item all over again. And a worker finishing after the loop closed cannot raise "Event loop is closed", because _chain_future._call_set_state returns early when dest_loop.is_closed() instead of calling call_soon_threadsafe; the historical Windows failure was a start-path write never awaited at all, whereas here the normal path awaits to completion and only a cancellation abandons the future. Unlinking inline in the cancellation handler was tried and rejected in review: it fixed the drop but put a filesystem syscall back on the event loop, which is the thing this issue is about. The start path still must not use this shape, because its write has to be ORDERED under the caller's registry lock and so cannot be allowed to land late; an end's unlink can, because the crumb name encodes the writer and the generation, so a late unlink can only ever reach the file its own session wrote. If the executor is already shut down the hop cannot be scheduled at all, and the crumb is left to the boot backfill rather than unlinked on the loop -- reachable only once the loop itself is tearing down, and the backfill's ownership check means it is claimed only after this process is gone. A cancellation at that hop is ABSORBED by all three end paths, and that is what makes the new suspension point safe to give a teardown. The registry pop is the caller's point of no return: past it only the caller's remaining cleanup can finish the job, and destroy deletes the session-map entry in a finally BELOW its end call while discard_conversation and reset clear the resume SID on the line after the lock. Raising would exit those callers before that cleanup, leaving a permanently destroyed session's mapping behind for the next boot to resume -- a data-loss window the synchronous version did not have, because it had no suspension point there at all. A cancellation cannot undo a teardown, only leave it half done, so the correct answer is to let the teardown finish. That cancellation is genuinely DISCARDED rather than deferred -- a swallowed CancelledError is not re-delivered by asyncio, so the caller runs to completion unless something cancels it again -- which is the intended outcome rather than a side effect, because finishing a teardown that has already popped is exactly what the callers' finally blocks exist for. The consequence worth knowing is that a caller wanting a hard cap on teardown must impose it outside the pop, since a cancellation delivered after the pop is not honoured; no caller does today (every close_all await site is bare). The two guarantees sit in two places deliberately: whether a CALLER may be aborted is knowable only where its post-pop cleanup is known, so it lives in the end paths, while whether the UNLINK survives a cancelled hop is a property of the hop, so it lives in the helper. Two more consequences are worth stating. The sample is emitted BEFORE the hop, the opposite of kirocrew.session.started's counter, because the risks are opposite: a cancelled start is rolled back and never existed, while an end cannot be rolled back (the pop has already consumed the only record of that generation), so emitting first is what keeps the sample under cancellation and what measures the lifetime at the pop rather than a worker round-trip later. And the four paths that pop MANY keys at once -- close_all, drain_all_providers, retire_kiro_identity_sessions and reload_provider_factory -- call record_sessions_ended instead, which pops the whole set under one _live_lock hold and unlinks it in ONE hop: a hop per key would put a cancellation point BETWEEN two keys, leaving every key after it popped but unrecorded, and on close_all (a path cancellation reaches by design) that would turn one orderly shutdown into a burst of fabricated crashes -- strictly worse than the on-loop unlink it replaced. discard_session_start is a coroutine for the same reason, and absorbs its cancellation for a second one too: its callers re-raise the failure that started the rollback on the very next line, so letting a cancellation escape would displace that failure with one of its own making. The crumb is written only under telemetry consent, checked fail-closed: it exists solely to feed this instrument, which is a no-op without consent, so writing one on an unopted host would persist state nothing can read against a documented default of collecting nothing. Bounded throughout: _MAX_BACKFILL_EMITS (2000) caps a runaway crumb directory while still unlinking everything it walks, and _MAX_LIVE_SESSIONS (4096) drops the oldest half of the table, unlinking each dropped generation's crumb as it goes — both cost samples, never memory. Own bucket family _SESSION_BUCKETS_MS. |
kirocrew.session.started |
counter | session_source (via messaging.link::telemetry_channel_of), harness (the session's EFFECTIVE harness, closed set below) |
metrics/sessions.py::record_session_started, one increment per session admitted to the registry, called from the sites that insert into _sessions (session_allocation.py and session_allocation_parts/get_or_create.py cold-start and resume paths, session_background.py, and the compaction path that keeps a session alive) with the provider the entry was built around. harness is read off that provider by metrics/harness.py::harness_of (through providers/acp.py::_provider_backend, the shape check the session-map label uses), so a per-session or per-agent acp_backend override counts as its own harness; see "The harness label" below. The denominator the kirocrew.session.duration population is a subset of: not every registry removal records an end, so started counts admissions while duration counts the endings actually observed, and the gap between the two is itself readable. session_source is the SAME telemetry_channel_of derivation the duration histogram and metrics/turns.py use, so all three group by one bounded label set (an unrecognised key classifies as other, never the key itself). The increment and the _live_starts write are in-memory and therefore safe under the session registry lock; only the crumb touches disk, off-thread. |
kirocrew.turn.duration |
histogram (ms) | outcome (ok / timeout / cancelled / tool_stall / stale_recover / stall_exhausted / error / unclassified), session_source (via messaging.link::telemetry_channel_of), model + provider (omitted when the caller cannot resolve them), harness (the session's EFFECTIVE harness; always present, unknown when unresolved) |
harness is resolved at each owner from the live provider that ran the turn: owner (1) reads it off model_source, owner (2) off the turn's client, both through metrics/harness.py::harness_of. It is not on the row store. Two emit owners, at most one sample per turn. Only a completed or timed-out turn is sampled: a dashboard turn that ends without EVENT_COMPLETE (Stop, the turn ceiling, process death, auth loss, a busy prompt, a session not found, a terminal or transient error) still writes its usage row with a synthetic stop_reason through chat_runner._persist_abnormal_turn_usage, which passes emit_metric=False and emits no kirocrew.turn.* sample. On the complete path the sample is taken before the row's shielded persist, so a cancellation landing during the persist does not lose it. (1) dashboard/handlers/usage.py::persist_token_record_async — the call EVERY dispatch surface already makes once per turn at EVENT_COMPLETE, which is what makes cron, the heartbeat, memory consolidation, subagents, task-runner steps, workflow stages and the messaging channels present in this metric at all, so the page does not report the interactive median as the system's health. It reuses the built row's own duration_ms, so the row store and the histogram cannot disagree about one turn. (2) dashboard/chat_turn/turn_stats.py::_emit_turn_metric, called from _on_complete in the same file, which passes emit_metric=False to its persist call and emits itself — that persist sits behind usage_has_billing, so a turn that timed out having billed nothing writes no row and would otherwise lose its sample, and only this surface holds the EFFECTIVE session key (a linked channel conversation is attributed to its channel, not to the slot.key the row is filed under) and the spent-recovery-budget knowledge that labels stall_exhausted. metrics/turns.py::turn_outcome owns the mapping (""/None/end_turn/stop/completed → ok; the two watchdog stop reasons map to their own outcomes — checked BEFORE the timeout substring — so a recovered stall is never counted as a generic fault and the stall population stays visible; a stall arriving with its 3-attempt recovery budget already spent, or on a NESTED turn (_prompt_depth > 0, which the recovery branches never re-queue — it dies with "please retry"), labels stall_exhausted, which IS a terminal fault to the aggregator, so the recovered-stall exclusion cannot hide a session that dies needing user action). cancelled is a user pressing Stop, matched by EXACT equality against STOP_REASON_CANCELLED so the watchdog's "error: cancel unacked" still reaches the error branch; folding it into error would put every deliberate cancel into fault_rate's numerator, so it is excluded from the numerator while REMAINING in the denominator (the turn ran). unclassified marks a turn whose surface passed a bare TurnUsage and so had no stop reason to give; it is in neither the numerator nor the denominator of fault_rate (in the numerator it would invent a fault for every clean background turn; in the denominator alone it would dilute the rate towards zero as background traffic grows), and its count is readable as outcome["unclassified"]. One histogram powers turn latency p50/p90 AND fault rate. The value is duration_ms or elapsed_ms: the acp provider always reports TurnUsage.duration_ms == 0 (only claude_code fills it), so the caller must pass the locally measured wall clock as elapsed_ms or nothing is ever emitted. A still-zero value skips the emit deliberately — absence renders as "no data", whereas a recorded 0 would render as a plausible 0ms p50. What it measures: the wall clock starts at turn start, so a turn parked on an interactive tool-approval prompt counts operator thinking time. No finer-grained source exists on the acp path, so this is "turn wall-clock", not pure model latency — a high p90 can mean slow approvals rather than a slow model. |
kirocrew.turn.tokens |
counter (token) | direction (input / output), model + provider |
Same two owners and the same exactly-once split as kirocrew.turn.duration, emitted from metrics/turns.py::emit_turn_usage off the same built row, so volume and latency describe one population. ONE instrument with a direction attribute rather than an .input/.output pair: the two series then carry an identical attribute set by construction (a model added to one cannot be forgotten on the other), the dashboard's counter path already reports every attribute combination under by_attr, and direction is a two-value enum so the cardinality cost is exactly 2x. Positive-only. Each backend fills only the dimensions it bills in; the kiro/acp backend reports zero for every token field, so emitting the zeros would publish a full series of them per turn and a recorded 0 reads as a measured zero rather than as "this backend does not report here". A negative value is also dropped — a monotonic counter cannot take it back. |
kirocrew.turn.credits |
histogram (credit) | model + provider |
The turn's billed amount on a credit-billing backend (kiro/acp). Emitted only when non-zero, which is what makes it mutually exclusive with cost_usd on a given host (acp/types.py: "Consumers read whichever is non-zero"). Two instruments rather than one with a currency attribute because credits and dollars are not the same quantity: percentiles over the union have no unit. Own bucket family _CREDIT_BUCKETS (0.025–2500), calibrated against 17,240 real per-turn rows (min 0.03, p10 0.076, p50 6.8, p90 53, p99 155, max 658). NOT in _HISTOGRAM_BUCKETS_MS, and claimed by name in _aggregate so it never reaches the *_ms surface — see the bucket-boundaries section. |
kirocrew.turn.cost_usd |
histogram (usd) | model + provider |
The same amount on a dollar-billing backend (claude_code / bedrock), same non-zero gate, same owners. Own bucket family _USD_BUCKETS (0.001–100). This is the one bucket array with no local calibration — a credit-billing host reports zero for every cost_usd row by construction — so it is sized from published per-token pricing against the observed token range and is worth re-checking once a dollar-billing host reports. Reported through _amount_stats, which keeps six decimals so a sub-cent turn does not round to 0.0. |
kirocrew.tool.call.duration |
histogram (ms) | tool_kind (read / fetch / search / edit / write / create / delete / move / execute / think / switch_mode / client_built_in / mcp / other), outcome (completed / failed / cancelled) |
Round-trip latency of one tool call, keyed by kind and never by tool name. The tool NAME is deliberately not an attribute: MCP names are unbounded (any installed server contributes its own) and the recorder caches one instrument per distinct attribute value with no eviction, so a name-valued label is a cardinality bomb. tool_kind is the ACP kind normalised by metrics/tool_calls.py::classify_tool_kind and pinned to the closed TOOL_KINDS set; the kind field arrives verbatim from the agent, so an unrecognised or agent-authored kind — including an empty one — folds to other, which is what stops a tool name minting a series. An MCP-served call is labelled mcp whatever kind it reported, decided by the trusted _meta.kiro.mcpServerName rather than the reported kind: an MCP server's kind vocabulary is its own, and "this call left the process over MCP" is the more useful fact than a builtin's finer verb. outcome carries exactly the three terminal ACP statuses — a non-terminal update (pending / in_progress / absent) records nothing and leaves the clock running, so no fourth value is reachable. A <= 0 span skips the emit, the same rule kirocrew.turn.duration follows. The clock is time.perf_counter, NOT time.monotonic: on Windows the latter advances in ~15.6ms ticks, so any call completing inside one tick measured exactly 0.0 and was dropped by that guard — which silently hid every sub-tick tool call on the platform, i.e. most cached reads. perf_counter is the highest-resolution monotonic clock available everywhere, so the guard keeps meaning "unmeasurable" rather than "fast". Site: two parser layers, one registry, one sample per call. There is no single choke point, because the two backends parse through different code: the shared builders in acp/_dispatch_parts/session_update_parser.py and acp/_dispatch_parts/tool_result_events.py (_build_tool_call_event starts the clock once the trusted MCP identity is resolved; _build_tool_result_event stamps the terminal status BEFORE its output parse can return None, so an output-less completion is still measured), where the scope arrives as cache_scope forwarded by the session handle (acp/session_handle_updates.py, acp/session_handle_permissions.py, acp/session_handle_dispatch.py) as the frame's sessionId; and acp/client_tool_events.py::_extract_tool_call_update with its result-extracting sibling, which shape the same frames inline and scope on the client's own _session_id. Every other surface consumes the emitted AcpEvent stream downstream of these two, so instrumenting both parsers covers all of them. The registry is keyed `scope |
kirocrew.crew.session_control.calls |
counter | op (create / send / fork / stop / close / continue / broadcast / status / read_message / summary / retry / reload / rename / set_model / end_wait / adopt / release / generate_title / other), caller (the agent_kind enum: default / crewmate / conductor / pipeline_conductor / ledger_conductor / security_conductor / worker / team_lead / dashboard_author / dashboard_manager / knowledge / research / heartbeat / lite / guest / other), outcome (ok / error) |
Which session-control operations agents reach for, and how often they fail. Session control is the set of MCP tools an agent uses to act on ANOTHER live conversation — open one, send into it, stop its turn, read its tail, adopt it, close it. People use the dashboard for all of that, so every caller here is an agent. kirocrew.tool.call.duration cannot answer the question: it labels a call by ACP tool_kind and labels an MCP-served call mcp whatever kind it claimed, which is the right trade there and the wrong grain here — it folds all eighteen verbs, every other MCP server's tools and their outcomes into one series. op is the TOOL's verb, not the route's path segment, so /api/session-control/read is read_message (the session_read_message tool) and /set-model is set_model: the question is which tool an agent reached for, and the tool verb is the stable name of the two. The route table is Kiro Crew's own, so the list is closed by construction rather than by trust, but classify_op still folds an unlisted value to other so a route added without a row cannot mint a series — revive and history are registered routes the list deliberately does not name and both land in other today. caller is derived from agent_files.OWNED_KIRO_AGENT_FILES (file stem minus kirocrew-, - rewritten to _) rather than tabulated, so a new owned agent file joins the enum with no table to update and cannot fall silently into other; the bare kirocrew agent maps to default, and a mate's pinned DM session is identified by its SLOT KEY's shape (members.is_member_session_key) because such a slot's agent field carries the mate's own crew name and could only answer other. An agent name outside the owned set — a third-party agent, a user-authored template — is other: the raw name is free-form with no cardinality bound, and folding it before the recorder ever sees it is what keeps the series bounded. A BLANK agent on a resolved slot is default, not other: _ChatSlot takes agent: str = "" and an ordinary dashboard chat is minted without one, so the blank is what the largest caller population carries, and crew_log/emit_session.py already reads it that way (agent or _DEFAULT_AGENT). A caller whose slot could not be resolved AT ALL is other — same blank agent, different answer, and classify_caller's slot_found is where that line is drawn. outcome is decided by the status the route answered with (below 400 is ok) or error for a handler that raised. Ceiling 19 × 16 × 2 = 608 series, and a backend that should not carry all of them can drop outcome from its dimensions. Why not kirocrew.gateway.request.duration. That histogram already covers these twenty paths, labelled method × route_template × status_class, so per-route volume and per-route failures are readable there today and this counter is not the place to go for them. What no instrument carries is WHO called: it has no caller-identity label, and adding one would multiply every route in the gateway's bounded template set by sixteen rather than only these twenty. caller is the axis that makes this counter exist; op and outcome are the dimensions it is crossed with, because "which verbs does a conductor drive, and which of them fail for it" is one question rather than three. Site: one wrapper per route, outside the auth gate. metrics/session_control_calls.py holds the enums and the emit; dashboard/handlers/session_control.py::_counted wraps each of the twenty registered handlers and is the ONLY emit site, so the deferred route binder calling a handler cannot count a call twice. The wrapper sits OUTSIDE the handler's own _require_internal / member gate deliberately: a caller refused there made a session-control call that failed, and a wrapper placed under the gate would not see it. A handler that raises is counted error and its exception re-raised untouched, so this only ever ADDS the count — the response, the status and the SEL audit line stay the handler's own. The caller classification reads LIVE STATE ONLY (the in-memory slot registry, via get_slot and then caller_access.caller_slot_key's walk), so no disk read lands on a route an agent is waiting on, and an unresolvable caller is other rather than a failure. Best-effort throughout, behind telemetry.enabled: a disabled process holds a recorder with no meter and the emit returns before touching an instrument. |
kirocrew.watchdog.action |
counter | action (deferral / probe / cancel), verdict (working / dead / unknown / stuck_input), evidence_class (established_flat / mcp_flat / remote_flat / shell / shell_absent / platform_limited / wait / degraded), window (narrowed / extended / working_cap / standard), agent_override (bool) |
acp/session_handle_watchdog.py::AcpSessionHandle._emit_watchdog_metric, one point per watchdog DECISION in _dispatch_events: deferral from _log_working_deferral (rides its 10-min rate limit, so an hours-long WORKING build contributes a bounded handful of points, not one per tick), probe at the stale-probe send, cancel before _end_stalled_tool. evidence_class is _watchdog_evidence_class — a prefix/shape bucket of the free-form oracle evidence (pids/deltas/commands never emitted). window encodes the effective window selection: narrowed = a tool-branch evidence TAG reduced the suspect window below the build-scale default (90 min) — established_flat to the model-silent budget (minutes), shell_absent to the ordinary silence budget (stale_window_secs, 600s) because the shell command has no process to its name, platform_limited to the same stale_window_secs only when the tool layer classified the shell command prompt-shaped, remote_flat to remote_flat_probe_secs (off by default; opt-in) for an MCP tool blocked on its own remote call; extended = model-wait-branch established_flat extended the stale window from 600s (stale_window_secs) to 1800s (model_silent_probe_secs) for a non-streamed server-side think; working_cap = an opaque MCP tool's mcp subtree active WORKING reading ran past tool_stall_hard_cap_secs and got the tool-stall cancel; standard = ordinary window in all other cases. agent_override is the per-agent watchdog-override BOOLEAN from the WatchdogSettings snapshot — deliberately NOT the agent name (free-form ⇒ cardinality bomb; per-agent joins happen via the row store's agent + stop_reason fields below). Guardrail query: action=cancel, evidence_class=mcp_flat, window=standard must not increase — the narrowed window may only affect established_flat, shell_absent, a prompt-shaped command's platform_limited and remote_flat. |
kirocrew.watchdog.idle.duration |
histogram (ms) | action, evidence_class |
Same emit helper, same decision points; value = the branch's idle clock (_tool_idle / _stale_idle) at decision time, converted to ms at the emit site because the dashboard's generic aggregation reports a histogram under *_ms keys unless its emitting module declares a non-millisecond unit for it, and this one declares none (a seconds instrument would render 1000x off). Answers whether 900s is right for LLM-shaped stalls (idle-at-action distribution per evidence class). Own bucket family _WATCHDOG_IDLE_BUCKETS_MS (1s–4h, densest at the 300/900/3600-second window boundaries). |
kirocrew.watchdog.recovery.outcome |
counter | mechanism (stale_recover / tool_stall / throttle / transient_5xx), outcome (recovered / exhausted), attempt_bucket (1–3) |
dashboard/chat_turn/turn_stats.py::_emit_recovery_outcome, derived from the per-slot retry budgets the stop-reason branches maintain (slot._stale_recovery_retries / slot._tool_stall_retries). throttle and transient_5xx count the interactive ladder's verbatim re-sends of a top-level turn after a pre-output provider error (slot._turn_retries, via chat_turn/recovery.py::_turn_retry_outcomes): recovered when the re-sent turn lands ok, exhausted when it still ends on the terminal error card, one point per mechanism the turn used. exhausted emits in the stall branches when a budget hits its cap ("start a new chat"); recovered emits at the budget-reset block when a turn completes with outcome ok while a stall budget is armed — the stall branches return early, so an armed budget reaching that reset is by construction a completed recovery cycle (gated on ok so a user cancel of the recovery turn never counts as a recovery). attempt_bucket clamps to the 3-attempt cap (closed enum, mirrors the CLI's attempt_number_bucket). Every stall recovered point is one prevented hang. Fault accounting for the exhausted case lives on the turn histogram, not here: the final turn of an exhausted cycle labels stall_exhausted (see kirocrew.turn.duration), so a dead session counts toward fault_rate while this counter stays pure mechanism telemetry. |
kirocrew.autonudge.loops.started |
counter | kind (loop / monitor) |
Every inactive-to-active transition of a loop, so a start is counted for each active period and every stop below has a matching start. Taken at the same commit point and from the same baseline as kirocrew.autonudge.loops.stopped (autonudge_service/store.py::_record_stops_committed, which calls _record_starts_committed): an id active in the rows just written and absent from the last committed store is a start, the mirror of the stop diff. No mutation names this counter, so the prompt arm, the structured arm, a revival through update (a user resume, and the Issue Radar and Research Lab reconcilers re-arming an inactive loop on every pass), the restore after a failed session close, a replacement rollback that writes the displaced row back, and any activation path added later are all counted the same way, after their write has committed; test_an_activation_path_that_never_names_the_counter_is_counted pins that. A pause, a resume and a later stop therefore record two starts and two stops, and a re-save of a loop that was already active counts nothing. A restart counts nothing: _load seeds the baseline from the rows active on disk. A row _load resumes from the torn-deactivation shape (inactive, no stop reason, live deadline) was inactive on disk, so it counts once, at the first commit that writes it active, and its later stop is paired. A row _load could not parse is rewritten verbatim on every commit but never served, so it is skipped. A refused admission, a refused revival, a 409 on an existing record and a failed persist commit nothing and leave the counter untouched. kind is read from the same autonudge_stop_log::_summary the stop record carries: monitor when the row carries monitor state (a structured controller record, or a prompt loop whose probe gate watches a subject) and loop otherwise, so a start and its stop share one label. No loop id, slot key or target is an attribute. |
kirocrew.autonudge.ticks |
counter | kind (as above), outcome (delivered / quiet / refused / failed) |
autonudge_service/firing.py, one point per tick that reached a verdict about spending a turn: the probe gate's quiet branch in _timer records quiet, and _run_fire_cycle records the other three immediately after the fire callback settles. The ratio of quiet to delivered is the number to read -- it says whether loops idle or work, and whether the probe gate is earning the subprocess it costs. refused (the callback declined: the slot was mid-turn) is kept apart from failed (the callback RAISED) because a refusal is the system working as designed while an exception is a defect, and folded together a wedged fire callback is indistinguishable from a busy session. A tick that returns before deciding about a turn -- a spent cycle cap, a stand-down, a hold for a person -- records NOTHING here: it decided about the loop, which kirocrew.autonudge.loops.stopped already reports, and counting it would put events that were never candidates for a turn into the denominator this ratio is read against. The quiet emit therefore sits INSIDE the loop.active and loop.id in self._loops guard: a terminal verdict ("the watched pull request merged") also answers quiet while having just deactivated the loop, so emitting above the guard would report one event twice, once as a tick and once as a stop. refused carries the same guard: a fire callback that removes the loop and then returns False (the session is gone, the channel is unsupported) is counted only as the stop. What this counter does NOT observe: a structured monitor record is driven by its own typed controller (firing.py hands it to _on_monitor_tick and returns) and reaches neither emit site, so it contributes nothing here; its per-tick verdict is kirocrew.monitor.decisions, which is the controller's own decision and the only honest reading of that path. The consequence to know before building a dashboard: within ticks, kind=monitor means a probe-gated PROMPT loop, a narrower population than kind=monitor on the started and stopped counters, so a per-loop rate must not be taken as ticks over loops.started for that kind. The quiet-to-delivered ratio is read inside ticks against its own population and is unaffected. test/metrics/test_monitor_loop_counters.py pins both: a structured record's tick records no point, and a terminal quiet verdict records a stop and no tick. |
kirocrew.autonudge.loops.stopped |
counter | kind (as above), reason (closed: cycle_cap / runtime_budget / stop_sentinel / monitor_terminal / autonudge_stop / approval_stall / session_start_failures / consecutive_failures / structural_terminal / model_refused / sentinel_dropped / verdict_stall / agent_turn_budget / token_budget / provider_error_budget / invalid_monitor_record / unsupported_monitor_version / completion_evidence_unavailable / session_unavailable / manual / user_stop / session_close / success / blocked / budget / target_unavailable / removed / unspecified / other), ran (lt_1h / 1h_24h / gt_24h / other) |
The store's one commit point (autonudge_service/store.py::_record_stops_committed), from the same stop record the WARNING line and the stop file are written from (autonudge_stop_log::stop_records). Exactly once per active-to-inactive transition, by construction: a record exists only for a loop that was active in the previously committed store and is not active now, so a loop resumed and stopped again is counted twice here and twice in loops.started, so a second autonudge_stop on an already-stopped loop produces no record and no increment, a restart does not re-report a stop already counted, and the dozen paths that can stop a loop -- the cycle cap, a spent runtime budget, an unanswered approval, a restart mid-cycle, a user delete, a closed session, an agent's own call -- cannot double-count between them. The stored reason never travels as itself: it is model-authored on some paths and composed at the call site on others (campaign_<status>, orphaned_worker), so metrics/monitor_loops.py::STOP_REASONS maps it to the closed value above and anything unrecognised becomes other. That table is BUILT from the constants and the enum that write those strings, never retyped beside them -- the *_REASON constants in autonudge_service/model.py, the MONITOR_STOP_* constants in monitoring/models.py, the terminal MonitorOutcome a monitor row falls back to when it carries no stop reason of its own, and autonudge_stop_log's two defaults -- so a reason added to any of those sources is reported under its own name without an edit here. A structured monitor that stops on a provider verdict stores the observation's own reason code (pull_request_merged, pull_request_closed, the monitor_blocked default), which is per-provider and not in that table; when the stored reason maps to other and the row is still present with a terminal MonitorOutcome, the counter reports that outcome instead (metrics/monitor_loops.py::monitor_outcomes), so a merged PR monitor reads success and a closed one blocked. A retyped table silently reported the one member it had missed as other, which is what this derivation prevents and what test_every_stop_reason_constant_is_covered and test_every_terminal_outcome_is_covered pin. A spent runtime budget is reported as runtime_budget, the same string the store, the WARNING line and the stop file carry, so one name finds it in all four places. One stored reason is reported under a different spelling, spelled out here because a reported value that differs from the stored string must be findable in one place: the legacy approval_stalled joins the structured spelling approval_stall under one value, because the two are the same event recorded by the two record shapes. ran is a counter with a closed bucket attribute rather than a histogram (RFC rule 2), with edges at 3600s and 86400s: lt_1h is a runtime under one hour, 1h_24h is one hour up to but not including twenty-four, gt_24h is twenty-four hours or more. A row whose creation time was missing or unreadable yields ran=other rather than being folded into lt_1h, which would claim a short life there is no evidence for. The stop record's free-text detail is never read. |
kirocrew.monitor.decisions |
counter | decision (no_change / record_only / wake_actionable / stop_success / stop_blocked / retry_provider / stop_budget -- the members of monitoring.models.MonitorDecision), kind (a registered monitored kind: github_pull_request / gitlab_merge_request / azure_devops_pull_request / bitbucket_pull_request / gh-pr / github_workflow_run, else other) |
monitoring/decision.py::decide_monitor, once per call, at the one place every path through the policy converges -- after the stall streak is folded, so what is counted is the verdict the caller acts on rather than an effect a later step may still replace. The population is POLICY EVALUATIONS, which is narrower than every verdict a caller acts on: monitor_records.py::apply_monitor_probe short-circuits some ticks to a verdict of its own (a record whose outcome is already written, a reading it refuses) without consulting the policy, and those are deliberately uncounted -- the series answers "the policy was asked, and this is what it said", which is what makes a rate over the decisions readable. monitoring/shadow.py::run_shadow_probe also asks the policy and is counted the same way; it has no caller in the gateway today, so every point on this series comes from a live monitor. A pure policy module may carry this because recording is an in-memory add that reads no clock, file or socket and swallows its own failures, so the module stays deterministic for its callers and its tests. kind is checked against monitoring/registry.py, the catalogue every boundary that gates a kind already reads, so a kind from a store row written by a newer gateway has no entry and is reported as other; the target, the objective and every fingerprint stay out. metrics/monitor_loops.py::MONITOR_DECISION_VALUES is DERIVED from MonitorDecision rather than restated beside it: a second hand-written copy of a seven-member enum could only ever catch a new decision by reporting it as other while reading as though someone had checked. |
kirocrew.context.section.duration |
histogram (ms) | section (one fixed label per assembled block: preamble / profile / workspace / docs / steering / thread_history / stop_notes / memory / skills / lessons / provenance / finalize, plus episodic from the build_message site), custom (bool) |
Two sites, both first-turn only. ContextBuilder.build_session_context (context_assembly/session_start.py) emits one point per section from monotonic checkpoints taken as each block is appended; ContextBuilder.build_message (context_assembly/turn_message.py) emits section=episodic for the query-dependent episodic retrieval that runs as that method's sibling rather than one of its sections. Why per-section: the block is assembled AFTER the user's message arrives and the caller awaits it before dispatching the prompt, so every section lands directly on time-to-first-token; as one opaque interval the cost is unattributable and diagnosis degrades to guess-and-rebuild. The spread within a single build is the widest of any instrument here — string appends under a millisecond alongside a query-embedding section reaching seconds — which is why it takes _FAST_BUCKETS_MS (0.5ms..60s) rather than a startup ladder. custom is a bool rather than the agent name deliberately: a populated install has dozens of agents, and one series per agent per section would multiply series count for no diagnostic gain. Sections under 1ms are still recorded as points but omitted from the companion INFO line to keep it readable. |
kirocrew.db.query.duration |
histogram (ms) | store (memory / knowledge / vector / history / other), op (search / read / write / delete / migrate / connect / other), outcome (ok / error) |
metrics/db_metrics.py; times logical SQLite store operations rather than SQL statements. Raw SQL and search terms never become attributes. |
kirocrew.crew_log.write.lost |
observable gauge (lifetime total) | cause (dropped / overflowed / wake_dropped / other) |
metrics/crew_log_health.py::register_crew_log_health_instruments; entries the crew log never got, and fold wakes its queue could not hold. Every cause is detected on the crew log's APPEND thread or on a producer's own thread, and is already counted there (WriterStats.dropped / .overflowed, crew_log/eager.py::dropped_wakes), so this adds no call at those sites: one observable callback runs on the metrics reader's ticker thread, reads the three counters through sys.modules (importing nothing) and folds each delta into a monotonic total. A GAUGE whose reading is that process-lifetime total, not a counter, and for the reason the process CPU and GC readings became gauges: reading a total this module does not own can only be an async instrument, an observable counter would export CUMULATIVE, and test_no_instrument_exports_cumulative allows no cumulative series anywhere because one forces every consumer into stateful whole-hour aggregation. Declaring DELTA for it instead is the move metrics/temporality.py records as tried and reverted, since the delta baseline lives in the provider and a consent change rebuilds it. The loss RATE is the consumer's difference between two readings. The first collection is a BASELINE, so losses accrued before telemetry was enabled are not reported, and the totals are in memory only, so a restart re-counts nothing. dropped and overflowed are real holes in the log (a stuck disk vs a saturated one); wake_dropped is not, because the next read carries the fold forward, so it only means a dashboard saw the value late. |
kirocrew.crew_log.fold.apply |
counter | scope (slot / session / other), fold (the crew log's fold catalogue: status / usage / timeline / tools / approvals / subagents / class / outline / otel_usage / otel_session_tree / ledger / radar / work / panel / agentic / mistakes / workstreams / worktree / other), bucket (lt_10ms / lt_100ms / lt_1s / ge_1s, bounds exclusive, so exactly 10ms is lt_100ms) |
crew_log/eager.py::_advance and ::_advance_session close each pass through metrics/crew_log_health.py::note_fold_pass. One point per fold the eager worker's own pass ADVANCED over one coalesced batch, on that worker's thread and nowhere else: every record site is a thread-local check that returns at once on another thread, so the same fold code reached from a dashboard read records nothing. A session pass walks one unit's file once for every session fold, so the duration is the pass's and is attributed to each fold that moved. A pass that advanced nothing records no apply, whether an unchanged live cell answered it, the read path refused it, or it raised, so fold.start above fold.apply is the share of eager folding that produced no value. READ THIS BEFORE READING THAT RATIO: when a dashboard read reaches a growing cell first, the read does the folding and the worker's pass behind it finds nothing changed, so that fold counts as a start and not as an apply; the pair describes eager folding, not all folding. The bucket measures the pass end to end, INCLUDING any wait for the per-cell fold lock, because a pass held up behind a reader is slow eager folding. The clock is read once when the pass closes, before any point is recorded, so every fold of one pass lands in the same bucket and a first recorder build is not part of the duration. A bucket counter rather than a histogram, per the no-new-histogram rule: the edges live in crew_log_health.bucket_for (shared with savepoint.write) and in this row, nowhere else. The fold catalogue has ONE owner, crew_log/projection_catalogue.py::FOLD_NAMES, read into a frozen set at import, so a fold the crew log gains is labelled with no edit here and the names above are the catalogue as it stands rather than a second copy that could drift. A second literal copy with a check that raised on drift would be reached through the fold surface's module-scope import of the metrics module, which would turn a forgotten label edit into a failed import of every dashboard read and session fold. |
kirocrew.crew_log.fold.start |
counter | scope (as above), origin (warm / savepoint / cold / other) |
The branches that decide where a fold's starting state comes from, which are crew_log/projection_session.py::_fold_attempt and crew_log/projection_slot_memo.py::_fold_slot_warm_locked, leave a thread-local mark (crew_log_health.note_fold_origin), and the eager worker turns it into a point when it closes the pass. A mark rather than a record because two of those branches hold a per-cell fold lock and a first get_recorder() builds the telemetry provider synchronously, which would hold that lock against a waiting reader; the marks are keyed by scope because resolving a slot fold's units can fold a session log. Exactly one point per eager pass: a pass that folded twice after the file changed identity under it reports the origin it finally folded from, and a pass that raised before choosing one records nothing at all. Eager fold worker thread only, as fold.apply is. A rising cold share is what a savepoint that is no longer being kept looks like from the outside; warm includes a pass an unchanged live cell answered without folding anything, which is the case fold.apply leaves out. |
kirocrew.embed.queue_wait / .inference |
histograms (ms) | queue wait: none; inference: chars_bucket (<=128 / <=512 / <=2000 / <=6000 / >6000) |
embeddings_llama.py::_emit_embed_timing; separates time waiting behind another embed from model inference time. |
kirocrew.telegram.api.duration |
histogram (ms) | method (Bot API method), outcome (ok / timeout / rate_limited / error) |
telegram/client_http.py::_record_api_duration; awaited Bot API round-trip latency. |
kirocrew.mcp.backend.acquire.duration |
histogram (ms) | warm (bool — not was_spawned) |
mcp_gateway/daemon/diagnostics.py::_emit_backend_acquire_metric — ensure_backend pre-flight + lazy-spawn paths; acquire-only duration captured before attach_stub/create_task overhead. |
kirocrew.mcp.lazy_load.count / .duration |
counter + histogram (ms) | transport (stdio) |
mcp_gateway/daemon/diagnostics.py::_emit_lazy_load_metrics — legacy lazy-spawn path (also emits backend.acquire). |
kirocrew.mcp.warm_pool.acquire |
counter | result (hit / miss) |
mcp_gateway/prewarm.py::HotKeyStore.record_outcome (emitted outside the lock). |
kirocrew.mcp_gateway.backends.live |
gauge (1) | class (first_party / third_party / other) |
metrics/mcp_gateway_gauges.py::register_mcp_gateway_gauges, callbacks run only at reader collection and observe only in the process that called bind_pool (gatewayd, in run_gatewayd). Live pooled backend processes, read through BackendPool.live_pooled_backends: the same population the dashboard's pool snapshot (BackendPool.metrics_snapshot_async, served by daemon/control.py) counts, so the two saving figures agree. Draining backends (an old process finishing calls after a credential rotation) and connection-private backends (never shared) are excluded: no session can be pooled onto either. class is first_party for a name in mcp_discovery_managed._MANAGED_SERVER_NAMES, third_party for any other name, other when the name is missing; the name is dropped after classifying. first_party and third_party are always published (0 is a real reading), other only when non-zero. |
kirocrew.mcp_gateway.sessions.attached |
gauge (1) | class |
Same module and population: the summed stub attachments (Backend.refcount) on those backends. sessions.attached - backends.live is the number of backend processes pooling avoided. One runtime opens one stub per server however many sessions it hosts, so this is a lower bound on sessions and the saving derived from it is a lower bound too. |
kirocrew.mcp_gateway.backends.rss_bytes |
gauge (By) | class |
Same module and population: summed current RSS of each backend through mcp_gateway/pool.py::backend_rss_kb, the probe the dashboard's pool snapshot uses (the backend pid AND its descendants, so a launcher that forks the real server is measured with it; KiB, exported as bytes). A pid with no reading (it died, or the platform has no /proc subtree walk: macOS and Windows) adds 0 and is skipped; a class where no backend could be read publishes nothing, a gap rather than a fake zero. bytes_saved ~= (sessions.attached - backends.live) * backends.rss_bytes / backends.live. gatewayd's own RSS is kirocrew.process.memory.rss_bytes where the resource has kirocrew.process.role=gatewayd; the main process's has no kirocrew.process.role. |
kirocrew.skill.lazy_load.count / .duration |
counter + histogram (ms) | hit (bool); the counter also skill (closed, see the usage metrics section) |
skills_body_reads.py::load_skill (bound as SkillsLoader.load_skill) via _emit_lazy_load_metric (best-effort; never breaks skill loading). |
kirocrew.subagent.spawned |
counter | concurrency (int — the live running count INCLUDING this spawn, bounded by the manager's own concurrency cap, so a MAX over it is the high-water mark without a second instrument), batched (bool — the spawn is a member of a fan-out wave rather than a lone one) |
subagent_manager/admission/pump.py::_PumpMixin._log_spawned_impl — the confirmed-start funnel, reached only after a spawn is admitted, so a rejected or never-started spawn is not counted (a promise an admission-time increment could not make). The emit_counter import is function-local, not module-level: bind_component_globals rebinds each *_impl's __globals__ to the owning namespace, where a module-level import here would not be visible. |
kirocrew.subagent.completed |
counter | outcome (completed / failed / stopped / unknown -- SubagentInfo.outcome's own canonical three-way classification, restated as metrics/events.py::SUBAGENT_OUTCOMES because metrics sits below the sub-agent manager in the dependency order, and folded through subagent_outcome so a record whose outcome cannot be read lands on unknown rather than being guessed as failed) |
subagent_manager/terminal.py::TerminalCoordinator._record_terminal_metric, called from _report_terminal_impl past the point its finalize claim has been granted. That point, rather than either crew-log closer, is what makes one increment per run true: the reaper and the run loop's own finally both race to report and exactly one is let through, whereas a counter on subagent/completed and a second on subagent/failed would count a run twice on the paths that reach both. Gated on telemetry.enabled alone and NOT on the crew-log emitter, so the population is not narrowed to installs that happen to keep a crew log. The emit_counter import is function-local for the reason the row above gives. |
kirocrew.webhook.requests |
counter | outcome, over two disjoint halves plus a remainder. Recorded runs (webhooks.VALID_OUTCOMES): completed / timeout / error / rejected_capacity / unauthorized / disabled. Refusals the run store never sees (webhooks.METRIC_REFUSAL_OUTCOMES): throttled (the per-source auth-failure throttle), unavailable (a store the endpoint must read is unreadable, or a destination agent could not be verified), rejected_body (the request's own content refused -- too large, unreadable, unparseable, or a field of the wrong type). Remainder: other. The halves are disjoint by construction and pinned by a test, so one request is never counted under both. No free text reaches an attribute -- the run's detail, name, hook id, session key and token id are all excluded. This extends RFC 6.4's list, which named only the recorded-run outcomes. |
TWO emitters, both through webhooks.py::count_request, and neither is redundant. WebhookRunStore.record counts each recorded run, outside its own try, since the series counts the DECISION and not the history row. dashboard/handlers/hooks.py::api_hooks_agent counts the requests it turns away before a run exists, with an inline count_request immediately above each of its 21 refusing returns -- recording there is impossible or self-defeating (the unreadable store is what would be written; the throttle exists to stop doing work). Inline rather than through a shared helper because the error-code contract gate requires a literal code in a dict literal at the json_response site, which a helper taking the body as a parameter cannot provide; completeness is guaranteed instead by test_webhooks_request_counter.py::test_every_refusal_branch_is_counted, which reads the function's syntax tree and fails if any answer reaches a caller without reaching the counter. The accepted path is NOT counted at admission: its run records its own terminal outcome later, and counting both would double-count it. |
kirocrew.inventory.webhooks.registered |
observable gauge | none | metrics/inventory_gauges.py::read_registered_webhooks -- entry count of the webhook callback registrations in hooks.json, reduced by webhooks.registered_context_count, the same owner the dashboard's listing and its id-addressed delete use, so the gauge cannot report a number no surface can confirm. The file is shared with the script-hook store, whose entries are not registrations and are not counted. A MISSING file is a genuine 0 (the steady state of an install that uses no webhooks); a file that is present and does not parse yields NO observation and increments kirocrew.inventory.probe.failures{probe=webhooks}, the same fault/gap split crons.json draws. registered_context_count reads it without logging, so a file that stays corrupt is reported at WARNING once, by the probe, not on every export interval. No hook id, session key or context summary is read -- each is agent-chosen free text, and the count rides in the value. |
kirocrew.cron.fires |
counter | kind (agent / script / command — the dispatch shape; script and command bypass the model entirely, so this is the split between a job that costs tokens and one that costs none), trigger (scheduled / manual) |
cron_service/service/run_execution.py::_run_job_isolated (bound as CronService._run_job_isolated), one increment per execution, emitted BEFORE the jitter sleep so a run cancelled during jitter still counts as fired. No job id, name, message or schedule string reaches the attributes. |
kirocrew.artifact.created |
counter | kind (the closed ALLOWED_KINDS set), source (the closed ALLOWED_SOURCES set), kind_auto (bool — the kind was inferred from the content rather than pinned by the caller) |
artifacts_writes.py::create (bound as ArtifactStore.create), emitted after the write and its change notification, so a failed create contributes nothing. kind and source are exactly the values _validate_kind / _validate_source already close over, so no slug or free-form label can enter the series. |
kirocrew.workflow.runs |
counter | authored (bool — the run writes its own script from an intent instead of executing supplied source), replay (bool — a restart-subtree reusing cached agent results rather than re-calling the model) |
workflows/runner.py::WorkflowRunner.run, the one place every run passes before executing anything, foreground or background (the background entry point drives this same method). Both attributes are booleans over run mode; no intent text, script or run id is emitted. |
kirocrew.work.created |
counter | kind (goal / item), depth (0 / 1 / 2+ — the creating board's own depth) |
dashboard/handlers/work_ledger_conductor.py::api_work_ledger_record via _count_recorded, after the write's work/recorded entry landed in the crew log, which is the point the route itself treats as the commit. kind=goal counts ONLY the goal write that opened the board (its committed header carries goal_version == 1); a reword, a round bump, a cap raise and an explicit new_goal are all later writes and count nothing, so the series reads as boards rather than as goal writes. kind=item is one minted item. Declared in metrics/work.py; no goal text, title, acceptance or item id is emitted. |
kirocrew.work.item_closed |
counter | state (accepted / rejected / abandoned — the ledger's terminal states, since close refuses open), fails (0 / 1 / 2 / 3+), duration (le_10m / le_1h / le_4h / gt_4h, each edge inclusive: 600 s / 3600 s / 14400 s from the item's created_at to its closed_at, beyond which gt_4h) |
Same site, on action=close, from the committed item rather than from the request. A duration is a bucketed counter rather than a histogram so the distribution survives a backend that keeps no explicit buckets; the edges live in metrics/work.py::DURATION_BUCKET_EDGE_SECS. An unreadable or missing stamp yields duration=other rather than a plausible-looking zero. Counted at the close rather than in the crew log's work fold because that fold forgets a closed item once WORK_CLOSED_WINDOW newer ones close, keeping only a total — so state, fails and the duration exist only at the moment the close commits. |
kirocrew.work.dispatches |
counter | target_kind (session / crew) |
Same site, on action=bind — the action that names the target — after that bind's work/recorded entry landed. Deliberately NOT on the crew/dispatch echo written just after it: that echo is best-effort and is skipped entirely for a board driven from a slot belonging to no crew, so counting on it would drop every dispatch such a board made. session is the only kind the bind builds; crew is declared because crew_log/entry_types.py::CREW_TARGET_KINDS names it, so a second target kind lands in its own series instead of in other. |
kirocrew.work.reports |
counter | status (progress / done / blocked / question / reply / failed; reply only while work_ledger.remote_dispatch is on), reason (approval / needs_human / none when the report named none) |
dashboard/handlers/work_ledger_worker.py::api_work_report, after the report's work/recorded entry landed, from the committed item. A refused report — an unknown or closed item, a done carrying no evidence, a status outside the worker's vocabulary — returns before this point and contributes nothing. reason is carried as the literal none rather than omitted, so "no reason given" is distinguishable from a dropped attribute. failed is declared because crew_log/entry_types.py::CREW_REPORT_STATUSES names it; the ledger's worker half can commit only the others, so this route never emits it today. No summary or artifact text is emitted. |
kirocrew.crew_mode.turned_on |
counter | via (switch / create), repeat (bool — a Crew Mode chat already existed) |
dashboard/chat_api/agent_switch.py::switch_slot_agent after a committed switch from a non-conductor onto a conductor (via=switch), and dashboard/chat_api/slot_create.py::api_chat_slot_create for a new chat born on one (via=create). repeat is the repeat-use signal: true when the gateway already knew another Crew Mode root. No session key, title or agent name is emitted. |
kirocrew.crew_mode.new_chats |
counter | crew_mode (bool) |
slot_create.py::api_chat_slot_create, one per new chat. The share of new chats in Crew Mode is sum(kirocrew.crew_mode.turned_on) / sum(kirocrew.crew_mode.new_chats) over a window. |
kirocrew.crew_mode.runs_finished |
counter | state (accepted / rejected / abandoned) |
dashboard/handlers/work_ledger_conductor.py::_count_crew_mode_run, after an action=close on a board a Crew Mode root owns landed in the crew log. A run is one crew member item a Crew Mode chat dispatched. |
kirocrew.crew_mode.steers_per_run |
counter | steers (0 / 1 / 2 / 3+) |
Same site: the closed item's own steer events, bucketed. A bucketed counter rather than a histogram so the distribution survives a backend with no explicit buckets. |
kirocrew.crew_mode.steers |
counter | on_item (bool — the steer landed as a steer event on a ledger item) |
dashboard/handlers/crew_workers.py::api_crew_worker_steer, one per delivered person steer. No message text is emitted. |
kirocrew.crew_mode.worker_waits |
counter | status (blocked / question), reason (approval / needs_human / none) |
dashboard/handlers/work_ledger_worker.py::api_work_report, after a committed report on a Crew Mode board that says the crew member is stuck or needs the person. Other statuses count nothing here. No summary text is emitted. |
kirocrew.context.compactions |
counter | success (bool — whether the compaction ATTEMPT itself completed) |
session_compaction.py::CompactionCoordinator._fire_compact_callback, one point per compaction that reached a verdict. Placed on this method rather than inside the callback because every compaction passes through here whether or not a surface registered a compact callback, and the early return below would otherwise drop exactly the surfaces that register none. The sum is compaction ATTEMPTS. success is deliberately NOT an effectiveness measure: the coordinator learns whether the backend finished the compaction, not how much context it actually freed, and no before/after context reading is taken, so a compaction can complete while reclaiming little. The counter's success is also not the callback's. _recycle_held fires this funnel with success=True because the SESSION now has headroom — which is what the callback needs to know — but it is reached exactly when an in-place /compact FAILED and the provider had to be replaced instead. Counting that as a successful compaction would report the failure mode as the success case, so the counter reports False there. The recycling marker is set for the whole of _recycle_held and popped only after this call, which is what separates the two populations at this point; test_a_failed_compact_that_recycles_is_not_counted_successful pins that ordering. |
kirocrew.mcp.reconnects |
counter | pool (bool — the reconnecting stub was a warm-pool bridge rather than a directly owned one) |
mcp_gateway/stub_link.py::_reconnect, one increment per stub reconnect to a restarted gateway daemon, emitted after the replayed handshake is delivered. Each increment is a bridge that died and was rebuilt under a live session, so a rising rate is daemon instability that sessions are absorbing silently. The daemon detail string and the per-session reconnect ordinal go to the log line only, never the attributes. |
kirocrew.mcp_gateway.liveness.escalated_probes |
counter | answered (bool — the daemon replied to the longer probe after missing the fast 2 s ping) |
mcp_gateway/manager.py, once per escalated liveness probe. answered=true is a loaded-but-alive daemon, the precursor to a kill cycle; false missed both probes |
kirocrew.mcp_gateway.liveness.escalated_latency_ms |
histogram (ms) | process (gatewayd) |
mcp_gateway/manager.py, recorded only when the escalated probe was answered (a miss has no latency), so the _LIVENESS_ESCALATED_TIMEOUT_SECS bound can be checked against field data |
kirocrew.approval.decisions |
counter | decision (allow / auto_approve / deny — the gate's own bounded action constant), security_deny (bool — separates a hard security refusal from a policy-state deny) |
hook_runtime/hook_types.py::ToolHookResult._count, called by the four result factories and nowhere else, so it fires exactly once per gate consultation. A surface that OVERRIDES the gate constructs a result directly (the chat runner downgrading an auto-approve to an interactive card, for instance) and is deliberately NOT counted: counting every construction would report one request as two decisions and keep a count for a verdict that was discarded. Counting from the factories rather than from __post_init__ behind a flag is what removes the flag — it had no reader but the counter, so it was state carried purely to signal. Both attributes are bounded by construction — no deny reason, tool name or command reaches the recorder. |
kirocrew.adaptive.decisions |
counter | action (decrease / increase / pause / probe / resume) |
adaptive/controller.py::AdaptiveController.apply; emitted only when adaptive concurrency changes a cap or pause state, not on hold/fixed ticks. |
kirocrew.sel.prune_failed.count |
counter | — | heartbeat.py; one point when the daily Security Event Log retention prune raises. |
kirocrew.taskq.depth |
histogram (1) | state (a taskq.model state name — the closed model.STATES set, terminals included: the sampler reads a bare GROUP BY state) |
dashboard/session_health.py::SessionHealthMonitor._sample_metrics, one observation per state per health computation: the durable queue's depth as the sampler saw it. The aggregator's last/max reading per state is the gauge. |
kirocrew.taskq.oldest_wait_secs |
histogram (s) | — | same sampler; age of the oldest row still waiting for dispatch (TaskStore.oldest_wait_secs). |
kirocrew.taskq.completions |
counter | outcome (the terminal state name) |
taskq/store.py::TaskStore.transition on every terminal write and TaskStore.cancel; the rate over it is the completion rate. |
kirocrew.taskq.effective_cap |
histogram (1) | lane_kind (subagents, spawn_gate, or a name a controller registers — code-bounded) |
same sampler; the cap currently ENFORCED per lane, equal to the user maximum only when nothing is degraded. |
kirocrew.taskq.pressure_reason |
counter | reason (the controller's closed reason enum) |
same sampler, one increment per health computation taken while a degrade reason is active. |
kirocrew.crew_log.savepoint.write |
counter | kind (session / slot / other), outcome (written / failed), bucket (lt_10ms / lt_100ms / lt_1s / ge_1s) |
crew_log/checkpoint.py::save (session) and ::save_slot (slot), one point per pass that actually offered a payload to the savepoint store — roughly once per 256 entries. A pass that returned before writing (no write earned yet, no witness, the unit removed, the lease contended, an unserializable state, every fold sitting at another boundary) records NOTHING: those are refusals, and counting them as failures would read as a failing disk. outcome compares what landed against what was OFFERED (written == attempted), not against the whole bundle — a session pass skips any fold whose boundary its witness does not certify, and holding the pass to those too would report a refusal as a failing disk. Whether the bundle may then CLAIM a savepoint is a stricter question answered separately in save. Both points are taken in a finally, so a raise through the store counts as failed rather than going unrecorded, and both are emitted after release_lease so the first (synchronous) recorder build cannot lengthen the window in which a removal of that unit is refused. bucket bands the store write itself from a time.monotonic() pair; both call sites are on the fold and read path, never a write thread. No unit id, slot key or fold name is an attribute. |
kirocrew.crew_log.eager.queue_depth |
gauge (observable) | — | metrics/crew_log_health.py::read_eager_queue_depth → crew_log/eager.py::queue_depth, the live Queue.qsize() read when a reader collects. A gauge from live state rather than an up/down counter, so a crash cannot leave it drifting. Absent, not zero, when this process has committed no eager entry: there is no fold thread and no queue, which is a different fact from an empty queue. The reader never calls _ensure_worker, so a telemetry collection cannot be the thing that starts the fold thread. |
kirocrew.crew_log.writer.buffered |
gauge (observable) | — | metrics/crew_log_health.py::read_writer_buffered, WriterStats.buffered read live at collection — crew log entries the writer is holding, the backlog number that distinguishes a stale board from a wedged disk. Same two rules as the queue gauge: live state rather than an up/down counter, and absent rather than zero when no writer was built in this process (it reads the installed instance directly rather than through the accessor, which would build one). |
kirocrew.crew_log.bus.subscriber_error |
counter | kind (fold_advanced / tree_advanced / other) |
crew_log/bus.py::publish and ::_call, the two branches that swallow a subscriber's exception (mutually exclusive — _call serves only the baseline replay in subscribe, so one exception is never counted twice). Failure path only, and the count is what the log line cannot give: the publisher is told nothing by design (it has already done the fold), and a WARNING per failed event carries neither a rate nor a total, which is what an operator asks of a subscriber failing on every fold. The bus has no queue — an event with no subscriber is dropped by design — so there is no backlog to measure. This is the one counter on this table a crew log WRITE thread can reach: tree_advanced is published from the session-tree projection, which a session/opened or session/spawned job advances while the write-behind is running that job. Three things bound it — a subscriber that returns normally costs nothing at all, the raising path already formats and emits a WARNING that costs more than the counter beside it, and the counter resolves this module from sys.modules rather than importing it, so a write thread can never trigger the first load of the metrics facade. fold_advanced is published only from the eager fold worker. The exception text and the subscriber's identity never become attributes; a kind the enum has not been taught maps to other. |
kirocrew.host.procs_peak / kirocrew.host.fds_peak / kirocrew.host.rss_peak_mb |
histogram | — | declared for the host budget's peaks since the previous sample (emitter: the budget snapshot, wave-3 integration). |
kirocrew.loop.lag_ms |
histogram (ms) | process (gateway / gatewayd) |
adaptive/controller.py::AdaptiveController._sample_and_tick, one observation per controller sample: how late the sample_secs timer fired on the gateway event loop, the same number the controller feeds its loop_lag signal. The cap decreases and pauses that signal drives are logged at WARNING. |
kirocrew.recovery.attempts |
counter | layer (L1_tool_call … L5_gateway), action (retry / escalate / notify / give_up) |
recovery/ladder.py::RecoveryLadder.observe_failure, one per decision. |
kirocrew.recovery.escalations |
counter | from_layer, to_layer |
same, one per hand-up. |
kirocrew.recovery.duration_secs |
histogram (s) | layer |
RecoveryLadder.observe_success: first failure of the run → the success that closed it. |
kirocrew.recovery.restarts |
counter | layer |
RecoveryLadder.record_restart (the gatewayd watchdog after each respawn; backend/runtime rebuilds at their sites). |
kirocrew.gateway.boot.duration |
histogram (ms) | server (dashboard / api), outcome (ready) |
dashboard/server.py::start_dashboard / dashboard/server_runtime/api_entrypoint.py::start_api_server — boot-to-ready: wall-clock from the server's start_time until full init completes and it is about to accept traffic. Emitted via metrics/http_metrics.py::record_boot_to_ready. Best-effort; never blocks startup. |
kirocrew.gateway.request.duration |
histogram (ms) | method (fixed HTTP-verb allowlist, else OTHER), route_template (matched aiohttp canonical TEMPLATE, e.g. /api/artifacts/{slug}, else __unknown__), status_class (1xx..5xx / other) |
metrics/http_metrics.py::make_route_latency_middleware — outermost gateway middleware on BOTH start_dashboard and start_api_server. Times full in-gateway HTTP handling; upgraded WebSocket connections and text/event-stream SSE responses are excluded so connection/turn lifetime cannot pollute request latency. Bounded cardinality (see below). |
kirocrew.crew.session.started |
counter | agent_kind (closed: the owned agent-file stems, plus default, crewmate, other) |
metrics/crew_log_export.py::_otel_usage_diff, from the crew log's otel_usage fold rather than from a session-lifecycle call site. Distinct from kirocrew.session.started above, which counts a gateway session allocation split by session_source: this one counts a crew LOG opening split by the kind of agent that opened it, and only for logs this process both wrote and could prove were opened after it installed the subscriber. Never over-counts across a restart; under-counts a session opened before install. |
kirocrew.crew.turn.started |
counter | actor (the crew log's own closed ACTOR_VALUES set), agent_kind (the same closed set as kirocrew.crew.session.started, for the session the turn ran in) |
Same module and fold; one add per turn/started entry, derived as a delta against the last cumulative render this process exported for that session. agent_kind is read from the fold's already-closed opened_agent_kind, never from the raw agent name, so turns run inside conductor and worker sessions can be counted apart. |
kirocrew.crew.export.lag |
histogram (ms) | — | Same module, on_fold_advanced; one observation per delivery an _EXPORTERS row actually handled, so the session-scoped deliveries for other folds that reach the same callback are not timed. Self-timing, so "the export is not slow" is read rather than asserted. SDK default boundaries for now — this instrument is deliberately absent from _HISTOGRAM_BUCKETS_MS until a real install's deliveries are measured. |
kirocrew.crew.session.tree.depth |
counter | depth (closed: 0, 1, 2, 3+) |
Same module, from the crew log's otel_session_tree fold; one add per session this process can prove is new to it, bucketed by that session's BIRTH depth as session/opened.depth recorded it (crew_log/otel_session_tree_fold.py::depth_observation / depth_bucket). A counter per bucket label rather than a histogram, so the distribution survives backends that do not keep explicit histogram buckets. A session whose depth could not be established (the tree unseeded, the reading incomplete, a creator cited with no log, a cycle) carries no depth and is left OUT of the distribution rather than defaulted to 0, roots included, so it is never censored on one side only. |
kirocrew.crew.session.tree.child |
counter | parent_kind (the creator session's agent_kind, same closed set as kirocrew.crew.session.started), nth (closed: 1, 2, 3, 4-7, 8+) |
Same module and fold; one add per child session a creator makes, read from the session/spawned entries on the creator's own log. nth is the child's ordinal among its creator's children (otel_session_tree_fold.py::nth_bucket_counts). Creators with at least k children = the sum of adds at nth >= k, so the width distribution is read without waiting for a creator to end. Same never-over-count rule: a creator not provably new to this process is baselined on first sight. |
kirocrew.crew.session.tree.roots |
gauge | — | crew_log/otel_session_tree_roots.py::register_session_tree_gauge, registered from provider.py beside register_inventory_gauges / register_process_gauges and so collected on the reader's own thread. How many roots in the folded session tree have at least one child — the count of trees that actually branch, read by folding the in-memory tree once per collection (bounded by TREE_UNIT_CAP). A reading the tree cannot give yields a gap, never a 0, on the same condition the depth counter withholds an add. |
| kirocrew.process.threads.python | gauge | — | metrics/process_gauges.py::register_process_gauges, callbacks run only at reader collection (no polling threads). threading.active_count(). |
| kirocrew.process.threads.os | gauge | — | Same module; platform_compat.process_thread_count(os.getpid()) — OS-level count that catches native pools (ggml, grpc) invisible to threading. Linux-only; None elsewhere (gap, not zero). |
| kirocrew.process.open_fds | gauge | — | Same module; delegates to platform_compat.count_open_fds (shared with gatewayd's zombie-diagnostic fd_count): /proc/self/fd or /dev/fd entry count minus the enumeration fd; Windows reports the kernel handle count (platform-dependent semantics). |
| kirocrew.process.memory.rss_bytes / .peak_rss_bytes | gauge (By) | — | Same module; delegate to platform_compat.proc_rss_bytes (current) / proc_peak_rss_bytes (high-water mark), both cross-platform. A 0 return maps to None: gap, never a fake zero sample. |
| kirocrew.process.cpu.seconds | gauge (s) | — | Same module; platform_compat.proc_cpu_seconds process-lifetime user+system CPU. An observable GAUGE, not a counter: as counters these four were the only CUMULATIVE series exported, and one cumulative series forces every consumer into whole-hour stateful aggregation. Listed in process_gauges.LIFETIME_TOTAL_METRICS, which the dashboard aggregator reads to reduce them window-relative rather than reporting a lifetime total as a current reading. A 0 return maps to None: gap, never a fake zero (which the reducer would take as a new baseline). |
| kirocrew.process.gc.collections / .collected / .uncollectable | gauge | generation (0/1/2) | Same module; gc.get_stats() per generation, process-lifetime totals reported as gauges (see the CPU row). Rules GC in/out of a leak diagnosis (rising uncollectable = reference cycles; flat collected with rising RSS = native leak). |
| kirocrew.process.memory.rss_sampled | histogram (By) | process | adaptive/controller.py::AdaptiveController._emit_process_histograms, one observation per controller sample, from the HostSample.rss_mb the loop already probes for its own decisions. A reading of 0.0 publishes nothing: platform_compat.proc_rss_bytes answers 0 on failure by contract rather than raising, so the probe's own except never fires and a failed read would otherwise enter a CUMULATIVE distribution as a fabricated zero that no later sample can correct — and a live process never has a true resident set of zero, so the strict predicate discards nothing real. The same quantity as the rss_bytes gauge above, kept as a SECOND instrument because merging a gauge across instances retains only min/max/mean — a fleet-wide p90 is not recoverable from it at any storage layer, while histograms sharing bucket boundaries merge element-wise. Recorded from the controller rather than from process_gauges because OTEL has no observable histogram: a histogram needs a caller on a timer, and this loop already is one. Own bucket family _RSS_BUCKETS_BYTES. |
| kirocrew.process.cpu.utilization | histogram (1) | process | Same emitter and cadence; share of the whole machine as a ratio where 1.0 is every logical core saturated. Computed client-side by process_gauges.cpu_utilization from two consecutive HostSample.cpu_seconds readings over the interval between the instants those two readings were TAKEN at, divided by read_logical_cores(). The instant is HostSample.cpu_clock, read inside probe_host beside the total rather than on the event loop after the worker thread resumes: the resumption delay varies from tick to tick so it does not cancel, and at the one-second floor of controller_sample_secs a few hundred milliseconds of it is a double-digit-percent error in a published value nothing downstream can correct. Computed here rather than downstream because process.cpu.seconds is a lifetime TOTAL: a consumer would have to difference consecutive samples per process lifetime and detect restarts to recover a rate, and once that gauge is merged across instances there is nothing left to difference. Returns a gap, never a fake zero, when either reading is <= 0 (a failed probe reads 0.0), the clock did not advance, the core count is unknown, or the total went backwards (two different processes). The first tick of a process has no predecessor and so publishes nothing. Values above 1.0 are NOT clamped — nothing samples the clock and the kernel's accounting at the same instant — which is why the bucket family carries a bound above saturation. Own bucket family _CPU_RATIO_BUCKETS. |
| kirocrew.inventory.crons.active | gauge | — | metrics/inventory_gauges.py::register_inventory_gauges, callbacks run only at reader collection. The enabled-count reduction is cron.enabled_count_from_disk, a module-level sibling of unhealthy_jobs_from_disk that returns (count, loadable) and is the single owner of that loop — CronService.count_enabled_from_disk calls it too and keeps only the count, so the probe cannot drift from what the scheduler considers enabled, and neither needs a running gateway. Deliberately NOT list_jobs: that re-arms the asyncio timer, which from a reader thread with no running loop raises AND cancels the live timer first, stopping every scheduled job. A missing store is a genuine 0 and no fault (a fresh install has no crons). A store that is present but unparseable yields a gap and increments probe.failures{probe="crons"} — _read_job_records calls that case a fault, and a silent gap would be indistinguishable from a host that stopped exporting, which is the confusion the counter exists to resolve. |
| kirocrew.inventory.monitor_loops.active | gauge | — | Same module; autonudge.get_instance() then a materialized list(...) snapshot of the loop registry counting active. The registry is mutated only under an asyncio.Lock on the event loop, which gives a reader thread no protection and cannot be acquired from it; the snapshot is safe anyway because each dict op is atomic under the GIL and a gauge stale by one loop is fine. get_instance() returning None (spawned agent process, unit test, auto-nudge off) yields a gap — a 0 there would be indistinguishable from a running service with none armed. |
| kirocrew.inventory.skills.installed | gauge | — | Same module; one module-global SkillsLoader(install_builtins=False) (install_builtins=True would make a metrics probe perform a content-hashing write sync). Listing is a recursive walk, so it is cached for _EXPENSIVE_TTL_SECS (300s) — five collection cycles on the default interval. One combined total, NOT a builtin/user split: builtins are copied onto disk into the same tree, so nothing in the listing distinguishes them and a split would be a guess. |
| kirocrew.inventory.memory.migrated | gauge | — | Same module; memory.migrated as 0/1. Reports the migration bit rather than a "memory enabled" switch because no such switch exists — the embedding provider is coerced to a real value on load, so memory is structurally always on and an "enabled" gauge could only read 1. The bit flips on the first boot that initialises vector memory, fresh installs included, so 1 is the healthy reading and a 0 on a live install is a fault: vector memory never initialised, migration aborted before the flip, or the flag write was skipped on an unparseable config.json. That is what keeps it from being a rollout metric that goes quiet once a fleet converges. |
| kirocrew.inventory.knowledge.documents | gauge | — | Same module; the raw source count. Deliberately NOT a constant 1 under a magnitude-band attribute: the quasi-identifying argument for banding does not hold when the only destination is the operator's own collector describing their own machines, while the band costs a series per band, hides growth inside a band (the drift the gauge exists for), and fixes the boundaries at emit time — a backend can band a raw count at query time and cannot recover a count from a band. Counts the sources table, not items (an item is one chunk, so counting items would report a chunking artifact as a document count), via a read-only sqlite3 URI guarded on exists(): constructing a KnowledgeStore would run schema init plus the writer-locked orphan migration and would CREATE the database on an install that never ingested anything. Cached 300s. Absent database yields a gap. |
| kirocrew.inventory.lessons | gauge | — | Same module; the raw lesson count of the global (V1) lesson store, same shape and rationale. Private Memory V2 crew-member lessons live only in each member's database and are not counted. Reads through one module-global LessonStore, whose own mtime cache makes a repeated read on an unchanged file a single stat — reused across ticks because that cache lives on the instance, and left uncached here because a second cache would add only staleness. An absent file is a genuine 0 (an install that never saved a lesson), unlike the knowledge database. |
| kirocrew.inventory.mcp.servers | gauge | class (first_party / third_party) | Same module; the roster merged from the agent config and _load_mcp_json_by_source, classified against mcp_discovery._MANAGED_SERVER_NAMES — reused rather than restated, since a second name list here would drift silently the first time a managed server is added or renamed. Server names are read to classify and then discarded: no name ever reaches an attribute. A roster is user-chosen free-form text, so publishing it would be both a cardinality bomb and a disclosure of what the user has installed. An empty merged roster yields a gap: the loaders fail soft to {}, so empty cannot be told apart from an unwritten config, and the managed servers are always present once installed. |
| kirocrew.inventory.config.toggle | gauge | key (closed enum, inventory_gauges.CONFIG_TOGGLES) | Same module; each declared feature switch as 0/1. Config is read ONCE for the whole set, so the gauge costs one fingerprint-cached load per collection instead of one per key. telemetry.enabled is deliberately absent — this module only runs inside the consent gate, so it could report nothing but 1. memory.migrated is absent because it has its own gauge. A key whose field cannot be resolved is OMITTED rather than reported as 0 (a renamed config field must read as a missing series, never as a switch someone turned off); test_every_declared_toggle_resolves_against_a_real_config is what keeps that from being a silent hole. |
| kirocrew.inventory.members | gauge | — | Same module; read_members, the count of crewmates the PERSON created on this install, reduced by members.count_person_created_crewmates. It walks the same rows GET /api/members builds (a dispatchable name with a resolvable slug) and counts the ones members.is_person_created_crewmate accepts, which is the roster's own dashboard_created rule (source == "kirocrew" AND a member id), so the gauge cannot drift from what the Crewmates page lists as made by the person. Mate itself (source == "builtin"), package-synced and app-made crews, legacy rows and the reserved default member are not counted: package-synced rows are the large majority of the raw roster on a busy host and would bury the number. Cost is one fingerprint-cached config load per collection, uncached like the config toggles. No name, slug or member id is published; the count rides in the value. An empty roster is a real 0. A config read that raises, or a half-read config (_base_unreadable or _overlay_unreadable), records a members probe failure and yields no observation: the loader keeps going on a file it cannot parse, so the crewmates in that file would otherwise drop out and publish a false decrease. One case it cannot separate: a file that parses with a non-object agents value, which the loader reads as empty without marking it degraded, so it reads 0 like an install with no crews. |
| kirocrew.inventory.probe.failures | gauge | probe (closed enum, inventory_gauges.ALL_PROBES) | Same module; times a probe raised, per probe. Absent on a healthy install — it publishes only once something has failed, so a healthy host adds no series, and roster assertions must exempt it (_HEALTHY_ABSENT in test_otlp_wire_e2e.py). It exists because absence has TWO readings that this family cannot afford to conflate: a missing series can mean the probe broke OR that the host stopped exporting, and distinguishing a drifted host from a dark host is the job these gauges were added to do. A broken probe therefore presents as data — the failure series is present and climbing, which a host that went dark cannot produce. The first failure per probe also logs at WARNING (a default install's journal keeps WARNING and above, so debug would be invisible on exactly the installs that need it); repeats drop to debug so a permanently broken probe cannot emit one WARNING per export interval for the process lifetime. An observable GAUGE whose reading is a process-lifetime total, declared in inventory_gauges.LIFETIME_TOTAL_METRICS (see the process CPU/GC rows): as an observable counter it was the last CUMULATIVE series exported, and one is enough to force every consumer into whole-hour stateful aggregation. |
All twenty registrations are wired in provider.py::_build_recorder (live path
only) and wrapped so a gauge failure can never disable telemetry as a whole. The
two families register in SEPARATE try blocks — they read entirely different
subsystems, and the process gauges must not go dark because, say, the skills tree
is unreadable. Each reader callback is individually guarded — a failing probe
yields a gap for that cycle, never an exporter error.
Because the inventory probes read subsystems rather than process counters, cost is
the binding constraint: a callback runs once per export interval per reader, so
an install with an OTLP destination configured enters them on two ticker threads
concurrently. Every probe is O(1) in-memory, a single small file parse, or
explicitly TTL-cached (_ttl_cached, whose produce call runs OUTSIDE its lock so a
second reader misses the cache rather than blocking on the first). A None
reading is cached like any other answer, so an install that can never answer does
not re-pay an expensive failing probe every cycle.
Scope of the kirocrew.inventory.* family, stated here because the names invite
the opposite reading: it is OPERATOR-fleet telemetry, not project-wide adoption
analytics. Collection is off by default, egress is a second opt-in, and OTLP needs
the kirocrew[otlp] extra, so any aggregate describes one operator's opted-in
machines. Adoption data structurally cannot ride this trunk — see "Why it is NOT
part of the OTEL trunk" below, whose four reasons apply to these instruments
exactly as they do to the beacon's payload. Do not later "promote" these to answer
install-population questions.
One publisher per install (mark_install_reporter). _build_recorder runs in
every process that touches telemetry, and an install runs several at once — the
gateway, gatewayd, spawned agents and apps (which is why the local exporter
shards per PID). That is correct for process-scoped instruments and WRONG for
install-scoped ones: each process would publish the same "this install has 14
crons" under its own resource, so an aggregate over installs counts one install
once per running process, and the bucketed gauges (whose shape is "publish 1, sum
by bucket") would make one install look like N. Deduplicating downstream would
work off the resource's install-scoped service.instance.id, but only as
query-time discipline every consumer has to remember (see below, where that trade
is what keeps this mechanism). So exactly one process publishes: the gateway
calls inventory_gauges.mark_install_reporter() during startup, and every callback
checks it at COLLECT time (not at registration), so it does not matter whether a
recorder was built before or after the claim. A process that never claims it
registers the same instruments and publishes nothing. The election is an EXPLICIT
call rather than inferred from, say, autonudge.get_instance() being non-None
(true only in the gateway today) — an inference stops holding the moment that
service's startup moves or becomes conditional, and the failure mode is inventory
silently disappearing.
That failure mode applies to the explicit call too, which is why the call SITE is
part of the contract: it sits in GatewayOrchestrator.run() at a point no branch
guards. A feature-flagged host does the same damage an inference would — the claim
first shipped inside _init_autonudge(), which returns early under
KIROCREW_AUTONUDGE=0, so one env var left no process publishing inventory at all.
Nothing reported it, either: the callbacks return before probing, so probe.failures
stays empty and the whole subsystem reads like a host that stopped exporting.
test_reporter_claim_is_unconditional parses the gateway and asserts the call is
reached from run() without passing through a conditional (try is allowed —
telemetry must never block boot).
The resource's persisted install-scoped id is the precondition for ever removing
this mechanism, and it is not by itself a replacement. An install CAN be
deduplicated downstream (max by (install) over its processes) — but that is
query-time discipline: the exported data would still contain one copy per process,
and a naively written fleet sum would read N times the truth. The election makes
the data correct at the point of emit, which no consumer has to remember. So
retiring it is a deliberate trade (fewer moving parts here, a rule every dashboard
must follow there), not something the identity settles on its own.
Scope of the guarantee: it elects one GATEWAY PROCESS, which is one install wherever
the product runs one gateway per data home. Two gateways sharing a data home on
different ports would both claim, because the dashboard's singleton is per port, not
per data home — a pre-existing property of that configuration that duplicates every
install-scoped fact, and one no flag can detect (a second gateway double-counts
whether or not it runs crons). A pod is not this case: it boots with its own
KIROCREW_HOME and copies no crons, sessions, or databases, so its inventory belongs
to a different install and publishing it is correct.
stats.py's in-memory singleton already counts adjacent ground: sessions_created,
subagents_spawned and its tool-approval trio, all consumed by one dashboard
system handler. Those tallies are NOT merged into the counters above, and the
counters are not derived from them. The singleton is a process-local count that
resets on restart and answers "what has this gateway done since it came up"; the
instruments above are time-series carrying attributes, survive a restart, and
answer "what is the rate, broken down by shape". Merging them would mean giving
the singleton attributes and durability it has no consumer for.
The hazard the split creates is that the names are near-identical while the
POPULATIONS differ, so the two numbers can legitimately disagree and be read as a
bug. kirocrew.approval.decisions is the sharpest case: it counts gate VERDICTS,
every consultation including the auto-approves and denies that never surface to a
person, whereas the singleton's approval trio accrues on the surface where a human
acted. kirocrew.subagent.spawned fires on the confirmed-start funnel after
admission, so it excludes rejected spawns a request-time tally would include.
Each row above therefore states which population its counter measures; a
comparison across the two systems is only meaningful once those definitions are
read.
Boundaries live in two metric-name → boundaries maps in
metrics/provider.py — _HISTOGRAM_BUCKETS_MS for millisecond instruments and
_HISTOGRAM_BUCKETS_BY_UNIT for everything else — merged by
histogram_bounds(), from which one View(instrument_name=…) is built per
instrument. Ten families, each sized to its instrument's measured range:
| Family | Range | Unit | Instruments |
|---|---|---|---|
_FAST_BUCKETS_MS |
0.5ms – 60s | ms | gateway.request, db.query, mcp.backend.acquire, skill.lazy_load, context.section, telegram.api, embed.queue_wait, embed.inference, mcp_gateway.liveness.escalated_latency_ms |
_STARTUP_BUCKETS_MS |
1ms – 60s | ms | session.startup, chat.first_token, mcp.lazy_load, gateway.boot |
_TURN_BUCKETS_MS |
1s – 1h | ms | turn.duration |
_WATCHDOG_IDLE_BUCKETS_MS |
1s – 4h | ms | watchdog.idle |
_TOOL_CALL_BUCKETS_MS |
0.5ms – 1h | ms | tool.call.duration |
_SESSION_BUCKETS_MS |
1s – 7d | ms | session.duration |
_CREDIT_BUCKETS |
0.025 – 2500 | credit | turn.credits |
_USD_BUCKETS |
0.001 – 100 | usd | turn.cost_usd |
_RSS_BUCKETS_BYTES |
32MiB – 16GiB | By | process.memory.rss_sampled |
_CPU_RATIO_BUCKETS |
0.001 – 1.25 | 1 | process.cpu.utilization |
_TOOL_CALL_BUCKETS_MS spans a sub-millisecond builtin read to a multi-minute
build in one instrument, so it keeps both ends: a _FAST ceiling would floor
every long shell command into the overflow bucket, and a _TURN floor would
collapse the entire builtin population onto boundary one. _SESSION_BUCKETS_MS
is the only family measured in hours and days — a speculative session removed
before its first turn lives seconds, while a dashboard tab left open across a
working week is ordinary — so it is densest from a minute to a few hours where
interactive sessions land, keeps sub-minute bounds because the short-lived
teardown paths (unclaimed, destroyed) are a real population, and carries a
7-day ceiling so a long-lived tab is not floored into overflow.
Why the unit map is separate. The dashboard's generic histogram aggregation
reports a statistic under *_ms keys unless the emitting module declares a
non-millisecond unit for that instrument, and the frontend formats those keys
with a millisecond suffix, so _HISTOGRAM_BUCKETS_MS membership is a claim that
the instrument really is milliseconds. A credit or a dollar amount registered
there would make both the claim and the rendered value wrong. The two billing
instruments are claimed BY NAME by _aggregate ahead of the generic branch and
reported inside the turn block via _amount_stats, under unit-neutral keys
(p50, p90, total, …) each carrying its own unit — claimed by name because
the turn block adds a per-model attribution split the generic surface has no
shape for, not for the unit.
The two sampled process histograms have no dedicated block to be claimed into, so
the generic branch handles their unit itself: it reads
events.NON_MS_HISTOGRAM_UNITS through _non_ms_histogram_units() and routes a
named instrument through _amount_stats instead of stats(). That mapping lives
with the emitter for the same reason LIFETIME_TOTAL_METRICS does — the module
that declares an instrument is the one that knows what its reading means — so no
unit is re-spelled by a reader. test_provider_bucket_views.py fails when an
entry in _HISTOGRAM_BUCKETS_BY_UNIT is missing from that mapping, which is what
keeps a new non-ms histogram from silently reaching the *_ms surface.
_amount_stats also rounds to six decimals rather than one, because a sub-cent
per-turn cost rounds to 0.0 at the duration family's precision.
Each amount also carries by_model: the same total split by the emitted model
attribute, as {model, count, total, mean} rows sorted by total descending. Without
it the attribute would be recorded by the emitters and then discarded at the one
place that reports the amount, leaving "which model spent this" unanswerable from
the API. Percentiles are deliberately NOT repeated per model — that would add a
bucket array per row to answer a distribution question nobody asks of a single
model — and the split is never truncated, the same rule (and reason) as the
cost_breakdown.by_model block below. A turn whose model was empty is counted
under unknown rather than dropped, so the rows always sum to the pooled total
beside them.
How the completeness guard finds instruments. An instrument missing from both
maps silently inherits OTEL's default 0–10000 boundaries.
test/metrics/test_provider_bucket_views.py does not detect histograms by a
.duration name suffix, which is a convention rather than a property: it would
miss kirocrew.embed.queue_wait / kirocrew.embed.inference (millisecond
histograms without the suffix) and by construction any non-duration one. The guard
walks the source by AST for histogram(...) calls, reading each instrument's name and its
unit= keyword, and asserts: every emitted histogram is registered in exactly
one map, every registered entry has an emit site, the ms map holds only
unit="ms" instruments, the unit map holds none, and no single name is emitted
with two different units.
Exponential histograms are not usable here yet. They would remove the
hand-tuned arrays, but the local JSONL shard is serialized with the SDK's
to_json(), and an exponential data point carries scale/zero_count/
positive instead of bucket_counts/explicit_bounds. The dashboard
aggregator classifies a point as a histogram by the presence of bucket_counts
and reads percentiles from explicit_bounds, so an exponential instrument would
be dropped from the payload entirely — a silent absence, which is worse than a
coarse bucket. Adopting them requires teaching the reader that shape first.
Why not one shared array. A single 1ms–60s array serving every histogram
through a catch-all View(instrument_type=Histogram) has a ceiling sized for
session startup, so a kirocrew.turn.duration sample (227589ms — an agent turn is a whole agent loop including tool
round-trips and any wait on an interactive approval) lands in the +Inf
overflow bucket. Since _pct_from_buckets can only report an overflow bucket's
LOWER bound, the aggregator returns p50 == p90 == 60000 — a ceiling artifact
presented as a real latency, while mean/max (exact, from sum/max) stay
correct beside it. p50 == p90 is the signature of this failure. Instruments in
this system span six orders of magnitude (sub-ms pooled acquires to multi-minute
agent turns), so one array must sacrifice either fast-end resolution or slow-end
truth.
Why there is no catch-all fallback. The OTEL SDK applies EVERY matching View, not the first. A per-instrument View plus a catch-all therefore publishes the same metric name twice with different bounds. The SDK offers no negation in View matching, so the catch-all had to be removed rather than narrowed.
Boundary generations never merge. Bounds are baked into each data point at
record time, so any boundary change makes the 14-day scan window straddle two
incompatible generations of one metric. handlers/telemetry.py::_Hist therefore
groups data points by their EXACT explicit_bounds and reports statistics from a
single group — the one holding the newest data point. Selection is by recency,
not volume: majority selection would let a stale generation keep winning while it
out-counted the new one, so right after a boundary change the old bounds would be
reported for up to the whole 14-day window — for turn.duration that means
continuing to serve the ceiling-pinned percentiles this grouping exists to remove,
while omitting the new samples. Recency makes the change take effect on the first
post-change sample; the reported population is then small but truthful, and
count says so. (count remains a tie-break for data points carrying no
time_unix_nano.) Outcome tallies
are grouped too — accumulated inside _Hist rather than alongside it, because
scoping only the buckets and count would leave the outcome breakdown summing
across generations: the page would show N turns beside an outcome bar totalling
more than N, and a fault_rate computed over a different population than the
latency next to it. other_generations (0 = a clean window; >0 = the window
straddles a boundary change and only the dominant generation is reported) and
total_count (samples across EVERY generation) are therefore returned by
_Hist.stats() itself, so they travel with every set of numbers they qualify —
the startup blocks, the turn block, each other histogram, and each
per-attribute split.
The dashboard renders the PAIR, not the generation count: it shows
"showing 1,134 of 2,926 samples" beside the affected figure. A generation count
is an internal unit a reader cannot convert into missing data, so "1 older
generation" left a truncated n=1134 unreconcilable against the 2837 hit
counter next to it, while the shown/total pair is directly comparable to both.
other_generations remains the structural fact (how many incompatible groups the
window holds) and stays in the response for diagnostics.
They are deliberately NOT pasted on by the response builder per block. That is
how it shipped, and the generic other instruments were never given the field:
the MCP acquire card reported one generation's n (1,154 of 2,926 real samples)
beside a full-window counter, with nothing anywhere saying a generation had been
dropped. A statistic and the caveat that makes it readable are one value, not
two.
This is load-bearing, not defensive: the historical shared array and
_TURN_BUCKETS_MS have the SAME bucket-count length, so a length-only check
would have merged them positionally — adding a pre-change sample from the old
+Inf bucket into the new +Inf bucket and reporting p90 = 3,600,000ms (one
hour), while a 5s sample landed in a 5-minute bucket. Grouping also keeps
count/sum/min/max consistent with the percentiles; accumulating those
across generations while only one generation's buckets survived would describe a
mean over one population and percentiles over another.
Completeness is therefore load-bearing. With no catch-all, a histogram
missing from the map silently falls back to OTEL's default 10s-ceiling
boundaries — reintroducing the same class of bug. test/metrics/ test_provider_bucket_views.py scans the source for kirocrew.*.duration metric
names and fails when one has no map entry (and when a map entry has no emitting
call site). It also pins the no-duplicate-streams property and asserts the
227589ms regression sample does not overflow. When adding a duration
histogram, add it to _HISTOGRAM_BUCKETS_MS.
The per-route latency label route_template is never the concrete request
path, query, id, or body — it is the aiohttp route TEMPLATE
(/api/items/{item_id}), whose {…} placeholders are constants baked into the
route table. The bounding is structural: collect_route_templates(app) snapshots
the finite set of registered templates once (lazily, on first request, after all
routes — including edition-contributed and post-middleware routes — are present),
and route_template() returns a value ONLY if it is a member of that frozen set;
anything else (an unmatched 404 aiohttp SystemRoute, or a template not in the
snapshot) collapses to the single sentinel __unknown__. Therefore the distinct
route_template label values are bounded by len(known_templates) + 1, a
constant fixed at startup that cannot grow with traffic. Combined with the fixed
method allowlist (≤ 8 values) and the fixed status_class domain (6 values),
total series are bounded by (len(known_templates) + 1) × 8 × 6. The test
test/metrics/test_gateway_http_metrics.py::test_bounded_cardinality_under_many_distinct_ids
proves this against real OTEL data points: 100 distinct ids yield exactly ONE
route_template value. Privacy: the only request-derived labels are
method / route_template / status_class — no prompt, content, token, path,
query, user id, or secret is ever recorded, and every string label still passes
the recorder's redact() guardrail.
dashboard/handlers/telemetry.py — GET /api/telemetry/startup scans the JSONL
shards over ONE reporting window shared by every block of the payload. The window
is ?days=N (rolling, ending now) or ?since=&until= (a fixed range in epoch
seconds; the panel sends local midnights, so until is the midnight after the
last picked day), resolved by usage.resolve_window. A request naming no window
(the panel's Default choice, where it sits until the reader picks) answers
exactly as before the range control existed: the OTEL and context blocks cover
the last 14 days (_WINDOW_DAYS) and the spend block the last 7
(SPEND_WINDOW_DAYS); the payload says so with window_default: true and
cost_window_days. A named window drives every block, and is CLAMPED, never refused: a rolling days to at
least one day, the start to
usage.MAX_WINDOW_DAYS (90) before now, the end to now rounded up to the minute
(so a range running to "today" keeps one until between 5s polls), an inverted
range swapped, and a collapsed one widened to the day before its end. The panel's
custom picker offers start days back to MAX_WINDOW_DAYS - 1 days before today,
so a picked range holds at most 90 whole dates, all inside the clamp. The
effective window is echoed as window_days, window_start/window_end (UTC
ISO-8601) and window_rolling, beside
metrics_retention_days (telemetry.retention_days, 0 disables age pruning;
telemetry.max_total_mb can still drop the oldest shards by size), which
the panel uses to say when OTEL figures cannot reach as far back as the window
and to title those cards with the span they do cover. The turn drill-down
(/api/usage/turns) is fetched over the echoed bounds, not the request, so it
lists exactly the turns its row summed.
Shard FILES are picked by their UTC day. With a named window each data point is
then kept only when its own time_unix_nano falls inside it (a point with no
timestamp keeps its shard's day); with no window every point in the picked
shards counts, exactly as on main. The cache keys on the window, that mode, plus
the shard fingerprint; a
rolling window keys on its length, so the moving start costs no cache miss and
the 30s TTL bounds its drift. Aggregation is offloaded via asyncio.to_thread;
it aggregates the startup histogram into p50/p90 split by
cold/warm (spawned attr) + outcome + daily series, the turn histogram into a
turn block (stats + outcome counts + fault_rate), and generically surfaces
every other kirocrew.* metric (other list) so new emit call-sites appear
without a handler change. Scalar (non-histogram) metrics in other are
classified by the SDK's own JSON markers — a Sum's data block carries
aggregation_temporality/is_monotonic, a Gauge's carries neither. DELTA sums
keep summing across cycles/PIDs; CUMULATIVE sums and the lifetime-total gauges
named in process_gauges.LIFETIME_TOTAL_METRICS (process CPU/GC, whose newest
sample is a running total rather than a state) buffer
samples per (PID, process-identity, attrs) stream and reduce them time-ordered
after the scan (shard iteration order is not chronological). Routing both shapes
through one structure is also what keeps a lifetime total continuous across the
shards written while those four were still observable counters. The identity half
of the key is the resource-level kirocrew.process.start_time token the
exporter stamps: a changed token for the same PID is a deterministic process
boundary, so a reused PID starts a fresh stream even when the new process's
first snapshot already exceeds the old maximum — the one shape value-based
detection cannot see — while an unchanged token across provider rebuilds
stitches the rebuild segments into one stream. Within an identity-keyed stream
a value below the running maximum is treated as shard garbage (one identity is
one OS process, whose counters are monotonic), never banked as a reset. Only a
STRING token counts as an identity: a corrupt shard carrying any other type
there reads as identity-less rather than minting a stream that mutes the
heuristic. Identity-less streams (legacy shards, or platforms without a
start-time read) reduce under the counter-RESET value heuristic — a snapshot
below the stream's own maximum marks a process boundary and banks the finished
segment, while re-emitted snapshots at/above the maximum are no-ops — so
provider rebuilds stay idempotent and pre-change shards keep their exact
totals. Either way the stream's first in-window sample is subtracted as a
baseline so a process older than the window reports only in-window activity,
never its lifetime total (stream total = banked + live segment - baseline; add
across streams). Non-finite
scalars (json's Infinity/NaN literals) are rejected per point via one shared
coercion helper; gauges emit kind: "gauge" with latest (the newest sample,
never a sum). Gauge samples are keyed per exporting shard PID so
concurrent processes (gateway + MCP daemons) never collapse into one series: a
single-PID window keeps the plain shape, a multi-PID window reports the newest
process's reading as latest and a pid=-keyed by_attr breakdown. Malformed
shard records degrade per-point (a garbage value skips that point, a garbage
timestamp sorts oldest) rather than failing the endpoint. Percentiles are interpolated from bucket counts (made
meaningful by the DELTA temporality + explicit-bucket View). Security: the
user-configurable telemetry.local_dir and each shard pass validate_file_path
(sensitive-path check) before any read. Cross-process: metrics are emitted by
the ACP/gateway processes, so reading the durable shards is the only correct
path (an in-memory reservoir in the dashboard process would never see them).
GET /api/telemetry/collection (api_collection_status) is the small
companion route behind the recording switch. It reports the effective enabled
state, the metrics directory (metrics_dir), whether an env var (env_pinned /
env_var) or a config.local.json overlay (overlay_override) pins the setting,
and whether an OTLP endpoint is configured (otlp_configured). It is deliberately
separate from /api/telemetry/startup, which parses every metric shard in the
window to aggregate percentiles — far too much work for a panel that only needs to
know whether a switch is on. otlp_configured reports only THAT an endpoint
exists: the endpoint string never leaves _telemetry_cfg(), because it can carry
credentials in userinfo or query parameters. It is what lets the panel disable the
enable direction rather than offering a write that comes back 409, while leaving
disable available — which is where an opt-out matters most. Nothing on this route
is an egress control, so unlike the beacon there is no governance ceiling to
report.
Both routes report the EFFECTIVE state, not the stored config flag.
_telemetry_cfg() resolves consent the way the collector does — env_pin() from
metrics/provider.py overrides telemetry.enabled — so enabled on
/api/telemetry/startup and /api/telemetry/collection cannot read "off" while
metrics are being written, or "on" while nothing is. The pin comes from the
provider rather than a second read of the env var here, because two resolutions are
two things to keep in sync and a control that disagrees with the collector about
what "on" means is worse than no control. env_pinned is what lets the panel's
switch disable itself instead of offering a write the collector ignores.
other histogram splits (_OTHER_SPLIT_ATTRS). An other histogram also
carries a splits map ("attr=value" -> the same stats shape) for a NAMED set
of low-cardinality attributes — currently warm only. This exists so one side of
a split can be reported alone: the dashboard's cold-spawn figure is
acquire.splits["warm=false"]. Splitting on every attribute present was
rejected because gateway.request.duration carries method+route, which would
grow one sub-histogram per endpoint and force an arbitrary truncation cap on the
payload; a named boolean keeps the split two entries wide with no cap.
Note that kirocrew.mcp.lazy_load.* is NOT the cold-spawn signal even though its
name suggests it. It is emitted only from the legacy pre-ensure_backend spawn
path, which modern stubs never take, so it records nothing on a current
deployment (0 data points across 47 shards / 12 days observed) while real cold
spawns are recorded on the acquire histogram under warm=false. The instrument
stays because that legacy path can still execute for an old stub; the dashboard
does not read it.
Startup phase gating. Only the end-to-end startup point (phase absent, as
on the claude path, or phase=total from the kiro path) feeds the startup
totals — count, cold/warm split, outcome, daily series, and the bucket
distribution. Per-phase points are aggregated separately into
startup.phases[] ({name, count, p50_ms, p90_ms, …}). Counting them as
startups multiplies the startup count by the number of phases and sums several
unrelated latency distributions into one set of buckets, which renders as a
spurious multi-modal "distribution".
Failures are reported as counts, not as success rates. Both outcome
instruments are fail-closed — AcpClient.ensure_ready and
AcpProvider._start_kiro_runtime (providers/acp_runtime_start.py) each default their outcome to error
and overwrite it with ready only on the success path, and _turn_outcome maps
a real stop_reason — so a zero here is a measurement rather than an instrument
that cannot report bad news. What a rate over these populations cannot do is
survive rounding: a window of ~1400 startups makes one failed startup 99.93%,
which renders as a flawless 100%. A rate had no reachable value between
"perfect" and a problem big enough to clear half a percent, and it saturated in
the direction that hides failure. A count has no such ceiling, so the page shows
the absolute number of non-ready startups and of non-ok turns. Where a rate
is still shown (turn fault rate, which answers a different question — faults per
unit of work), a non-zero value below the rounding threshold renders as <1%
and never in the success colour.
Both figures count everything outside the success value rather than naming the
failure values. Enumerating them (error + timeout) silently excluded any
third outcome — auth_required, and the unknown that shards predating the
attribute aggregate under — which put the displayed count and the rate beside it
on two different populations.
context block. The response also carries per-turn context-window
occupancy — {turns, p50_pct, p90_pct, max_pct, sessions[]} — sourced from the
per-turn token row store below, NOT from the OTEL shards: occupancy is a
per-session ratio and slot keys are unbounded-cardinality, which must not become
a metric label. sessions[] is EVERY session in the window ordered by peak
occupancy (bounded only by a payload backstop of 500), each reporting peak
plus the LATEST turn's identity (agent/model/surface) and absolute
used/window. Rows whose window is missing or zero are skipped rather than
defaulted. The block is null when no row carries the fields, and is served
independently of the telemetry.enabled switch because those rows are always
written (the panel therefore renders it even with OTEL export off).
The panel's source. website/src/pages/TelemetryPanel.tsx is the panel:
the window control, the tab choice, the off, empty and error states, and the
OTEL-shard figures (the health strip, the Latency and Startup tabs), which
format through its fmtMs. Beside it, telemetryPanelModel.ts holds the
payload types, the measured window (PanelWindowCtx) and usePersistedChoice;
TelemetryPanelBlocks.tsx the sortable DataTable, Sums and Histogram;
and TelemetryPanelConversations.tsx the Spend and Context tabs with the
formatters and colour thresholds only they read. Every other module imports
the panel, never an owner.
Separate from the OTEL histogram sink above (~/.kiro/crew/metrics/, DELTA
histograms for trends/alerting), the gateway also keeps a per-turn row store
for cost and context analytics: one JSON object per model-spending turn appended
to <data home>/usage/tokens/YYYY-MM-DD.jsonl (shards partitioned by the user's
local date). It is always on (not gated by telemetry.enabled) and never
egresses the host. Written by dashboard/handlers/usage.py::persist_token_record
(sync) / persist_token_record_async (the chat hot path — builds the record
on-loop, offloads the append via asyncio.to_thread); both are best-effort
(exceptions swallowed, no fsync). Aggregated for the dashboard by
_parse_token_history (30-day window, shard-fingerprint + 120s-TTL cache).
Each row (_build_token_record) carries:
| field | type | meaning |
|---|---|---|
_type |
str | always "tokens" (record discriminator) |
ts |
str | ISO-8601 local timestamp of the turn |
slot |
str | chat slot / session key |
provider |
str | LLM backend (acp / claude_code / bedrock / …), "" if unknown |
model |
str | resolved model id for the turn; "auto" when the completed turn only exposes the Auto request, or the backend reported auto as the resolved id (a session on the default model, whose _model stays ""); "" if no model source is available. A billed row persisted with a blank model logs a WARNING (_warn_unattributed_billing) so the path that lost the model can be found; the provider-wrapper walk that resolves it is bounded at _WRAPPER_CHAIN_MAX_NODES (64) |
input / output |
int | prompt / completion tokens (structurally 0 on the ACP backend — kiro-cli bills credits only) |
cache_create / cache_read |
int | cache-write / cache-read tokens |
cost |
float | provider-reported USD cost (0.0 on ACP) |
credits |
float | kiro-cli per-turn credit spend (float-coerced) |
turns |
int | model calls the turn made (TurnUsage.num_turns); on kiro-cli counted from its per-call _kiro.dev/metadata frames (see acp-client "Per-call context readings"), 0 when the provider reports none |
duration_ms |
int | provider-reported turn duration (0 on ACP) |
surface |
str | canonical dispatch/session source. Dashboard-backed turns derive the source from the effective session key through telemetry_channel_of (so linked Slack/Telegram sessions retain their transport); Task Runner writes taskrunner; other writers use their dispatch origin (cron, subagent, monitor, heartbeat, webhook, workflow, …). "" if unset |
agent |
str | agent id resolved for the turn; "" if unset |
context_used |
int | context-window tokens occupied after the turn (int-coerced) |
context_window |
int | served context-window size in tokens (int-coerced) |
context_sum |
int | the context-window occupancy after each of the turn's model calls, added up (TurnUsage.context_sum_tokens): the volume of context the turn's calls carried, cached or not. Measured per call on kiro-cli; 0 when the window is unknown or the provider reports no per-call reading. Not a billed token count |
stop_reason |
str | the turn's terminal stop reason read off the EVENT_COMPLETE event, or a synthetic reason on a dashboard turn that ended without one (_persist_abnormal_turn_usage: the caller's reason, else cancelled); "" when the producer has none, e.g. a bare TurnUsage from provider_last_turn_usage. Free-form is fine HERE (the row store has no cardinality limit, unlike OTel attrs) — this is where per-agent stall analysis happens: joining stop_reason (error: tool stall / stale_recover) against the row's agent field attributes watchdog outcomes to free-form agent names retroactively |
The surface / agent / context_used / context_window / context_sum fields and
stop_reason are all additive — every field defaults ("" / 0) so existing
callers stay valid and shards predating a field (which lack its key) remain
parseable; readers must tolerate their absence.
The row carries no injection breakdown. context_trace, the one reader of
ctx_blocks and phase, is served from the crew log's usage projection; the same composition is recorded by crew_log.emit .on_context_composed as context/composed, with the same block labels and the same
phase. Shards written earlier still carry both keys and stay parseable — nothing
reads them.
context_used / context_window
are read from the provider at the persist call site via
usage.read_context_tokens(source), which calls the provider's public
context_used_tokens() / context_window_tokens() accessors
(providers/base.py, implemented for ACP in providers/acp.py +
acp/session_provider.py) behind getattr guards and returns (0, 0) on any
missing accessor or exception — so non-ACP providers and test doubles record
zeros and the analytics helper never breaks the turn it measures. surface
lets channel-backed turns and background dispatch surfaces (cron/subagent/
monitor/heartbeat/webhook/taskrunner/workflow) retain their canonical source;
context_occupancy normalizes the historical task_runner spelling to
taskrunner when rows are read. Zero-token surfaces (cron script=/command=
modes, heartbeat maintenance ticks) never call a model and must not write a row.
Per-turn injection breakdown (context/composed). Recorded in the crew log
rather than on this row (see above). Its sources are
produced by context_blocks.split_blocks(prompt, user_chars=…), which attributes
the FINAL assembled prompt to the blocks that produced it by matching the bracket
markers the assembly emits ([CRITICAL RULES, [Memory, [Skills:],
[USER PROFILE], [UI LANGUAGE], [RESPONSE PREFERENCES, [CURRENT USER REQUEST,
[REPLY FORMAT RULES], …) rather than counting at each of the ~30 append sites.
Reading the OUTPUT means the attribution cannot drift from what was actually
sent. _MARKERS is deliberately kept in sync with EVERY opener the assembly can
emit — including the identity/session banners ([USER PROFILE], [UI LANGUAGE],
[CHANNEL], [INCOGNITO SESSION], [TEMPORARY SESSION], the cancelled-turn
preamble), generated request-prefix openers assembled inside build_message
before the current request ([THEME PERSONA], triggered [Skill: …] bodies,
and [REPLY FORMAT RULES]), and openers prepended AFTER build_message returns
(the re-injected [Previous chat history for this tab …], [Hook context] — in
both emitted spellings, with and without the colon — and the [System: …]
regenerate line): a marker absent from _MARKERS does NOT surface as its own
bucket, it
folds into the PRECEDING recognised block and mislabels those bytes, so the set
must stay complete.
Budget cuts beside the breakdown (context/composed.omitted). Block sizes say how
big each block was, not whether a budget dropped anything to reach that size, and the
omission notice inside the prompt is read by the model alone. Each budget decision
that writes such a notice also calls context_omissions.record with the numbers the
notice states: the lesson rule and findings tiers (render_lesson_tier, and
render_withheld_tier for findings withheld as off-topic), the pref.* preference
facts, the preferences file past the protected-content ceiling, the background
admission pool (admit_background), a member briefing at its character cap, and folder
steering (a collection ceiling records that the section is incomplete; the section's
character cap and the member essentials envelope record the documents not loaded
whole). _turn_build_prompt (chat_turn/prompt_assembly.py) opens context_omissions.collect()
around build_message, and _turn_dispatch_gates writes the collected record as omitted on
that turn's context/composed. The collector is a ContextVar, so two sessions composing against
one store never see each other's numbers, and the composer's worker thread inherits it through
copy_context. Outside a collection record is a no-op. The record is bounded by
construction: record admits one entry per kind in context_omissions.KINDS, each a
reason from REASONS and at most four integers or the closed set of background
section names, which is exactly what the entry declares; the crew log refuses an
undeclared member, and that refusal would also lose the block sizes. Omitted text,
keys and rules are never recorded.
Which lessons a turn carried (usage/injections/). The lesson tiers are ranked
against the request, trimmed to a budget and filtered by topic, so the set a turn
received cannot be rebuilt from the store afterwards. render_lesson_tier (and
render_withheld_tier) also calls injection_receipts.record with the memory keys of
the rows it KEPT and the omitted count; _turn_build_prompt opens
injection_receipts.collect() beside context_omissions.collect(), and
_turn_dispatch_gates, right after context/composed, appends one JSON line to
<data home>/usage/injections/YYYY-MM-DD.jsonl (local day, owner-only): ts,
session_key, turn_index (the crew log ordinal), user_msg_id (the meta.mid of the row that opened the turn, frozen on entry;
empty on a synthetic turn), model, agent, surface, phase,
memory_store, lessons, lessons_omitted, findings, findings_omitted, and
turn_lessons: the keys turn_lessons_block adds to a follow-up under
memory.inject_lessons_per_turn. Only an admitted turn whose composition rendered a
lesson block writes a line, so a per-turn build that added none writes nothing: the
model still holds what the session's earlier lines recorded. Incognito and temporary sessions write nothing. The write runs on a worker
thread (no suspension point in the gate) and never fails the turn. Legacy JSONL lesson
rows have no key: a kept one is not named, and a cut one still counts in *_omitted. Lesson
text is never written. Purpose: scoring which lessons change behaviour (#20446).
Current-request recency boundary. Injected blocks are not the only proof that a
turn is contextual. A keyed warm provider session (is_new_session=False) carries
native conversation history even when this turn emits no Kiro Crew context block,
and a cold session/load resume carries restored native history when
resumed=True. Both paths mint [REPLY FORMAT RULES] (when interactive) and the
[CURRENT USER REQUEST …] header before the current turn so the user's text owns
EOF. Only a standalone call with no session key, no restored/native history, and
no injected blocks preserves the legacy raw-user-text-first shape with guidance
trailing; it has no older provider topic from which to regress.
A block owns the span from its opener up to the earlier of the next opener and
its OWN closer (_CLOSERS, keyed by the same labels — only the closers the
assembly actually emits are listed, so extending it is a data change rather than a
scanner edit). A label with an opener in _MARKERS but no _CLOSERS entry keeps
the absorbing behaviour this table exists to remove, so the two are kept in step
by grepping the assembly for [End / [END rather than by adding a closer only
when one is noticed. One opener can have MORE than one closer spelling — a hook
context is closed [End of hook context] by context_assembly/turn_message.py and [End hook context]
by dashboard/chat_turn/turn_dispatch_gates.py — and an entry covering one spelling silently leaves the other
block absorbing what follows it, so such a label carries an alternation rather than
whichever spelling was found first. The closer taken is the LAST match before the next opener, not the
first: a block's content can quote its own closer — a custom agent prompt that
documents the envelope it is injected into embeds [END AGENT SYSTEM PROMPT]
verbatim — and first-match would end the block at that quotation and book the rest
of its real body as unclassified. Last-match is right by construction, because
nothing of the block follows its real closer, so any earlier occurrence in range is
content. The closer search is bounded by the next opener, which is what
keeps a WRAPPER correct: [SESSION CONTEXT closes long after the memory family
opens inside it, never finds its own closer in range, and ends where it always did
— unbounded, it would swallow every block nested within. The blank line a block
emits right after its closer stays with that block, so a closed block does not
leave a two-character crumb behind it. Escaping matters in one place worth naming:
[End of skill] is a PREFIX of [End of skills], so an unanchored closer would
let one loaded skill claim the whole skills index that follows it.
Characters that fall between a block's closer and the next opener therefore
surface as unclassified, as do leading bytes before the first marker. This is
the honest reading and it replaces a silent one: without closer awareness a span
ran all the way to the next opener, which turned unattributed bytes into
MIS-attributed ones with no way for a reader to tell a genuinely large block from
a small one that had absorbed its neighbours. The trigger is ordinary rather than
exotic — the blocks between two openers are conditional (workspace identity and the
docs pointer are skipped for a custom agent, the memory family for a session sealed
from the user's memory), so their absence is exactly what lets an earlier block
absorb everything downstream of it. Measured on one real session, a ~470-character
[USER PROFILE] block was reported as 8,116.
The returned sizes sum EXACTLY to len(prompt) (closure); the user's
own text is carved into the your_message label using the exact (start, end)
span that build_message reports through its user_span_out out-parameter. That
span is authoritative because build_message is the only code that sees every
transform applied to the turn: a HOOK_MODIFY transform hook can REWRITE it
wholesale, marker neutralization rewrites forgeable boundary markers (changing the
length of anything ahead of the user's text), and the return path folds
_MULTIBYTE_TABLE punctuation over the whole message (— → --, … → ...).
A caller measuring the pre-transform message cannot know the post-transform
offsets, so it passes the bounds of the user's text within the text it hands
over and build_message maps them forward. When a transform hook has replaced
the turn, the caller's bounds describe text that no longer exists, so the hook's
output is attributed to your_message in full — it IS the user's turn at that
point. Steps that PREPEND to the finished prompt after build_message returns
(the incognito/temporary notice, re-injected history, hook context, the
regenerate system line) slide that span, so the caller re-derives it once from the
length delta and verifies the shifted slice still holds the same text; if it
does not, the span is dropped and the reconstruction below is used rather than
persisting a wrong attribution. The legacy user_chars / user_offset reconstruction stays as the
fallback for callers that do not assemble through build_message.
Sizes are characters, not tokens — characters are exact, free
and tokenizer-independent, whereas the only tokenizer available here is OpenAI's
BPE, which would add a systematic unknown error against a Claude backend, and
comparing one turn against another (the whole point of the breakdown) wants an
exact unit rather than an approximate one. phase separates the one-off
session-start injection from the much smaller per-turn one so a reader never
pools the two populations.
Non-finite floats are rejected at the single write chokepoint.
_write_token_record dumps with allow_nan=False and, only on the resulting
ValueError, rewrites non-finite floats to 0.0 — so the common path pays no
per-turn scan. The guard lives there rather than per field because provider
floats (cost_usd, credits) reach the record unvalidated and a float is
exactly what a bad one looks like, so a type check cannot catch it. Without it,
json.dumps writes the bare tokens NaN / Infinity, which are not valid
JSON while json.loads still accepts them: one such row travels silently into a
web.json_response body no browser can parse, taking down every panel that
reads the store instead of losing one turn's numbers. cost_breakdown
additionally skips non-finite rows on read, because shards written before this
guard can already contain them.
Subagent turns are excluded from both readers. usage.is_session_slot(slot)
drops any row whose telemetry_channel_of is subagent, at the point the row is
READ rather than where it is grouped — so the window totals, the prior-period
deltas, by_model, by_channel, by_category, the context bands, the priciest
turn and the occupancy percentiles are all computed over one population and cannot
contradict each other. A subagent is a fragment of another session's turn rather
than a session with its own lifecycle, and its usage row carries no field pointing
back at the session that spawned it, so it can be neither listed nor attributed.
The consequence is deliberate and worth stating: these totals are the totals of the
sessions the panel lists, NOT of the account.
Read side. usage.context_occupancy(days) aggregates these rows into
per-turn occupancy percentiles plus a per-session peak ranking (own
shard-fingerprint + 30s-TTL cache, same contract as _parse_token_history), and
handlers/telemetry.py serves it as the context block of
GET /api/telemetry/startup (a plain module-scope import — handlers.usage
imports nothing from dashboard.handlers beyond its own usage_* owners, which
import nothing back, so there is no cycle to dodge).
usage.context_trace(slot, days) is the per-session drill-down, and it reads the
slot's usage PROJECTION rather than the shards. The fold's context part is one
fixed-size state that does not grow with the session (projection_usage_measured):
it never reads the log back, an entry that does not concern it returns the same
state object, and its worst case is bounded whatever the turn count: 31 full rows (the ring plus the start row), about 894 KB in memory, under the slot-fold cache's 995,342-byte member ceiling. What it renders, and so pushes, is slim: the newest turn whole and the chart's numbers for every other row, about 6 KB on a real session and at most about 20 KB. A past turn's whole panel is a separate read (below). The trace
returns what that state renders:
latest-- the newest turn's one breakdown: Crew's five running sums (ours: message, memory, rules, skills, other) since the engine's context last restarted (a new unit or an engine compaction restarts them, a compaction from the compacting turn's own prompt; acompaction/appliedthe gateway ran in place, which the engine did not measure, keeps only what was composed after the newest turn that closed with an occupancy reading, so a deferred verdict keeps the prompt composed before it arrived), the engine's own categories when it measured the turn (engine, from the newestcontext/measured, last-wins, with its percent, counts, splits, window and media), its time to first output (ttft_ms), and the occupancy paircontext_used/context_windowfrom the newestturn/completedreading. A new unit resets that reading, because the previous session's occupancy says nothing about this one. When the engine did not measure the turn,loop_charsis the occupancy reading atchars_per_tokenless Crew's sums, withloop_estimatedset.trend-- the newest 30 turns (TREND_TURNS, the panel'sMAX_CHART_TURNS), oldest first, each as the chart's numbers only: its ownturnordinal, Crew's five sums, the agent loop's part (loop, measured turns only: the chart shows no estimate), the turn's own occupancy reading in tokens (used, 0 with none) and the row'sversion. The state keeps, per row, what that turn's own panel needs: its Kiro Crew blocks by name (at mostBLOCKS_PER_TURN, 64, the composer's own most; a name underTEXT_LIMIT; the rest counted inblocks_omitted), a copy of its engine measurement (all 14 categories, the history split, list counts, the toolIO, workspace and memory splits, window, media and time to first output), and its own occupancy reading.GET /api/telemetry/context-trace?slot=<key>&turn=<ordinal>(usage.context_turn_detail,projection_usage_measured.context_turn) serves one kept row as a whole-turn view inlatest's shape (turn_detail), read from the in-memory fold state throughread_slot_state, never from the log:400 turn_invalidunlessturnis a positive ASCII integer,404 turn_not_keptonce the turn has left the ring, and the same dashboard-only gate and app-caller 404 as the trace. The panel reads it when a past turn is picked and caches it by slot, ordinal and that row'sversionin the trace (rowVersion), which the fold stamps from a write counter (row_writes) every time it writes the row, so a turn sent again is re-read even when its numbers are unchanged and a new turn elsewhere re-reads nothing; while a changed row is re-read, the previous read of the same ordinal stays on screen, disclosures open. A pick belongs to its session: the tab is remounted per slot, so switching sessions opens the new one on its newest turn. Each ring row stores the ordinal it was closed at, so a slot with several session starts labels every row correctly. A turn sent again (a retry, a regenerate, an edit-resend of an earlier message) keeps its ordinal, and every row from that ordinal on (ring and start row) is dropped because the conversation was cut back to it, so no two rows carry one number andturn_countis the real count. The ordinal is read from the ordinal memo (turn_memo: the current unit's turn numbers in ordinal order, bounded atCONTEXT_ORDINAL_MEMO_LIMIT, 10,000, the slot's live message cap, so it reaches any turn a rewind can), never computed from turn numbers, which count durable transcript rows; a rewind past the ring keeps its ordinal and cuts every ring row after it, and a refused turn (turn/refused) takes an ordinal too, since it holds its number in the conversation; its previous attempt's measurement, time to first output and engine turn id are dropped until the new attempt reports its own.start-- the newest session-start turn's row, kept out of the ring because the panel lists it above the chart, withfigure, the size its own panel heads with (the engine's characters when measured, else its occupancy reading in tokens).revision-- the fold revision the page was read at.earlier-- exactly how many turns precede the ring's first row, andturn_count, the one turn counter.
turns, totals, injected_chars and user_chars are built from that state for
the Session Breakdown tree, which reads them per sub-agent; each of turns is what
that kept turn sent, by block name, read from the state (context_turn_blocks), not
from the pushed value: {phase, blocks, total_chars, ordinal, first_start}. The state
keeps no per-turn time, model or reading for those rows, so they carry none, and the
page has no engine_unknown map. The occupancy pair peak_context_used / context_window is the
provider's own reading and the window it was measured against, both from the single
turn/completed closed in the last days that is the FULLEST: the highest used /
window ratio, which the tree turns into a fill ratio (context_peak). Fullest is
the ratio and not the largest used, because a model switch moves the window: 90k of
a 100k window is fuller than a later 200k of a 1M one. The comparison is integer-exact
by cross-multiplication, >= keeps the newest of equally full turns, and a reading
with no stated window is not a candidate. The state keeps the newest
PEAK_READINGS (200) dated readings for it (peaks), across units, the reach the
per-turn rows had; with none inside the span the pair is 0 and the newest non-zero
window request/configured stated in this unit (conf_window). days selects
nothing else and is echoed as window_days.
Block sizes are characters and the occupancy pair is tokens. A session recorded with
the crew log switched off reads empty -- this surface depends on that switch, which
is the one thing it is not independent of. Billing is not on this payload:
slot_turn_usage (below) is the per-turn reader for credits / duration_ms.
The engine measures a turn only when asked: every KAS session/prompt carries
contextBreakdown: "summary" and ttft: true in _meta.kiro (kas_profile.PROMPT_META).
Turns KAS starts on its own (its workflows) are not measured: Kiro Crew does not
enable KAS workflows today, and the session-level declaration
(automaticTurnProfiling) will be added when it does. Cron, sub-agent and task
turns open no crew-log turn, so a measurement written for one attaches nowhere and
never overwrites a parent's breakdown.
How the panel draws it. The chat Activity panel's Context Breakdown tab
(website/src/pages/ContextBreakdownPanel.tsx, ContextEngineBreakdown.tsx) shows the
picked turn first (the newest by default), as one breakdown in the style of a window: a headline, a bar, and
a flat list, zero rows hidden.
- Measured turn (
latest.enginepresent): the headline is the request's characters and, when the engine reported its window, about how many tokens that is by the window reading. The bar is the request at full width, each part at the engine's own percent. Row 1 is always Kiro Crew (rules, memory, skills, session details: every Crew block outside the other three, such as the date, workspace, user, folder, replayed history and reply format), a group that can be folded and opens by default; then Your messages; then, largest first, Messages (the engine's history less Crew's blocks: replies, reasoning, summary and "Other text sent as you") and every other category the engine counted, each with its own split where it sent one. Engine-side rows are named apart from Crew's own: "Engine system prompt" (not Crew's rules) and "Skill list (engine)" (not Crew's skills). A line under the bar gives the window share. - Turn not split (no
engine): every row is in tokens atchars_per_token, marked~. Kiro Crew and Your messages are Crew's own sums; "Rest of the context" is the occupancy reading less that estimate, never below 0 (a warning says so when the estimate passes the total). The bar is drawn against the whole window. An info tip on "Rest of the context" names the session's harness (thebackendofGET /api/chat/slots/{slot}/selection-capabilities, the route the model picker reads) and the place to get a split: Settings, the Agent Harness tab, the Kiro Agent choice, and says that new sessions then use Kiro Agent while this session keeps its current agent. On Kiro Agent the tip says the row is estimated because the engine has not measured the turn yet. A short tag beside the tip names which tip it is before any hover: "How to split it" on another harness, "Not measured yet" on the newest Kiro Agent turn, "Not measured" on a past one. When that route fails, its error is shown beside the trace's and the tip and its tag are left off, since the harness is not known. With no occupancy reading, the rows stay in characters and nothing else is claimed.
Below it, the trend: the start row ("Turn N · session start", its characters),
listed above the chart because its one-off size would pin a linear axis, then a
stacked-area chart of the trend rows (x = turn ordinals as the fold stored them,
y = characters; estimated tokens at chars_per_token instead when the panel is in
tokens, that is the newest turn is not split, has an occupancy reading, and no row
has a measured loop, so the chart and the rows above it share one unit), with a "Turns 1-N earlier" line from earlier that says when the
start row is one of them. Every column is a real button (arrow keys move the pick, aria-pressed marks it; below 12px a column, one plot-wide surface maps a click to the nearest turn), and the start row is a button too: picking one shows that turn's detail exactly as the newest turn is shown, the engine's own rows when the engine measured it and the estimated view otherwise (on Kiro Agent, "Estimated. The engine didn't measure this turn."), plus a disclosure listing the Kiro Crew blocks that turn sent, by name. The newest column shows latest itself. While a past turn is picked, a "Show latest turn (N)" button goes back to latest, and so do Escape and Home on a column: the newest turn may still be running and so not be a column yet, and without it nothing could undo a pick. Its bands are Crew's five categories and "Rest of the
context" (the measured loop; on a token chart, the row's own used less Crew's
estimated tokens, so each column reaches that turn's harness total; a turn with
no reading gets no such band and its column is drawn hatched, not as a drop, which
the caption then says), and the legend lists only the bands that are non-zero. When
the newest turn is not a column yet, a line under the chart says it is still running.
When the headline is in tokens but the chart counts characters (an unmeasured Kiro
Agent turn), a note under the headline says the unit changes. The blocks disclosure's
heading counts every block the turn sent and carries a chevron (right when closed,
down when open); the positive entries before the every-turn members are grouped into
one list row, plus blocks_omitted, whose line is styled as plain text. X-axis labels are strided from the measured plot width
(axisLabelIndices). Fills come from the --ctx-cat-* and --ctx-eng-* theme
tokens (website/src/styles/design-tokens.css). Grey (--ctx-eng-loop) is the
rest of the context in both the list and the chart, and the engine's own Other has
a pink of its own (--ctx-eng-other) wherever it is drawn, so the two never read
as one thing. An empty trace reads "not
recorded while the crew log is off" when the payload's recording is false, naming
the .env the gateway reads from the payload's env_file (sent only then), and the
"nothing recorded yet" copy otherwise.
handlers/telemetry.py::api_context_trace
serves it as GET /api/telemetry/context-trace?slot=<session key> (400 when
slot is missing or blank). It is independent of the telemetry.enabled switch but
dependent on the crew log it reads (empty while KIROCREW_CREW_LOG is off). The
endpoint is dashboard-only because, unlike /api/usage/turns, this reader has
no row-ownership model: an app caller is refused outright with the standard
indistinguishable 404 (code: not_found) and the refusal is SEL-audited
(deny-by-default, App Kit §5.2). An app that needs its own turns' billing has
/api/usage/turns.
Row-timestamp parsing has one owner. usage._parse_row_dt (in usage_store.py) is the single
spelling for reading a stored row timestamp (Z rewritten to +00:00 for
py3.10's fromisoformat; a naive stamp left naive so a caller's
.timestamp() reads it in local time); _parse_row_ts derives the epoch form,
and every shard/row reader in the usage handlers (slot_spend, context_occupancy,
cost_breakdown, slot_turn_usage, the token-history and transcript-day
readers) resolves timestamps through them. Two readers of the same rows must
not disagree about which rows a window contains. _usage_number is likewise
the one guard for copying a numeric field out of a row (bool is not a count;
ints are accepted directly because math.isfinite would overflow on an
oversized int; a non-finite float is dropped).
Per-turn usage rows for one session. usage.slot_turn_usage(slot, days) is
the per-turn drill-down under slot_spend's aggregate: one row per turn with
ts, model, and the numeric fields named by usage.TURN_USAGE_FIELDS
(tokens in/out, cache create/read, credits, cost, duration_ms, and the
context meter pair). A non-numeric or non-finite field is dropped from its row,
never the row itself. handlers/telemetry.py::api_usage_turns serves it as
GET /api/usage/turns?slot=<session key>[&days=N | &since=&until=] (400 on a
missing slot; the window is read by the same resolve_window as the panel, so the
Spend table's drill-down lists the turns of the range its row summed, defaults to
SPEND_WINDOW_DAYS, and clamps to MAX_WINDOW_DAYS rather than refusing; the
response is {slot, days, turns}, with days the clamped length and no
start/end echo, because the drill-down already sends the bounds it measured). This is the endpoint an app is
granted through its manifest's permissions.api to account for what its own
agent slots cost — apps otherwise have no path to credits, and the shard files'
location and row shape stay this module's private contract. App isolation is
ROW-level (App Kit §5.2, deny-by-default): each row is stamped with the
owning app at write time (_build_token_record's app field, threaded from
the turn's slot), and an app caller receives only rows stamped with its own
app — however the slot is named, and whether or not it is still live. A
live-slot ownership check was deliberately rejected: it leaks on slot-name
reuse (a recreated slot vouches for the previous owner's retained rows) and
denies an app its own completed sessions, which are exactly what an audit
reads. A foreign slot key answers 200 with no rows — indistinguishable from
a slot that never ran — and rows predating the stamp are invisible to app
callers. A disabled app is refused outright (is_app_enabled,
deny-by-default, the same gate the opt-in builtin routes wrap every handler
in): disable revokes read access, not only future writes. Every app-caller
decision is SEL-logged — including a malformed request's refusal — and SEL
plus the enablement probe run off-loop. The window is enforced per row,
not only per shard file (the oldest shard in a window covers a whole day);
a row whose timestamp cannot be parsed is excluded — accounting excludes
what it cannot date. Dashboard users (empty request app) read any slot; the
Telemetry panel's Spend table is the dashboard consumer — each session row
expands into its per-turn rows through this endpoint, which is where a
mid-session model switch or a single runaway turn becomes visible (an average
hides both). The Spend tab's first screen is a summary: KPI figures for the period (credits,
the change against the previous period, turns, per-turn cost, the priciest turn), then
bars for credits by origin, credits by model and the top sessions. The per-session
table sits under a Full table disclosure that starts closed and remembers its state
in telemetry:spend-table-open.
Stamping boundary: rows are stamped at the two write sites that can run
app-owned work — the dashboard chat runner (the slot's _app) and the
subagent completion path (info.app, an app-dispatched subagent's spend).
The task-runner, workflow, Slack and background-one-liner writers do not
stamp because those surfaces are not app-owned — an empty stamp there is the
correct value, not a gap. Webhook-session rows are currently unstamped and
therefore invisible to app callers; if webhook sessions gain app ownership,
that write site must stamp too. Same independence from the
telemetry.enabled switch as the context trace.
Where it renders. The breakdown is a per-session side-panel tab
(ViewKind context, opened from the panel's + menu directly under Logs), not
a section of the global Telemetry page. It is a developer surface, listed
in the + menu's diagnostics group with Logs. The tab is scoped to the chat slot it was
opened from, which is why it carries no session picker: a per-turn drill-down that
first asks the reader to choose a session cannot say anything until they do, and
Logs — the other "what actually happened in THIS session" view — already sets the
precedent. It never polls: the tab reads once on mount (refetchOnMount: 'always'), then re-reads once per pushed usage session_projection frame
(coalesced per slot, website/src/hooks/websocket/contextTraceRefresh.ts) and once on
a socket reconnect, because
the trace gains a row per turn and a tab left open would otherwise go stale. The Telemetry page keeps the aggregate
context occupancy block only.
Both readers stay OUT of the OTEL metric pipeline for the same reason: occupancy
and the per-turn breakdown are per-session, per-turn detail, and slot keys are
unbounded-cardinality labels that must never become metric labels (the
context_occupancy docstring states the same rule). The bounded half — block
label → size, aggregated — is what could belong on a metric; this per-session
half is the drill-down. Without these readers the fields were write-only:
recorded on every turn, read by nothing.
usage.cost_breakdown(days) is the second reader, same cache contract, serving
the cost block of the same endpoint. It answers "where did the credits go"
from fields the row store already carries — no new instrumentation:
| sub-block | derivation |
|---|---|
| totals | credits summed over the window, plus the preceding window of equal length and the delta between them |
by_model |
per-model credits, share, credits-per-turn, and per-model delta vs the prior window. Every model, never truncated — a top-N cut hides exactly the cheap-model-creep this block exists to show |
by_channel |
same shape, keyed by telemetry_channel_of(slot) |
context_bands |
mean credits per turn bucketed by absolute context_used, which is what makes the cost/context relationship legible (a turn at 900k costs ~4.7x one at 100k) |
conversations |
EVERY session in the window (not a top-N), each with its category, the unollapsed channel beneath it, peak occupancy, span, per-turn growth rate, and a projected turns-to-compaction. Named conversations for payload compatibility; the entity is a session |
by_category |
same shape as by_model, keyed by usage.session_category(slot) — the taxonomy the panel groups by: bg for an unattended session (cron, heartbeat, task runner), otherwise the transport (dashboard, telegram, slack, …) |
Channel comes from the slot key, not from surface. New writes now derive
surface from the effective session identity, but historical rows may contain a
wrong, non-empty value (for example a Telegram turn stamped dashboard). The
row format has no schema version or writer/trust marker that distinguishes those
legacy values from corrected writes. Cost/category attribution therefore keeps
the slot key authoritative; preferring a non-empty surface would silently
misattribute existing history. The stored slot/session key remains stable across
the write-side correction and is still the compatibility boundary for this
reader.
The dedicated _consolidate session retains the consolidation telemetry
channel and is grouped under bg in Spend, alongside _bg background work.
Two origin columns, deliberately, and they are labelled apart. The page shows
the session-origin dimension twice, derived two ways, because the two answer
different questions. Spend's Origin column (session_category, from the slot
key) answers WHICH SURFACE OWNS THE SESSION; the Context tab's Turn surface
column (the row's own surface field) answers WHICH CODE PATH RAN ONE TURN. So
the same unattended session reading all background on Spend and heartbeat on
Context is correct, not a disagreement, and the two column headers each carry a
tip naming their own derivation. The ruling that fixes this shape: a trusted
surface may NOT introduce Spend attribution values that do not exist today, and
monitor spend stays booked to the conversation it nudged. A monitor nudge exists
only to serve one conversation, so moving its credits into a separate monitor
bucket would make that conversation under-report what it actually cost — the one
number the Spend tab exists to give. The consequence is that the two vocabularies
are permanently different sets: monitor (slack/gateway_runtime/nudge_channel_fire.py) and webhook
(handlers/hooks.py) are written as row surfaces and are NOT members of
TELEMETRY_CHANNELS, and nothing in the read path compares one against the other.
Unifying them is therefore a product decision that has been made and declined, not
a cleanup: test/metrics/test_telemetry_column_taxonomy.py pins the two sets apart with
those two values as the known non-members, so a later unification has to confront
this ruling rather than discover it.
A conversation's title is attached by the endpoint, from two sources in
order. cost_breakdown names nothing — slot keys are all the row store holds.
handlers/telemetry._with_conversation_titles resolves the live slot's
display_title first, so a rename shows before it has been flushed; a session
with no live slot falls back to the title on its transcript's metadata line,
read through ConversationLog.get_metadata off the event loop and keyed by
slot_transcript_key(slot), which is what folds a channel-born slot onto the
channel's own transcript. Only an explicit metadata title counts: list_sessions
would answer with the first user message and then with the session key, which
turns a ranking label into prompt text and leaves no way to tell a named
conversation from an unnamed one. A row with neither source reports an absent
title rather than its key. Both sources pass the same two scanners on the way
out, so where a title came from cannot change what leaves the endpoint.
The growth slope is fitted per segment, never across the window. Occupancy
is a sawtooth: a compaction drops it back toward empty, and 7 of the 8
top-spending conversations measured carry one to four such drops. A secant from
the first turn to the last therefore crosses discontinuities and is dragged down
by every reset, understating the live rate by ~47x on real data — one 104-turn
conversation projected 795 turns of headroom while its current segment climbed
at 4%/turn, i.e. ~17 turns from compaction. Erring toward "plenty of room" is
the damaging direction for a figure whose only purpose is a warning, so the
slope is fitted on the stretch since the last fall larger than
_COMPACTION_DROP_PCT (sub-threshold drift is ordinary jitter in what counts
toward context_used, not a compaction). Growth and the projection are
withheld unless that CURRENT segment holds _COST_MIN_GROWTH_TURNS points —
a long conversation freshly past a compaction knows nothing about its new
trajectory, and withholding is the honest answer rather than extrapolating from
two or three points.
Tests: test/test_usage.py (TestReadContextTokens,
TestBuildTokenRecordContextFields,
TestPersistTokenRecord*), test/metrics/test_context_occupancy.py
(aggregation, skips, latest-turn wins, historical Task Runner spelling),
test/metrics/test_cost_breakdown.py
(channel attribution incl. the bare dashboard slot form, no-truncation,
prior-window deltas, band bucketing, post-compaction slope, growth
withholding, non-finite rejection), test/test_dashboard_chat_token_usage.py (effective
session source at the dashboard write site),
test/test_turn_duration_task_runner.py (canonical Task Runner writes), and
test/metrics/test_telemetry_titles.py
(title redaction + cache purity, and the closed-conversation fallback: the
canonical transcript key, live-wins-over-persisted, one read per shared
transcript, and no title where the metadata line names none), plus the
context-trace endpoint.
metrics/provider.py imports config.loader at module top; call sites reached
from inside config.loader's import chain (e.g. acp/client_start.py) MUST import
get_recorder lazily (inside the function) so the provider is never loaded
during that chain.
There are now five independent telemetry paths. Keep them straight:
| Path | Purpose | Data | Egress | Switch |
|---|---|---|---|---|
OTEL metrics (metrics/) |
Ops observability | DELTA histograms / counters | Never (local JSONL; OTLP only if the operator sets an endpoint) | telemetry.enabled (off) |
Token row store (usage/tokens/) |
Cost + context analytics | One row per model-spending turn | Never | always on |
Injection receipts (usage/injections/) |
Lesson precision scoring | One line per admitted turn that rendered a lesson tier: memory keys and counts, no text | Never | always on; skipped for incognito and temporary sessions |
Beacon (beacon.py) |
Product analytics | One anonymous ping per install per day | Yes — to the Kiro Crew endpoint | telemetry.beacon_enabled (on) |
Install receipt (apps/install_receipt.py) |
Official app adoption | One anonymous receipt after a successful official-catalog install/update | Yes — to the same endpoint | telemetry.beacon_enabled (on) |
src/kiro_crew/beacon.py, tests test/test_beacon.py. Fired from
slack/gateway_runtime/process_entry.py::run_gateway on a detached daemon thread (never awaited —
a 5s blocking urllib call must not delay boot or pin interpreter exit), and
skipped entirely under --test-mode so the offline E2E gate cannot egress.
Four independently disqualifying reasons — do not "consolidate" them later:
- The privacy guardrail eats the payload.
MetricsRecorder._guard()runs every attribute throughschema.redact(). A 64-char sha256 install id becomes"[REDACTED]"(40+-hex rule), so DAU would silently compute as 1. Ids that merely survive redaction are no better:schema.pymandates low-cardinality enum-like values and the instrument cache never evicts, so a per-machine id is precisely the "cardinality bomb" that contract prevents. - OTLP egress is an extra.
opentelemetry-exporter-otlp-proto-httpships inkirocrew[otlp], not the default dependency set, so a beacon riding it would measure only users who installed an optional extra. telemetry.enabledis a published no-egress promise (config help, this spec, the dashboard panel all say "nothing leaves this machine"). Hanging an outbound heartbeat off it would retroactively change what that switch means.- Shape mismatch. OTEL carries pre-aggregated data points; DAU needs one row per install per day deduped at query time. Aggregating away the id makes DAU uncomputable.
The one thing it does share is the atomic create-once file pattern from
handlers_system.py::_get_telemetry_salt (os.link, owner-only mode,
in-memory fallback).
The metric is DailyActiveInstances — deliberately not "DAU" and not
"Installations". The distinction matters because it is easy to misread the number:
- It measures activity, not installs. The
install_idis a persistent identity generated once at first run, not a per-install event counter. A copy that runs on ten days contributes to ten daily buckets with the same id;COUNT(DISTINCT install_id)per day therefore answers "how many copies ran today". Install volume is a separate metric (NewInstallations, from thefirst_seenbit, plus model-CDN first-fetch). - It is not DAU, because the denominator is not a person and cannot be. There is no account system — only a random per-data-home id — so one operator on three machines counts as 3, and three people sharing one machine count as 1. Calling it DAU would invite the reader to treat "14" as 14 people.
- The over-count is larger here than for a typical CLI: Kiro Crew supports
pods, worktree previews, and
KIROCREW_HOMEoverrides, so one person on one machine easily has several data homes. Dev homes and CI are suppressed, but a user's own pods are not.
Instance is the honest middle: it names a running thing (not the act of
installing) without claiming to have resolved it to a human.
payload() is the only producer; there is no caller-supplied field, so the
shape cannot be widened from a call site.
| Field | Example | Why |
|---|---|---|
id |
32-char hex | Random UUID4. Dedup key for COUNT(DISTINCT). |
v |
0.1.2 |
Version adoption. Release only — every build stamp is stripped by release(). |
py |
3.12 |
Minor only. Answers "when can the floor move off 3.10". |
dist |
dmg |
Which install path users take. Clamped to a fixed set. Baked at build time (see below). |
first_seen |
1/0 |
One bit → new-install and "launched once, never again" rate. |
chan (release channel), os, arch and gov (governance posture) were part of
this payload and are gone. Each was individually low-cardinality and individually
defensible; the problem was collective. The install id is stable, so every
attribute on the request is correlated with every other, and channel + OS +
architecture + governance posture together partition the population far more
finely than any one of them suggests — a nightly, ARM, governed-and-verified
install is a small crowd to hide in even though no single field is identifying.
So the rule for this payload is not "is this value low-cardinality?" but "how much smaller does this make the crowd this install hides in?" For all four the answer was "too much for what it bought":
chan— a nightly install is a small population by definition, which is exactly what makes it identifying when paired with a stable id. This one is a real capability loss, not a relocation:release()strips the prerelease label, so a nightly0.1.2-nightly.<stamp>and a stable0.1.2both sendv=0.1.2and are indistinguishable on the wire. Channel-split adoption is no longer answerable from the beacon. It remains observable from the release feeds / CDN fetch counts (feed/<channel>/latest-cli.json), which are per-artifact and carry no install id — a better source for that question anyway.os/arch— the platform mix is answerable from download/CDN telemetry, which is per-artifact and carries no install id at all: a strictly better source for the same question.gov— governance adoption is an enterprise-fleet question, and a fleet operator knows their own posture. Correlating it with a stable id bought a number nobody was blocked on.
The client is authoritative: beacon._fields() returns exactly four keys (plus
id) and test/test_beacon.py asserts the key set, so a re-addition fails a test
rather than shipping quietly. website/src/test/PrivacyPanel.test.tsx asserts the
four names are absent from the user-facing disclosure for the same reason.
Historical rows keep the old params. Log lines already in S3 carry
chan/os/arch/gov forever, and un-upgraded clients keep sending them until
they update. That is harmless — the aggregator simply no longer reads them, so the
fields age out of the analysis without a migration.
dist answers "which install path do users actually take", so it has to describe
the artifact, not the environment the artifact happens to run in.
Resolution order is baked module → env var → "source":
kiro_crew/_build_info.py, generated byscripts/stamp-distribution.shand written by each packaging path. Authoritative because it ships inside the artifact and a running install cannot change it. Imported once at module import intobeacon._BAKED_DISTRIBUTION(an optional-dependencytry/except ImportError, since the module exists only in a packaged artifact); that binding is also the seam tests patch, because writing a real file into the installed package is process-wide shared state that races under the default-n auto.KIROCREW_DISTRIBUTION, kept as a build/test override.DEFAULT_DISTRIBUTION("source"), the correct answer for a git checkout, where the module is absent (and gitignored, so it is never committed).
The env var is deliberately the weaker source. It is inherited by every child process and settable by anyone with a shell, so a stray export in a profile would relabel that host's daily count. An unknown baked value falls through to the env var rather than going on the wire, so a bad stamp can never emit an unclamped value.
| Path | Where it stamps | Value |
|---|---|---|
| Wheel | .github/workflows/build-wheel.yml, before python -m build |
wheel |
| macOS desktop | packaging/build-desktop.sh → build_backend |
dmg |
| Linux desktop | packaging/build-desktop.sh → build_backend (re-stamped per LINUX_TARGET_DISTS target) |
appimage, deb, rpm |
| Windows desktop | packaging/build-desktop.sh → build_backend |
nsis |
| Container | docker/Dockerfile, after the wheel install |
docker |
| Git checkout | nothing | source |
Three non-obvious constraints:
- The desktop stamp writes into the installed tree inside each backend
bundle, not the repo. A universal macOS build produces two backends and both
must carry it, and stamping
$ROOTwould leave the module in a developer's checkout. - The container re-stamps after installing the wheel. The wheel it installs
is already stamped
wheel, so without the overwrite every container reports the wheel channel. It overwrites the module rather than setting the env var, because the baked value outranks the env var by design. - The value is derived from the electron-builder target, not the host OS
(
mac.target: dmg, the Linux AppImage/deb/rpm targets,win.target: nsis). A Linux host also builds wheels, so branching on the host would mislabel them. The stamp is read by more than the beacon:platform/update_capability.pykeys the externally-managed classes on it, so a desktop artifact that carries the wrong value is offered the CLI release feed and the wheel installer command.
The three tests that execute stamp-distribution.sh skip when bash cannot run
it, probed by actually invoking the script rather than by shutil.which("bash"):
Windows resolves that name to a WSL launcher stub which then fails and cannot
see a Windows path. test_stamp_script_exists is asserted unconditionally so a
deleted script still fails on a host without bash.
test/test_beacon.py::TestDistributionStamp pins the precedence, the clamp, the
gitignore entry, and that the script accepts every KNOWN_DISTRIBUTIONS value,
so adding a channel to the frozenset without teaching the script fails a test
instead of failing at release time.
__version__ is not low-cardinality in the field. Dev and nightly builds
carry a per-build timestamp (0.1.2-nightly.20260731t065756,
0.1.2.dev20260731065756), so sending it raw both fingerprints and misleads:
- It fragmented the one number the field exists to produce. A real
57-install
0.1.2population reported as 35 plus a fringe of one-install series — which reads as adoption decay when nothing changed. - It was silently lossy. The aggregator caps each breakdown at
BREAKDOWN_LIMIT(25) per day. Past that, real low-install releases fall below the cut and vanish from both CloudWatch and the permanent rollup — and a long-tail release is exactly the one you most need to know is still running. The limit now queriesLIMIT+1and reports truncation (log line plus atruncatedkey in the invocation result) instead of dropping quietly. - It was a fingerprint. A near-unique per-build value combined with a stable install id picks out individual machines, which is precisely the correlated- attribute hazard the rest of the allowlist is designed to avoid.
The channel is discarded, not moved to a field of its own (see "Four fields
were REMOVED" above). beacon.channel() and its markers are deleted along with
it, so there is no dormant helper for a future call site to re-wire.
Normalized on both sides. release() clamps at the client, but a client only
stops sending raw stamps once it upgrades, and rows already in S3 keep them
forever. So the aggregator derives v from the raw v param in SQL
(_VERSION_EXPR) as well. That makes the metric correct for pre-clamp clients and
means re-aggregating a historical day yields the clean value. An unparseable stamp
becomes the single bounded bucket unknown, never a new metric.
Anti-fingerprinting is a design constraint, not a side effect. Every value is a low-cardinality constant or coarse bucket, because the id is stable and attributes on the same request are therefore correlated: enough precise attributes (exact patch level, CPU count, RAM, timezone, uptime) would combine into a quasi-unique identifier even though each is individually "anonymous". Do not add precise numeric or high-cardinality fields.
Never sent: prompts, model output, file contents, paths, repo names,
credentials, hostname, username, IP. Notably it does not reuse
handlers_system._get_owner_hash() — that is HMAC(salt, hostname + ":" + username); it stays local-only, and sending it would change its character.
No beacon failure may ever block, delay, or surface an error in a user action. A telemetry path that can fail a turn is worse than no telemetry, so this is enforced structurally, not by care:
- Boot is never delayed.
run_gatewaystartsbeacon.sendon a detached daemon thread and never joins or awaits it. Measured cost to the boot path with a beacon hanging 30s: 0.08 ms (the thread spawn).daemon=Truealso means it cannot pin interpreter exit.TestFailOpenpins both, plus a source assertion against a refactor toawait asyncio.to_thread(...), which would silently reintroduce up toHTTP_TIMEOUT_SECSof boot delay. - Every failure returns
Falsesilently. Verified across ten real modes: DNS failure, connection refused, TLS handshake failure, HTTP 500, HTTP 403 (corporate block), socket timeout, captive-portal garbage bytes, malformed URL, network-unreachable, and disk-full on the stamp write. - Both filesystem probes are inside the guard.
should_send()andpayload()touch the data home, so they run insidesend()'stry— an unwritable home raisesPermissionErrorfromconfig_dir(), and a container whose UID has no passwd entry makesPath.home()raiseRuntimeError(notOSError). Both are caught; the documented in-memory-id fallback is what engages. http.client.HTTPExceptionis named explicitly in the except tuple: it subclassesExceptiondirectly, so it is neitherOSErrornorValueError. Without it,http.client.InvalidURL/BadStatusLineescaped into the daemon thread andthreading.excepthookprinted a traceback to gateway stderr on every boot.- Even an unexpected exception class is contained — it dies inside the daemon thread and the main thread is unaffected.
status()never tracebacks. A diagnostic has to be readable precisely when something is broken, so its probes are guarded too.
The one thing deliberately not swallowed is a stamp-write failure's
consequence: _mark_sent() runs only after a delivered request, so a failed send
retries later rather than silently losing the day.
telemetry.beacon_enabled defaults true and gates the repo's only default-on
egress family: the heartbeat and official-app install receipts.
telemetry_permitted() suppresses both when KIROCREW_TELEMETRY_DISABLED is
truthy, an enterprise governance ceiling pins capabilities.telemetry off,
the config toggle is false, the process looks like CI, KIROCREW_HOME is
non-default (dev home / pod / worktree preview), or this install has never sent
and dashboard.privacy_acked is still false (the first-egress gate below).
beacon.should_send() adds the heartbeat-only daily throttle; receipts are
event-based and do not use it.
Each suppression carries a stable Verdict.code from beacon.REASONS alongside the
English reason, so the dashboard can translate the outcome instead of printing an
operator diagnostic.
The Settings toggle, the CLI and the env var are all operator controls: a user
on the machine can flip any of them, and so can the agent for the first two. A
managed fleet often may not egress to a vendor endpoint at all, which needs a
control the running app cannot undo. That is the capabilities.telemetry
SCOPE_CATALOG capability row (capability_default=True, so policy-absence keeps
the documented default-on behavior for standalone users).
It is read from the trust-root security_policy.json, which lives in
security._SENSITIVE_HOME_DIRS (defined in security/paths_lists.py). The agent cannot
rewrite its own ceiling: the sandbox mounts the file read-only, and it stays readable
on purpose because boot_platform() resolves the ceiling inside the sandbox; only an
enforced harness's child has it withheld (see security). That is what makes this enterprise-pinnable where a config.json field
would only be a suggestion. Authoring is a normal capability block:
{"version": 1, "boot": {"fail_closed": true},
"capabilities": {"telemetry": {"enabled": false}}}Enforced at four chokepoints — one send gate plus every write path to
telemetry.beacon_enabled, because any one alone is a half-control:
| Chokepoint | Behavior when pinned |
|---|---|
beacon.should_send() |
Refuses the send, reason disabled by governance policy (capabilities.telemetry) — ranked above the config flag so a managed host reports the policy, not the local value |
PATCH /api/config/kirocrew |
403 on telemetry.beacon_enabled=true; writing false is always allowed (tightest-wins — a narrower local choice composes with the ceiling) |
kirocrew telemetry enable |
Exits 1 without writing config.json; disable still works |
kirocrew config set [--local] telemetry.beacon_enabled true |
Exits 1 without writing. Easy to miss: the generic setter reaches the same key, and --local writes config.local.json, which takes precedence over the base file — so leaving it ungated would make it the one remaining way to store true on a pinned host |
Adding a fifth write path means adding a fifth gate. The rule is that no path
may leave a pinned host storing true; test_beacon.py::TestGenericConfigSetterIsGated
and test_config_patch.py pin the existing ones.
Without the write refusals a pinned host could sit storing beacon_enabled: true
behind a toggle that does nothing — the same false-promise-on-a-privacy-control
failure the overlay check already guards against.
Level-1 POLICY only. The probe requires layer == "policy", so a Level-2
profile pinning capabilities.telemetry does not suppress the beacon: the probe
runs from the boot thread with no session, and a bare not-permitted test would read
a transient deny-all-profile race as an administrator pin on an ungoverned host.
Full rationale in governance.md → "Anonymous telemetry".
beacon.is_governance_pinned_off() is the public probe, surfaced as
governance_override on GET /api/telemetry/beacon and as the strongest of the
three "pinned" notes in the Privacy panel (it outranks the env-var and overlay
notes, which would otherwise offer remedies the ceiling makes pointless).
It fails CLOSED (fail_closed=True), like capabilities.theme_install and
capabilities.publish. The two dispositions look symmetric and are not: a wrong
DENY loses one heartbeat, but a wrong PERMIT egresses from a fleet that
explicitly forbade egress — breaking the exact promise the administrator was
given, on a payload that leaves the machine. fail_closed also makes
governance_permits audit the degrade as a critical SEL event, which is precisely
the condition under which a silent degrade-to-permit would be indefensible.
The probe distinguishes two failure sources, because conflating them produces a different bug in each direction:
| Source | Identified by | Disposition |
|---|---|---|
| Evaluator could not answer (degrade, or the call/import itself died) | reason starts with GOVERNANCE_ERROR_REASON |
pinned — fail closed |
| Evaluator answered, but from Level 2 | layer == "profile" |
not a pin (see the policy-only note above) |
The second row is load-bearing: a transient deny-all-profile race on a host with no
policy at all arrives as an ordinary Decision, so no except can catch it, and
reading it as a pin would blame an administrator who does not exist.
TestGovernancePin pins all three outcomes.
The enforcement decision is SEL-audited; the probe is not. should_send passes
audit_tool="beacon_send", routing through vet_and_audit so a suppressed
heartbeat lands a governance_decision record. status() passes audit=False
because it backs GET /api/telemetry/beacon, which the Privacy panel refetches —
auditing an inspection would flood the trail.
is_default_home() compares against ~/.kiro/crew directly, never against
config_dir() — config_dir() honors KIROCREW_HOME, so comparing the two
always matches and the suppression would never fire (a real bug caught by
TestDefaultHomeDetection).
kirocrew telemetry status | disable | enable — status prints the exact
payload and never materializes an id (install_id(create=False)). Its numbered,
choose-one opt-out list leads with kirocrew telemetry disable because that choice
persists to config.json and survives a new shell. Each method is a separate visual
block: the environment override groups separately labelled macOS/Linux, PowerShell,
and Command Prompt syntax, followed by the equivalent config key.
beacon.is_env_opted_out() is the public probe for "the env var pins this off".
It exists so the dashboard can distinguish off because the stored flag is false
(a toggle can flip it) from off because the environment says so (a config write
would be accepted and then have no effect).
The GUI twin of kirocrew telemetry disable. It writes the same key —
telemetry.beacon_enabled via PATCH /api/config/kirocrew — so the two controls
cannot disagree and the choice survives restarts and upgrades.
- Only the boolean is dashboard-editable.
telemetry.beacon_endpointis deliberately absent from_EDITABLE_CONFIG: exposing it would let a dashboard caller redirect the heartbeat to an arbitrary host. GET /api/telemetry/beacon(handlers/telemetry.py::api_beacon_status) feeds the control. It returns the stored flag (enabled) and the effective verdict (would_send+reason), because the env var, a CI host, a non-default data home, or aconfig.local.jsonoverlay each suppress sending independently of the flag. A privacy control that reads "on" while something else silences the beacon — or "off" while it still sends — is a false promise, so the panel states which one is in force.env_overridereports specifically whetherKIROCREW_TELEMETRY_DISABLEDpins the state; when it does, the toggle is disabled rather than offering a write that cannot take effect.overlay_overridedoes the same for aconfig.local.jsonentry.config.local.jsondeep-merges overconfig.json— the file the toggle writes — so an entry there would let the switch snap back to the overlay's value after a successful save. The endpoint reports it and the panel disables the toggle and names the file (the same casekirocrew telemetry disabledetects and reports; seecli_commands._telemetry). The probe is best-effort: a missing, unreadable, or malformed overlay reports "not pinned", sinceenabledalready carries the authoritative effective value.- The handler routes
KiroCrewConfig.load()and the overlay probe throughasyncio.to_thread— both stat/read files, and this runs on the aiohttp event loop, where a synchronous read stalls every other request behind it. - The endpoint is read-only, never materializes an install id
(
status(create=False)), does not return the id itself, and fails toward off on an unreadable config — a diagnostic must not 500, and must never claim telemetry is on when that cannot be proven.
The dashboard discloses the default-on beacon without turning disclosure into a consent flow:
- The first-run Privacy chapter is the disclosure surface. There is no passive banner above the routed content: a dismissible strip competed with the chapter that already discloses the same thing, and its "Privacy details" link was the only thing it added.
- Settings → Privacy is the durable surface. It explains the
default-on, at-most-daily beacon and its exact five fields: random
installation id, app version, Python minor version, installation channel, and
first-run flag.
PrivacyPanel.test.tsxasserts both that all five are named and that the four removed fields are absent, so this copy cannot drift frombeacon.payload()in either direction. It also states the excluded data, carries the opt-out toggle (above), and keeps the status/disable commands beneath it — the CLI remains documented because it is the only way to override aconfig.local.jsonoverlay and the only control on a headless host. - The mandatory first-run Privacy chapter (
components/PrivacyChapter.tsx, chapter 2 of 3) shows the same disclosure and the same toggle in theOnboardingChapterShelllayout, so a new user sees what is sent and can opt out before reaching the product. Every path out of chapter 1 routes through it, including "Skip all", and it offers no skip and no Escape. It is still not a consent gate: Continue is always enabled and never requires a choice, because the default is a decision the user can change here, in Settings → Privacy, or from the CLI. - The first heartbeat is withheld until that chapter is acknowledged. The
gateway starts the beacon thread at boot, before the dashboard has rendered, so
an ungated fresh install would ping before the user could decline: an opt-out
offered only after the fact. Continue persists
dashboard.privacy_acked(andkirocrew telemetry enable|disablesets it too, for headless hosts), whichbeacon.telemetry_permittedreads. The gate is first-egress only (beacon.is_first_send()): an install that has already sent is past the disclosure, and keying every heartbeat on the flag would silence it permanently rather than once.dashboard.privacy_ackedfalls back todashboard.onboarded, so a user who finished first run before the chapter existed is never re-gated. It gates the install receipt too, via the shared consent ladder. - The disclosure copy and the control are single-sourced in
components/PrivacyDisclosure.tsx(PrivacyDisclosureSections,TelemetryToggle,PrivacyCommandList) and consumed by both surfaces, so the first-run explanation and the durable panel cannot drift. - The durable surface distinguishes local usage/context records from optional
performance metrics. Performance metrics are off by default and remain local
when enabled, with one explicit exception: they egress only when the operator
configures an OTLP endpoint. It also carries the recording switch over
telemetry.enabled— a second, independent control from the beacon toggle above — fed byGET /api/telemetry/collectionand disabled when an env var, aconfig.local.jsonoverlay, or a configured OTLP endpoint means the write cannot take effect (or would start egress). - The panel renders a stable
reason_code(beacon.REASONS), never the siblingreasonstring.reasonis untranslated operator prose (already sent today (2026-08-04)) kept for logs and bug reports; interpolating it put a developer diagnostic on screen in all 10 languages. An unrecognized code falls back to a generic translated note rather than rendering a raw key. - These surfaces add no tracking. They delay exactly one thing, the first heartbeat, until the disclosure has been shown, and otherwise only explain behavior and controls that already exist.
A successful install_from_registry call emits one best-effort receipt only when
the selected row came from the bundled or edition-contributed official catalog.
The row's _registry marker is the authoritative negative discriminator: a
user-configured external registry row is refused before dispatch. Local-path
installs and self-registration never call the sender. This provenance gate keeps
private/corporate app names on the host.
The sender mirrors the beacon's posture: the same endpoint, the factored shared
consent ladder (telemetry.beacon_enabled, KIROCREW_TELEMETRY_DISABLED, the
capabilities.telemetry ceiling, CI/test suppression, and non-default-home
suppression), a detached daemon thread, a 5-second timeout, and silent failure.
It intentionally does not share the beacon's daily throttle because each
successful app install is a separate event.
The exact GET route is:
/b/1/install/<app-slug>?t=<token>&k=<fresh|update>&v=<release>
| Location/field | Contract |
|---|---|
<app-slug> |
Public official-catalog identifier in the path. No custom-source or local app slug is eligible. |
t |
First 32 hex characters of HMAC-SHA256(key=receipt_secret, msg=b"app-install:" + app_slug), where receipt_secret is a 64-hex random secret generated on first use, stored owner-only as app_receipt_secret under the data home, and never transmitted anywhere — deliberately independent of the beacon install id, which the collector already holds from every heartbeat and could otherwise use to recompute tokens for public slugs and link one installation's receipts across apps. Deterministic for one installation and app, different across apps, and joinable neither into an installed-app profile nor to any heartbeat row. If the secret cannot be read or created, no receipt is sent. |
k |
Exactly fresh or update, derived from whether the app was installed before the successful call. Updates must not inflate adoption rank. |
v |
Kiro Crew release normalized by beacon.release(); build stamps are removed. |
test/test_install_receipt.py mocks the network and pins every suppression,
token, URL, provenance, success-only, and fresh/update property. Beacon tests
remain the regression contract for the shared gate's original behavior.
The telemetry-account owner must extend the existing CloudFront → S3 → Athena → Lambda aggregation outside this repository. The product-side contract is:
- Select only
/b/1/install/<slug>rows with an official catalog slug, a 32-character lowercase-hext,kin{fresh, update}, and a normalized releasev; retain the existing no-client-IP/no-User-Agent log projection. - For
k=fresh, publishAppInstallationsas the count of distinct(app_slug, t)pairs per UTC rollup window (equivalentlyCOUNT(DISTINCT t)grouped by app slug). Duplicate delivery or reinstall from the same data home must not inflate the app's count. - Keep
k=updatein a separate update series; never add it to install rank.vmay provide a bounded release breakdown, but is not part of uniqueness. - Feed only aggregate per-app results to catalog-ranking publication. Never expose or join raw tokens, and never join receipt rows to heartbeat ids.
The Athena query and aggregator Lambda live in the telemetry AWS account, so that implementation and deployment are an explicit infra-account task outside this repo and outside this PR.
A snapshot restored onto a second machine must not clone the id — two hosts sharing one would collapse into a single Daily Active Instance, and the copied stamp would suppress the new host's sends.
This is guaranteed by non-selection, NOT by a basename filter, and the
distinction matters: portability.EXPORT_EXCLUDE and
snapshot_components.NEVER_SNAPSHOT_FILES are matched by BASENAME over the staged
workspace/, plan_memory/ and skills/ trees, so registering a beacon
filename there would silently drop any user file that happens to share the
name — real data loss on restore, in exchange for nothing. Root-level export
copies a hard-coded allowlist (config.json, hooks.json, crons.json,
notifications.jsonl, project_dir, workspace_dir) and snapshot staging
copies an explicit per-component list (CORE_FILES); neither names a beacon
file, so the root paths are never selected in the first place.
TestSnapshotAndPortabilityRegistration pins the whole property in both
directions: the names are absent from both filters, absent from the export
allowlist and CORE_FILES, and a workspace file sharing either name survives.
All beacon state goes through _read_state(), never Path.read_text(), with
three guards:
- Regular files only (
lstat+S_ISREG).read_textFOLLOWS symlinks, so a link atbeacon_install_idpointing at/dev/zeromade the read allocate unboundedly until OOM — inside the gateway's beacon thread. The check also rejects FIFOs (whoseopen()blocks forever) and device nodes, without ever opening them. - Bounded length (
_MAX_STATE_BYTES, 4 KiB). Even a regular file can be enormous if a log is rotated onto the name. - Lenient decode (
errors="replace"). A strict decode raisesUnicodeDecodeError, which is aValueErrorand not anOSError, so it escaped the callers' handlers and killedkirocrew telemetry status.
Anything unreadable returns "", which every caller already treats as
absent/corrupt — so the id regenerates rather than merely not crashing. The
/dev/zero and FIFO tests use a real thread timeout, because the failure mode is
"never returns", which a plain assertion cannot catch.
Zero application code — the access log is the data product:
client ─GET /b/1/<id>?v&py&dist&first_seen─> CloudFront E1YM983XX3ASBM
│ CloudFront Function returns 204 at the edge
▼
standard logging v2 → s3://kirocrew-beacon-logs (PERMANENT, tiered)
▼
Athena kirocrew_analytics.beacon_logs (partition projection)
▼
kirocrew-beacon-aggregator Lambda (daily 00:20 UTC) writes BOTH:
├── CloudWatch KiroCrew/Product → dashboard (~15-month view)
└── kirocrew_analytics.beacon_daily → PERMANENT record
Metrics published. DailyActiveInstances, BeaconPings, NewInstallations,
ActiveByVersion, ActiveByPython, ActiveByDistribution, and ActiveByCountry.
ActiveByChannel / ActiveByOS / ActiveByArch were removed with their source
fields; their historical CloudWatch points and rollup rows are left in place (the
data is real for the days it covers — deleting it would be rewriting history to
match a current schema).
country is retained, and is the one field the client does not send. It is
derived at the CloudFront edge and is coarse (a 2-letter code); the IP it comes
from is never written to storage, since the log delivery does not select c-ip at
all. Dropping it would mean removing c-country from the delivery's
recordFields, which shifts every column in the TSV — the Glue table's six columns
are positional, so new log lines would silently misparse (the status = '204'
filter would match nothing) until the table was migrated. Keeping the least
identifying field in the set was the better trade than a schema migration on a live
pipeline.
CloudWatch cannot be the durable store. Metric data is retained for at most 15 months and expires on a rolling basis (1-min → 15 days, 5-min → 63 days, 1-hour → 455 days), and that ceiling is not configurable. So the dashboard is inherently a ~15-month window, by AWS design rather than by our choice.
The permanent record is therefore two things in S3:
- Raw logs —
s3://kirocrew-beacon-logs. The lifecycle policy has noExpirationon any rule; objects only transition (Standard → Standard-IA at 90d → Glacier Instant Retrieval at 365d) to cut cost. Glacier Flexible Retrieval / Deep Archive are deliberately avoided — they require an async restore before a read, which would silently break the long-range Athena queries this design exists to support. Versioning is on; only noncurrent versions are pruned (365d). - Daily rollup —
kirocrew_analytics.beacon_daily(Parquet, stays in Standard forever). One small row per(day, metric, dimension, value). This is what makes "permanent" useful: raw logs grow linearly forever, so a multi-year dashboard query would scan every line ever written, while the rollup keeps such queries fast and cheap.
_persist_rollup() is idempotent — it deletes the target day's rows before
inserting, so a backfill or a retry after a partial failure cannot double-count.
It runs last in the handler, after the CloudWatch puts, because CloudWatch is
best-effort presentation while the rollup is the durable store: a rollup failure
must surface as a Lambda invocation error (visible on the dashboard's error
widget) rather than being masked by an otherwise-successful metric write.
Destructive-rewrite guard (do not remove). That same idempotent delete makes
an empty query result destructive: it rewrites a day to nothing. A if not facts check is not sufficient, because DailyActiveInstances, BeaconPings
and NewInstallations are appended unconditionally — as zeros — so an empty
day still reaches the delete with three all-zero rows.
A run that queries a partition whose logs do not exist reports SUCCEEDED with
no error and publishes zeros, so without a guard it would wipe the rollup to zero
rows — the durable record destroyed by a "successful" invocation. The guard skips
the rewrite unless at least one fact is non-zero (an all-zero day
carries no information, so skipping is lossless), and an empty partition logs an
explicit WARNING naming the partition, because a silent success is what makes
the failure hard to see.
Object Lock is deliberately NOT enabled. It would make the logs literally undeletable, which sounds like "permanent" but is the wrong trade for a privacy-sensitive dataset: it would also remove our own ability to purge after an operator mistake, a schema error, or a future deletion obligation. The goal is "retained indefinitely by policy", not "physically impossible to delete".
The aggregator cannot delete the permanent record. Its IAM policy grants
s3:PutObject/s3:DeleteObject on kirocrew-beacon-logs/rollup/* only —
raw-log access is read-only, so the component that consumes the history has no
permission to destroy it.
No client IP is ever stored. The log delivery's recordFields selects only
date, time, cs-uri-stem, cs-uri-query, c-country, sc-status —
c-ip, x-forwarded-for, User-Agent and Cookie are simply not delivered.
Verified against a real delivered log file. This is why the design uses a
Lambda-free CDN log path rather than CDN logging with default fields, and it is
what makes the "no IP" claim structural rather than a promise.
Aggregator timestamp rule (load-bearing): metrics are stamped at the
end of the target day, clamped to now - 1min. CloudWatch accepts
timestamps up to two weeks old but points 24h+ old can take 48 hours to
become queryable, while <3h old are near-immediate. Stamping midnight-of-
yesterday from an 02:30 run put every point in the 48-hour bucket and the
dashboard rendered empty despite the data being accepted; the clamp is
needed because a same-day backfill's 23:59 is in the future and
PutMetricData rejects >2h ahead. Both failure modes were hit in development.
Model CDN. embeddings_settings.py::_DEFAULT_MODEL_URL points at
kirocrew-models (distribution E2UX23B48LKM6V, OAC-only bucket access). The
_GGUF_SHA256 pin remains the sole integrity gate, so a tampered CDN object can
only fail verification. Because _ensure_downloaded() returns early when
model_ready(), a CDN request is a first-install signal, not a DAU signal —
the dashboard shows it as "downloads", deliberately separate from DAI.
context_blocks.measure_prompt reuses the existing marker classifier and an
authoritative user span to report exact characters and UTF-8 bytes at the
crew_assembly boundary. ContextBuilder.build_message emits both through the
existing recorder as kirocrew.context.block.chars and .bytes, with bounded
section, domain, boundary and lifecycle attributes. Lifecycles are fresh,
warm, resume, reinjection and minimal; they describe composition, not proof of
provider receipt or retention. Domains distinguish request, contract, replay,
following-interaction guidance and background. Unknown marker gaps remain visible.
No body, source path, user identity, token conversion or cost is recorded.
The reading explicitly reports native=UNKNOWN and external_mcp=UNKNOWN.
Provider-owned history, resources, native prompts and external MCP serialization
cannot be measured from this string. Dashboard prefixes added after this boundary
remain covered by the existing final-prompt character classifier, not by a claim
that the assembly reading measured them. These extents are not full model input.
Synthetic Unicode fixtures assert both closure sums and request attribution.
ContextBuilder.build_message skips measure_prompt when both its metrics
recorder and DEBUG logging are disabled. Attribution depends on the caller's
accurate user_text_range: Slack passes the actual user's slice after adding a
cancelled-turn restore prefix, so that prefix remains context rather than request
text. Character and UTF-8 byte totals describe Crew assembly only; native provider
inputs, external MCP schemas and actual model calling behavior remain unmeasured.
These instruments answer what Kiro Crew is used for. Every attribute value is a
closed list defined in code and pinned by a test; an unknown value is other.
None carries an id, a name the operator chose, a path, a URL, a model id or text.
Shared attributes (metrics/usage_attrs.py):
| Attribute | Values | Source |
|---|---|---|
trigger_source |
user, cron, heartbeat, monitor, subagent, workflow, webhook, app, agent, system, other |
stated by the code that starts the work; the dashboard maps its turn actor, and user needs a person's send that is not a restored queue row. A caller that states nothing records other, never user |
origin |
the 11 channel namespaces, dashboard, cli, webhook, api, internal, other |
a rollup of telemetry_channel_of; a webhook names itself |
session_kind |
chat, channel, cron, subagent, workflow_step, task_runner, heartbeat, background, other |
a rollup of session_source |
| Metric | Type | Attributes | Recorded at |
|---|---|---|---|
kirocrew.turn.completed |
counter | outcome, trigger_source, origin, harness |
inside emit_turn_duration, so it counts the same turns as kirocrew.turn.duration with the same outcome (metrics/turn_completed.py) |
kirocrew.turn.completed.duration |
histogram (ms, turn buckets) | outcome, trigger_source, harness |
same; only for a positive duration. Not in PRODUCT_METRICS yet: the allowlist takes counters and gauges only, and kirocrew.turn.duration carries open model and provider attributes that a destination cannot narrow away, since it filters by name |
kirocrew.session.started |
counter | adds session_kind, trigger_source |
record_session_started; a key whose kind fixes the opener uses it, else the caller's or the running turn's trigger |
kirocrew.subagent.completed |
counter | adds result: text / empty / none |
the terminal report, past the finalize claim |
kirocrew.autonudge.loops.stopped |
counter | adds ended_by: agent_stop / user_stop / budget / subject_done / stalled / error / other |
the store's commit point; a fixed grouping of reason |
kirocrew.memory.recalled |
counter | path: tool / context; hit |
an explicit recall after its receipt commits; each admitted turn's composition |
kirocrew.lesson.injected |
counter | hit |
each admitted turn composed with a lesson tier |
kirocrew.skill.lazy_load.count |
counter | adds skill: a shipped skill id (nested ones by their path), custom or other |
each on-demand skill body load; the duration histogram keeps hit alone |
kirocrew.app.task.completed |
counter | app: a shipped built-in app id (BUILTIN_NAMES, its apps/builtins/ directory name) or private; outcome: ok / error / other |
a finished Papyrus compile (record_app_task) |
kirocrew.channel.message.received |
counter | channel, admitted, automation |
run_channel_turn, once admission decides |
kirocrew.ui.control.clicked |
counter | control (8 values) |
POST /api/telemetry/ui-control, owner only, from the dashboard's option chips, question card and follow-up card |
Recording is an in-memory add after the measured action commits, and a recording failure never changes that action's result.