Skip to content

Latest commit

 

History

History
213 lines (156 loc) · 22.1 KB

heartbeat.md

File metadata and controls

213 lines (156 loc) · 22.1 KB

Heartbeat Module

Overview

The heartbeat service (kiro_crew/heartbeat.py) runs periodic background tasks on a configurable interval (default 60s).

Responsibilities

  1. Task processing — reads ~/.kiro/crew/workspace/HEARTBEAT.md, sends non-empty tasks to the agent
  2. FTS index rebuild — every _FTS_REBUILD_TICKS ticks (~15 min at the default interval)
  3. Daily retention prune — every _PRUNE_TICKS ticks (~24h at the default interval): prune_history(keep_days=memory.history_max_days) plus sel().prune(). Both run on the maintenance executor, never on the event loop, and a failed SEL prune increments the kirocrew.sel.prune_failed.count counter rather than aborting the beat.
  4. Idle-session consolidation — HistoryConsolidator.check_idle_sessions() on every tick, not on a multiple
  5. Rotating memory backup — scheduled on the first beat and then every _MEMORY_BACKUP_TICKS (1440) ticks at offset _MEMORY_BACKUP_OFFSET (30), so it never lands on the prune tick; it copies every active memory store on the maintenance executor and runs only when memory.backup_enabled is on, keeping memory.backup_keep copies

While require_memory_ready() raises MemoryStartupUnavailable (Global memory awaiting owner recovery), a beat runs only idle consolidation (and the backup schedule above) and returns: no task processing, no FTS rebuild, no prune.

HEARTBEAT.md Format

# Heartbeat Tasks

<!-- Add tasks below (one per line). KiroCrew picks them up on next heartbeat. -->
- check my pipeline status
- summarize open PRs
  • One task per line (no multiline support)
  • Lines starting with #, empty lines, and HTML comments (<!-- -->) are ignored
  • A closed comment (ended by --> or --!>, as in HTML) hides only what is inside it, even across lines; text before <!-- or after --> on the same line is still a task
  • A deliver tag inside a comment is part of that comment, so commenting out a task together with its tag keeps it hidden
  • A <!-- that is never closed hides only the rest of its own line, and logs a warning naming that line. It never hides later lines. This is a choice of "run them and warn" over "skip them" (how HTML reads it): a stray opener must not silently stop every task below it
  • A leading list bullet (-, *, +, 1., 1)) and then a leading checkbox ([ ], [x], [X]) are stripped from the task text the agent gets; each is a marker only when whitespace or the end of the line follows it
  • A pass rewrites only the lines that held a finished task (_rewrite_heartbeat_locked). Every other line (headings, comment notes, commented-out tasks, kept tasks, line endings) is written back byte for byte and in order, so a file the heartbeat wrote itself round-trips unchanged. Dropping a finished task's line keeps any part of a multi-line comment that line opened or closed, and never takes the newline off the line before it
  • append_heartbeat_task starts a new line first when the file's last line has no newline, so an entry never glues onto that line
  • File is auto-created on first start with an empty template

Task Lifecycle

  1. Tick fires → read file → extract tasks
  2. For each task: call on_task callback (gateway sends through ACP, posts result)
  3. Callback returns response text; heartbeat checks for HEARTBEAT_KEEP sentinel
  4. Tasks with HEARTBEAT_KEEP in response are retained for next tick (incomplete)
  5. Tasks without the sentinel are removed (complete)
  6. Tasks that raise exceptions are retained automatically (retry)

Task Retention (HEARTBEAT_KEEP)

The agent can include HEARTBEAT_KEEP anywhere in its response to signal incomplete:

Progress: 3/5 files processed. HEARTBEAT_KEEP
  • _should_keep(result) checks for the sentinel (case-insensitive)
  • None return (legacy) treated as complete (removed)
  • Sentinel stripped from display text before posting
  • Deliver tags (<!-- deliver:slack:<channel_id> -->) preserved on retention

Concurrency

  • Task processing runs as its own tracked task (_process_task), started by the tick and never awaited by it, so a turn that runs up to HEARTBEAT_TASK_TIMEOUT_SECS does not hold the FTS rebuild, the daily prune or idle consolidation; those keep their tick cadence
  • A tick that finds the previous pass still running (_process_task not done, or _processing set) skips the file, so passes never overlap and no task is double-processed
  • A failing pass is logged (Heartbeat task processing failed) by the task itself; it never reaches the tick
  • stop() cancels the loop, the running pass and the backup pass; wait_closed() awaits them (bounded by _STOP_DRAIN_SECS), so a cancelled turn unwinds its session release and cycle-end recycle before gateway shutdown moves on. A cancelled pass rewrites nothing: its tasks stay in HEARTBEAT.md
  • Uses dedicated heartbeat:task session key — no interference with user or cron sessions

Per-Task Timeout (Unattended Turn Bound)

A heartbeat turn runs without a human present. The _heartbeat_task callback in slack/gateway_runtime/heartbeat_turns.py wraps stream_and_collect in asyncio.wait_for(..., timeout=HEARTBEAT_TASK_TIMEOUT_SECS) (1800s / 30 min, mirroring cron's _JOB_TIMEOUT_SECS). This is the analogue of cron's _execute_with_timeout.

Without it, if the agent calls a non-allowlisted tool, the interactive-approval callback would block on the human-approval wait with no human present — wedging HeartbeatService._processing=True so no HEARTBEAT.md task would ever run again.

On asyncio.TimeoutError:

  1. Reset the heartbeat session (sessions.reset(HEARTBEAT_KEY)) BEFORE the finally releases it — kills the lingering harness turn/process (whatever backend serves the session) so it does not outlive the timeout. A failing reset is logged and swallowed. This is safe because asyncio.wait_for has already cancelled the in-flight stream_and_collect, and any concurrent sibling task in the same cycle is blocked on the per-key semaphore (held until our finally releases) — they pick up the freshly-recreated session.
  2. Log a warning.
  3. Return a graceful incomplete result string (the loop is NOT crashed).
  4. The finally block calls release(HEARTBEAT_KEY) ONLY — no per-task reset. Cycle-end recycle is owned by SessionManager.recycle_heartbeat, called once via HeartbeatService.on_cycle_end after asyncio.gather completes.

Background approvals additionally deny-fast (see security / slack-gateway specs), so the common "non-allowlisted tool" case resolves in minutes; this hard timeout is the backstop for any other long-running turn.

Cycle-End Recycle

Heartbeat runs all tasks for a cycle through asyncio.gather so the wall clock stays bounded. Concurrent tasks share HEARTBEAT_KEY, so a per-task reset(HEARTBEAT_KEY) in finally would tear down the session a sibling task is still using (the per-key semaphore guarantees serialization, but resetting inside the critical section means the next holder receives a torn-down provider).

So the per-task finally only releases the semaphore. Cycle-end recycle is SessionManager.recycle_heartbeat, invoked once via HeartbeatService(on_cycle_end=...) after gather completes. It tears the heartbeat session down at the end of EVERY cycle that has one; the context percentage is only logged. The next cycle's first task starts a fresh session.

Gateway Wiring

HeartbeatService is started in slack/gateway_runtime/heartbeat_turns.py after cron service:

  • on_task callback: prepends a HEARTBEAT_KEEP reminder to the task text, opens a session under HEARTBEAT_KEY with agent="kirocrew-heartbeat", streams the response (gated by the heartbeat allowlist in _heartbeat_approval), posts the result, and releases the per-key semaphore
  • on_cycle_end callback: invokes SessionManager.recycle_heartbeat once after asyncio.gather completes — tears the session down unconditionally
  • Callback re-raises exceptions so heartbeat can track failures
  • Stopped during gateway shutdown: exit_path calls stop() and then awaits wait_closed()

Session Identity

Heartbeat runs in its own session (HEARTBEAT_KEY = "_hb" in session_parts/policy.py), distinct from the shared BACKGROUND_KEY = "_bg" used by cron / consolidator / chat-title. The session uses the dedicated kirocrew-heartbeat agent (installed by _install_heartbeat_agent in agent_parts/spec_rebuild.py) — a minimal MCP surface (kirocrew-core, plus the default-spec servers behind the edition's heartbeat_safe_tools() pins; a standalone install pins none) so cycle cold-starts stay cheap. SEL audit logging stays gateway-side in _heartbeat_approval regardless; the per-agent narrowing is purely a cold-start cost reduction.

The heartbeat spec copies the default spec's heartbeat_mirrored_servers() entries — kirocrew-core with autoApprove included and the main agent's tool filters stripped, then each server an edition pinned a tool on verbatim minus autoApprove (kiro-cli would honour that field locally, before _heartbeat_approval saw the call, on the one agent with no approver) — so it is a derived spec like kirocrew-worker. A name the default spec does not mount is skipped, so an edition contributes a heartbeat server by also returning it from extra_mcp_servers(). Every spawn runs require_fresh_derived_spec, which compares the heartbeat file's own grant surface with the one heartbeat_mcp_servers derives from kirocrew.json as it stands: its mcpServers (volatile per-launch env values aside), its tools, no key the installer does not write, and an includeMcpJson that is literally false (kiro-cli reads a missing key as true and merges the global mcp.json). A copy that does not match is re-derived once, and the start is refused with DerivedSpecStale when the re-derived file still does not match. An edit to any other part of the default spec does not touch the heartbeat copy; the fingerprint covers exactly the mirrored servers, edition ones included.

The session is shared across all tasks in one cycle (so concurrent gather'd tasks reuse the warm provider) and recycled by recycle_heartbeat at the end of every cycle; it is never reused across cycles.

Each task's prompt is built without a session key, but the record of skill bodies the session already holds is kept under HEARTBEAT_KEY (build_message(..., skill_bodies_session=HEARTBEAT_KEY)). A skill that several tasks on one session match is sent in full to the first and as its pointer line to the rest; a fresh session starts the record again. Each task settles the record with rollback_skill_bodies before it releases the session, because the next task builds on the same session: a task whose prompt never landed rolls it back, so the next task gets the full body. When the backend reports a completed compaction during a task's turn (stream_and_collect(on_compaction=...)), the task arms the session's one-shot reinjection flag before the release. The next task consumes it with consume_reinjection and passes needs_reinjection=True, so its prompt sends the skill bodies and session-start context the compaction dropped, and rearm_reinjection puts the flag back if that prompt never lands.

HEARTBEAT_KEEP Injection

Every heartbeat task text is prepended with a fixed instruction at the gateway (_HEARTBEAT_KEEP_INJECTION in slack/gateway_runtime/heartbeat_turns.py) before ctx_builder.build_message. The instruction tells the agent it must include HEARTBEAT_KEEP in its response when the task is incomplete. Inline injection survives context compaction and webhook-restored sessions where skill / system-prompt copies of the same instruction can drift out of effective context.

Tool Approval (heartbeat allowlist)

Heartbeat is unattended — there is no human to click an approval button. Tool approval uses HOOK_BASED policy with a heartbeat-scoped HookManager (rebuilt for every task by _build_heartbeat_hooks(self.ctx_builder.hooks), so a live change to denied commands applies without a restart) and a custom callback (GatewayOrchestrator._heartbeat_approval) that auto-approves only a tool whose trusted identity _is_heartbeat_safe_tool accepts and rejects everything else with a SEL audit event (outcome=denied, reason=no_trusted_identity when the event carries no trusted identity or an empty trusted tool_name (the cache hit on empty names a backend with no _meta identity channel produces), reason=not_in_heartbeat_safe_tools otherwise). Both approve and deny outcomes emit log_tool_invocation so every permission decision is auditable; the row names the server/tool identity the decision keyed on when the event is trusted, and the redacted title otherwise.

The heartbeat-scoped hooks drop the user's auto_approve_tools so the allowlist is the sole approval authority — llm_helpers._resolve_permission would otherwise consult the hooks BEFORE the _heartbeat_approval callback, and a user config like auto_approve_tools=["*"] would auto-approve any tool, bypassing the allowlist entirely. The user's auto_deny_tools IS preserved (denies can only narrow what runs in heartbeat, never widen). The scoped hooks also set HooksConfig.defer_reads_to_approver, so the gate's read-only classifier tier gives no verdict: it auto-approves on the ACP kind, the title and is_shell, all of which the agent influences, so an edition MCP tool declaring kind="read" would otherwise run before _heartbeat_approval and its SEL audit ever saw it. Heartbeat judges reads itself, on the trusted identity, in _is_heartbeat_safe_tool; the deny tiers above still run first.

A hook TOOL_AUTO_APPROVE that DOES fire in _resolve_permission (cron and autonudge turns run with the user's full hooks, read-only tier included) is honoured for a shell command only after the name-grant check confirms each program name still resolves to the program it appears to name; a refusal downgrades to the caller's approver — _heartbeat_approval (exact-match allowlist) or the interactive card — never a hard block, audited as outcome=auto_approve_declined with reason=name_grant. On Windows the check models the shell's lookup and returns per-command verdicts the same way it does on POSIX, except that two host states still decline every name grant: windows_lookup_not_modelled when Windows cannot report where the user's Documents folder is (so whether a PowerShell profile runs before the command cannot be established), and inherited_env_can_redefine_programs (name_grant.AMBIGUOUS_ENV) when a per-user PowerShell profile is present at one of the paths derived from Documents (kiro-cli starts the shell without -NoProfile, so that profile runs before every command and a function it defines resolves ahead of any program on PATH — the same threat BASH_ENV poses on POSIX); in those two states unattended cron/autonudge/heartbeat turns fall to their approver for shell tools the read-only tier auto-approves elsewhere. The check runs UNCONDITIONALLY; a refusal on a caller with no interactive approver rejects the tool (deny-by-default) rather than falling through to the caller-less auto-approve — an unattended turn is exactly where a shadowed name would otherwise run unwatched.

The allowlist is identity-based and exact-match only — no verb / heuristic fallback, and no fallback to the model-authored title. event.mcp_identity_trusted must be set (the tool_name / mcp_server_name pair came from the adapter's _meta channel) and tool_name must be non-empty, since a cache hit on empty names also sets the flag; a backend that publishes none (acp/_dispatch_parts/tool_call_identity.py names claude-agent-acp and opencode) gets no heartbeat tool approval at all and runs as a text-only poller — intended, not a gap to paper over with a title fallback. Each such denial is audited as reason=no_trusted_identity and logs a gateway warning that the backend published no trusted tool identity, so every heartbeat tool call is denied on it. Heartbeat polls untrusted external content (CR comments, ticket bodies) where prompt-injection could try to widen approval via a clever read-shaped tool name (get_all_credentials, list_env_secrets, etc.). Strict enforcement is auditable and cannot be widened that way; this is deny-by-default per the security-controls guideline.

The allowlist is curated for read-only / observation tools, each matched on its own identity shape (slack-gateway.md, Heartbeat Tool Allowlist, has the exact rules): the trusted host built-ins hooks._HOST_READ_ONLY_BUILTIN_TOOLS names (fs_read, glob, grep, web_fetch, web_search — the spellings are the trusted channel's own, not the Read / Grep / Glob / WorkspaceSearch titles an earlier allowlist carried), side-effect-free reads from kirocrew-core (learn_list, spawn_list, spawn_status, artifact_list, artifact_get, artifact_versions, local_knowledge_search), each matched only as its qualified kirocrew-core identity, a host shell command whose recovered text classifies read-only and runs no git (git reads .git/config on invocation and core.fsmonitor / core.hooksPath name a program there that no text classifier sees; the interactive and cron paths have a human behind them, heartbeat does not), and the @server/tool identities an edition pins in SlackEnterpriseGate.heartbeat_safe_tools(). Write tools (send_message, file_send, cron_add, fs_write) and any shell command the classifier does not confirm read-only are rejected.

When a legitimate new read tool needs to run in heartbeat, operators observe SEL denied events (or the gateway-log warning Heartbeat blocked tool call: <name>) and explicitly add the name to the right set behind the heartbeat allowlist (_HEARTBEAT_SAFE_CORE_MCP_TOOLS for a kirocrew-core read, or the hooks built-in allowlist). A no_trusted_identity denial is a property of the backend, not a missing name.

Constants

Constant Value Location
_DEFAULT_INTERVAL 60 heartbeat.py
_FTS_REBUILD_TICKS 15 heartbeat.py
_PRUNE_TICKS 1440 heartbeat.py
_MEMORY_BACKUP_TICKS / _MEMORY_BACKUP_OFFSET 1440 / 30 heartbeat.py
_KEEP_SENTINEL HEARTBEAT_KEEP heartbeat.py
HEARTBEAT_TASK_TIMEOUT_SECS 1800 heartbeat.py
_STOP_DRAIN_SECS 10 heartbeat.py
HEARTBEAT_FILE HEARTBEAT.md heartbeat.py
HEARTBEAT_KEY _hb session_parts/policy.py
_HEARTBEAT_SAFE_CORE_MCP_TOOLS frozenset of kirocrew-core reads slack/gateway_runtime/tool_policy.py
_HEARTBEAT_KEEP_INJECTION reminder string slack/gateway_runtime/heartbeat_turns.py
kirocrew-heartbeat agent minimal-MCP agent JSON installed by agent_parts/spec_rebuild.py:_install_heartbeat_agent

Known Limitations

  • No multiline tasks — each line is a separate task
  • Tasks appended to HEARTBEAT.md while a cycle runs are kept: the rewrite re-reads the file under a file lock, drops only the finished tasks' lines, and replaces it atomically (_rewrite_heartbeat_locked). A finished task is matched by its text and deliver target, first occurrence first, so a duplicate appended mid-cycle survives and the original goes
  • Exception-retried tasks have no max retry count

Delivery Modes

Tasks can specify a delivery target via HTML comment tags:

- [ ] Check CR-123 <!-- deliver:prompt:dashboard:chat-0 -->

Supported Modes

Mode Syntax Behavior
Default (no tag) Routed per heartbeat.default_deliver: slack (default) = owner's Slack DM + dashboard bell; dashboard = new dashboard slot + bell only
Dashboard prompt <!-- deliver:prompt:dashboard:<slot> --> Injects result as a user prompt into a specific dashboard chat slot (e.g., chat-0, chat-3), triggering an agent turn
Dashboard slot <!-- deliver:dashboard:<slot> --> Appends result to a specific dashboard chat slot + bell (no agent turn)
Dashboard <!-- deliver:dashboard --> Appends result to a new dashboard chat slot + bell
Slack DM <!-- deliver:slack --> Posts result to owner's Slack DM only (no bell)
Channel <!-- deliver:slack:<channel_id> --> Posts result as a new message in that Slack channel + bell, if the channel is allowed (see below). A report long enough to split posts its first part top-level and threads the rest under it
Thread <!-- deliver:slack:<channel_id>:<thread_ts> --> Replies in that Slack thread + bell, if the channel is allowed (see below)
Silent <!-- deliver:silent --> Logs only

A slack: tag with an empty channel id (<!-- deliver:slack: -->) falls back to the owner's DM: posting to channel "" would raise inside the swallowed Slack error handler and lose the report.

The deliver tag is agent-writable — the kirocrew-dev/babysit skill tells the agent to append HEARTBEAT.md entries itself, and HEARTBEAT.md is not a fenced path — so the channel id on a slack:<channel_id> tag is untrusted input. Both channel forms therefore require the id to be a well-formed Slack channel id (validation.CHANNEL_ID_RE, so a U… member id is refused rather than resolved to an IM) AND to be either in slack.tracking_channels or the owner's own DM channel — the same allowlist api_send_message states in its 403, because an unattended post reaches a NEW audience. A refused target is SEL-logged heartbeat_channel_deliver / outcome=denied and falls back to the owner DM, so the report is never lost.

Dashboard Delivery (prompt:dashboard:<slot>)

Resolves chat-N slot names to active session keys. The result is injected as a user message into the target slot, triggering an LLM response in that session. Useful for heartbeat tasks that should report back into an active dashboard conversation.

Slot resolution: chat-0 → first active slot, chat-3 → fourth slot. A busy slot queues the prompt. A slot that cannot be resolved drops the delivery: SEL heartbeat_prompt_deliver with outcome=not_found and a warning, with no Slack DM fallback.

Slack Suppression for Incomplete Tasks

When a task response contains HEARTBEAT_KEEP, Slack delivery is suppressed. The task is retained for the next tick without notifying the user. Only completed tasks (no HEARTBEAT_KEEP) trigger Slack/dashboard delivery.