Repository navigation
✨ knowledge: hot-load connectors saved from the dashboard (no hive restart) - #18099
Conversation
…start) The connector syncer was built once at boot and stayed nil when the hive started with no knowledge.connectors, so a connector added from the Knowledge Integrations dashboard was saved to config but every Check, Sync and Connect call answered "no knowledge connectors are configured" until a restart. - connector.Syncer gains Reconfigure: adds new names, rebuilds entries whose config changed (keeping cursor and counts, as across a restart), removes dropped names, and starts/stops their Run loops while it runs. Unchanged connectors keep their connector and loop. Entries and order are now guarded by the syncer mutex. - The dashboard PUT of knowledge.connectors calls a new Dependencies.ReloadKnowledgeConnectors hook and swaps the runtime under a lock; every read goes through Server.knowledgeConnectors(). - boot wires the hook: build and start the syncer from the boot inputs when none is running, else reconfigure it in place. - The knowledge.publish mirror is unchanged and still needs a restart; the docs say so. Hive-Run: #18091 Hive-Plan: issue-18091 hot-load knowledge connector syncer Hive-Spec: knowledge-connectors#hot-load Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Mutation testing1 mutant(s) in changed files are not reached by any test (NOT COVERED). A maintainer may apply the
ℹ️ 2 mutant(s) on added lines were SKIPPED because gremlins v0.6.0 only honours the first
Verdict: |
|
@Danathar this PR is back in your court ( Failing checks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (0 waiting): fixes are handed out before new work as soon as you have capacity. |
… test TestSleepRatchet failed (197 time.Sleep calls vs baseline 195): the two sleeps added in TestSyncerReconfigureWhileRunning. Disable the connector on its own and wait for the syncer's loop WaitGroup to drain, which proves its loop stopped, before adding the second connector. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Danathar <Danathar@users.noreply.github.com>
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
There was a problem hiding this comment.
Decision needed: @clubanderson should decide whether #18099 or #18100 survives, since both close #18091 with the same design and cannot both merge.
This appears to duplicate #18100, which fixes the same issue (#18091) in the same files. Flagging for a maintainer: if you agree, this one could be closed in favour of #18100. I have not closed anything.
What I verified, from both diffs and both bodies:
- Both add a
Syncermethod that reconciles a running syncer with a new connector list (Reconfigurehere,Applyin #18100). Both moveentries/orderunderSyncer.muand rebuild a changed entry on the predecessor's sync mutex. Both reload throughServer.knowledgeConnectors(), and both touchsyncer.go,boot.go,boot_knowledge_connectors.go,main.go, the dashboarddeps.go/server.goand the same API docs, so they will conflict. - #18100 covers more. It refreshes Inbox sources after a save (
cmd/hive/boot_inbox.go,pkg/inbox/service.go), and it addsruntimeandruntime_errorto the PUT response plus the matching dashboard UI. It retries an unchanged entry whose build failed, and it resets the cursor when a connector's type or layer changes. #18099'sReconfigureskips an entry whose config is unchanged even if its build failed (syncer.goin thereflect.DeepEqualbranch), and it keeps status across a type or layer change. - #18099 opened about two minutes earlier and is narrower: backend only, no UI or Inbox changes.
I read the syncer.go diffs and both bodies. I did not read the tree or the callers in depth, so this is a duplicate finding and not a correctness review of either PR.
— hive: agent=reviewer backend=claude model=claude-sonnet-5-5 claude=2.1.322
|
Needs a decision, not a merge:
Head: verdict requires_human @3c6e5d8, 68/68 checks green, DCO yes, soak 0s of 10m0s. How to release it:
Clear the cause and the hive re-evaluates on its next pass; the stage moves to |
There was a problem hiding this comment.
Decision needed: @clubanderson should decide whether #18099 or #18100 is the one implementation of #18091 that lands, since both rewrite connector.Syncer with incompatible APIs and cannot both merge.
This overlaps #18100: both close #18091 with the same behaviour but different, conflicting APIs. I read both diffs. I have not closed anything.
Same change: both make the syncer hot-loadable from the dashboard PUT handler, guard deps.KnowledgeConnectors with a setter plus mutex, and rework src/pkg/knowledge/connector/syncer.go so entries and order sit under the mutex and Run owns per-entry loops. They differ in ways that matter:
| #18099 | #18100 | |
|---|---|---|
| Syncer API | Reconfigure(cfgs) |
Apply(cfgs) |
| Dashboard hook | Dependencies.ReloadKnowledgeConnectors (boot builds or reconfigures) |
Dependencies.NewKnowledgeConnectors factory plus an optional Apply interface |
Syncer kept on boot |
yes, under knowledgeConnectorsMu (boot_knowledge_connectors.go) |
no, held only by the dashboard (its own comment says so) |
| Failure reporting | logged only (api_knowledge_connectors.go, reloadKnowledgeConnectors) |
runtime and runtime_error in the PUT response |
| Inbox / Plaud | not touched | inbox.Service.SetSources plus refreshInboxSources after each save |
Because both edit the same syncer.go and deps.go regions, the second one merged will conflict with the first.
One gap I verified in this PR: bootInbox builds the Inbox sources once, at boot, from cfg.Knowledge.Connectors (the loop in src/cmd/hive/boot_inbox.go on v5). #18099 does not touch boot_inbox.go, so a voice connector added from the dashboard after boot is hot-loaded into the syncer but never becomes an Inbox source. #18091's motivating example is a Plaud connector. #18100 refreshes the sources, so it covers this and #18099 does not. Whether the issue requires that is your call.
Suggestion: land #18100 if you want the Inbox/Plaud path and visible runtime_error in scope. Land #18099 if you prefer the smaller diff (14 files against 19) and will take Inbox as a follow-up. Either way, close the other.
Other notes, neither verified:
relative links and anchors resolve,native-queue-policyandchangelog-fragment-guardare red on both PRs. I did not read the failure logs.- I did not trace the
Reconfiguregoroutine and shared-mutex logic line by line, so I make no claim about its correctness beyond the passing test jobs.
— hive: agent=reviewer-2 backend=claude model=claude-sonnet-5-5 claude=2.1.322
…tload Signed-off-by: Danathar <Danathar@users.noreply.github.com>
The connector hot-load wiring moves the server.go route table, so 16 api-reference citations pointed at stale lines (failing changelog-fragment-guard, the relative-links check and, through them, native-queue-policy). Rewritten with check-api-reference-citations.sh --fix on top of v5 @ a68f5c8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
Fixed the three failing checks from the nudge at head 0337456. All three had one cause. The hot-load wiring shifts the server.go route table, which left 16 api-reference citations pointing at the wrong lines. Both
This doesn't change the open needs-decision on #18099 vs #18100. It only makes sure that whichever PR is kept is not blocked by stale citations. 🐝 Hive Agent: |
|
Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor. Evidence bundle `#180990337456` sha256:250d9420feece41dc4f6e9590931578bc3d1b8972b8fffcc78b86cc323df2c01 — download from the dashboard or `hivectl review evidence #18099`.Review findings from Docs-currency delta re-review at 0337456 (vs 52caf8b). Delta: the 17 delta files are all merge-from-v5 churn (other PRs' changelog entries, api-reference re-anchoring, unrelated Go files); none touches this PR's own docs, Previous findings: both still present (neither file changed in the delta).
Docs finding tied to the first one (in-scope, since the docs promise behavior the code does not deliver): Review findings from This appears to duplicate #18100, which makes the same change for the same issue (#18091). I read both diffs in full and both PR bodies. What is the same: both rework Concrete differences, which make them conflict rather than merely overlap, because the public names differ (
I lean towards #18100 as the survivor: it is a superset on behaviour (Inbox, error reporting), and #18099 is already labelled Flagging for a maintainer: if you agree, this one could be closed in favour of #18100. If #18099's locking approach is preferred, then #18100 should be the one to close. I have not closed anything. |
There was a problem hiding this comment.
HUMAN DECISION NEEDED — @Danathar: confirm the overlap with #18100 (same hot-load goal, still open) and whether inbox sources are refreshed after hot-load, both carried over from the earlier head.
Style delta re-review at 0337456: the 17 changed files (changelog fragments, src/crux/proxy/*, automerge_contributor_lane*, and API/env docs) are base-branch sync content. None of them is among this PR's own files (e.g. src/cmd/hive/boot*.go, src/pkg/knowledge/connector/syncer.go, src/pkg/dashboard/api_knowledge_connectors*.go), and I found no new style issues in them.
Previous findings, checked by comparing the compare file list, which has no hunks in boot_inbox.go or syncer.go:
- [correctness] inbox sources not refreshed after hot-load: still present. Neither file changed in the delta.
- [intent-alignment] conflicts with #18100: still present. #18100 is OPEN.
— hive: agent=adjudicator-4 backend=claude model=claude-sonnet-5-5 claude=2.1.322
Summary
A connector added from Knowledge → Integrations after boot was written to config, but the syncer was built only once at boot (and stayed
nilon a hive that started with zero connectors). Every Check / Sync / Connect call then answeredno knowledge connectors are configureduntil someone restarted the hive. On a multi-tenant hosted hive, that restart drops every repo's in-flight runs. This PR makes the connector runtime hot-loadable instead.connector.Syncer.Reconfigure(cfgs)applies a new connector list to a built, possibly running syncer:Runis activeentriesandorderare now guarded by the syncer mutex, and a sync that finishes after its connector was removed does not bring the removed status backPUT /api/config/knowledge/connectorssave calls a newDependencies.ReloadKnowledgeConnectorshook and swapsdeps.KnowledgeConnectorsunder a lock. Every read now goes throughServer.knowledgeConnectors(). If the hook is nil (bare test deps), behaviour is the same as before.reloadKnowledgeConnectorsbuilds and starts the syncer from the same inputs asbootKnowledgeConnectorswhen none is running. Otherwise it reconfigures the running syncer in place. Boot behaviour for connectors already in config is unchanged.knowledge-connectors.mdnow says a change to the publish connector still needs a restart. The two "restart the hive" notes for dashboard saves are updated.check-docs-citations.py --fix, becauseserver.goandapi_knowledge_connectors.golines shifted.Tests
syncer_reconfigure_test.go: add, change (status kept), remove, a duplicate name rejected without changing the syncer, a stale build error cleared once the config is fixed, and loops started and stopped whileRunis active.api_knowledge_connectors_hotload_test.go: a server booted with zero connectors returns 404 on sync. After the dashboard PUT, the connector is listed,runtimeis true, andPOST /api/knowledge/connectors/docs/syncreturns 202 and completes. Disabling it shows it disabled; removing it drops it from the list and sync returns 404 again.boot_knowledge_connectors_reload_test.go: boot with zero connectors, then a reload builds and starts exactly one syncer, and a second reload reconfigures that same syncer.Per AGENTS.md, I did not run tests locally; CI is the verdict.
gofmt -lis clean on every file I changed.Closes #18091
🤖 Generated with Claude Code
— hive: backend=claude model=claude-opus-5-5
🐝 Hive Agent:
contributor| SHA:5d8ec0aec