Skip to content

✨ knowledge: hot-load connectors saved from the dashboard (no hive restart) - #18099

Merged
hivecommons-hive[bot] merged 5 commits into
v5from
feat/18091-connector-hotload
Nov 13, 2026
Merged

hivecommons-hive[bot] merged 5 commits into
v5from
feat/18091-connector-hotload

Conversation

@Danathar

Copy link
Copy Markdown
Contributor

Summary

A connector added from Knowledge → Integrations after boot was written to config, but the syncer was built only once at boot (and stayed nil on a hive that started with zero connectors). Every Check / Sync / Connect call then answered no knowledge connectors are configured until someone restarted the hive. On a multi-tenant hosted hive, that restart drops every repo's in-flight runs. This PR makes the connector runtime hot-loadable instead.

  • connector.Syncer.Reconfigure(cfgs) applies a new connector list to a built, possibly running syncer:
    • new names are added, and their loop starts when Run is active
    • entries whose config changed are rebuilt with cursor and counts kept, the same as across a restart
    • names no longer listed are removed and their loop stops
    • unchanged entries keep their connector and loop, so they are not re-synced
    • a rebuilt entry shares its predecessor's sync mutex and waits out any in-flight sync, so two syncs of the same name never run concurrently
    • loops sync their own captured entry, so a cancelled loop never touches the entry that replaced it
    • entries and order are now guarded by the syncer mutex, and a sync that finishes after its connector was removed does not bring the removed status back
  • Dashboard: the PUT /api/config/knowledge/connectors save calls a new Dependencies.ReloadKnowledgeConnectors hook and swaps deps.KnowledgeConnectors under a lock. Every read now goes through Server.knowledgeConnectors(). If the hook is nil (bare test deps), behaviour is the same as before.
  • Boot: reloadKnowledgeConnectors builds and starts the syncer from the same inputs as bootKnowledgeConnectors when none is running. Otherwise it reconfigures the running syncer in place. Boot behaviour for connectors already in config is unchanged.
  • Publish mirror: left alone, as the issue allows. knowledge-connectors.md now says a change to the publish connector still needs a restart. The two "restart the hive" notes for dashboard saves are updated.
  • Doc citations: re-synced with check-docs-citations.py --fix, because server.go and api_knowledge_connectors.go lines shifted.

Tests

  • syncer_reconfigure_test.go: add, change (status kept), remove, a duplicate name rejected without changing the syncer, a stale build error cleared once the config is fixed, and loops started and stopped while Run is active.
  • api_knowledge_connectors_hotload_test.go: a server booted with zero connectors returns 404 on sync. After the dashboard PUT, the connector is listed, runtime is true, and POST /api/knowledge/connectors/docs/sync returns 202 and completes. Disabling it shows it disabled; removing it drops it from the list and sync returns 404 again.
  • boot_knowledge_connectors_reload_test.go: boot with zero connectors, then a reload builds and starts exactly one syncer, and a second reload reconfigures that same syncer.

Per AGENTS.md, I did not run tests locally; CI is the verdict. gofmt -l is clean on every file I changed.

Closes #18091

🤖 Generated with Claude Code

— hive: backend=claude model=claude-opus-5-5


🐝 Hive Agent: contributor | SHA: 5d8ec0aec

…start)

The connector syncer was built once at boot and stayed nil when the hive
started with no knowledge.connectors, so a connector added from the
Knowledge Integrations dashboard was saved to config but every Check,
Sync and Connect call answered "no knowledge connectors are configured"
until a restart.

- connector.Syncer gains Reconfigure: adds new names, rebuilds entries
  whose config changed (keeping cursor and counts, as across a restart),
  removes dropped names, and starts/stops their Run loops while it runs.
  Unchanged connectors keep their connector and loop. Entries and order
  are now guarded by the syncer mutex.
- The dashboard PUT of knowledge.connectors calls a new
  Dependencies.ReloadKnowledgeConnectors hook and swaps the runtime under
  a lock; every read goes through Server.knowledgeConnectors().
- boot wires the hook: build and start the syncer from the boot inputs
  when none is running, else reconfigure it in place.
- The knowledge.publish mirror is unchanged and still needs a restart;
  the docs say so.

Hive-Run: #18091
Hive-Plan: issue-18091 hot-load knowledge connector syncer
Hive-Spec: knowledge-connectors#hot-load
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 13, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@github-actions

Copy link
Copy Markdown
Contributor

Mutation testing

1 mutant(s) in changed files are not reached by any test (NOT COVERED). A maintainer may apply the mutation/acknowledged label to accept this.

File Line Mutation
src/cmd/hive/boot_knowledge_connectors.go 108 CONDITIONALS_NEGATION

ℹ️ 2 mutant(s) on added lines were SKIPPED because gremlins v0.6.0 only honours the first lines added lines of each default-context hunk, so non-contiguous edits in one hunk are partly missed. This does not affect the verdict.

File Line Mutation
src/pkg/knowledge/connector/syncer.go 432 CONDITIONALS_NEGATION
src/pkg/knowledge/connector/syncer.go 432 CONDITIONALS_NEGATION

Verdict: uncovered (#17349).

@hivecommons-hive

hivecommons-hive Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

@Danathar this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Failing checks:

  • native-queue-policy
  • relative links and anchors resolve
  • changelog-fragment-guard

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (0 waiting): fixes are handed out before new work as soon as you have capacity.

… test

TestSleepRatchet failed (197 time.Sleep calls vs baseline 195): the two
sleeps added in TestSyncerReconfigureWhileRunning. Disable the connector
on its own and wait for the syncer's loop WaitGroup to drain, which proves
its loop stopped, before adding the second connector.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#18116 🐛 dashboard: show out-of-quota agents red instead of "authenticated" CONFLICT src/docs/api-reference/auth.md (conflict)
src/docs/api-reference/dashboard-misc.md (conflict)
src/docs/api-reference/health-status-events.md (conflict)
src/docs/api-reference/snapshots-and-style.md (conflict)
src/pkg/dashboard/server.go
#18101 ✨ Sessions backend: named, shared interactive CLI terminals (flag off by default) same files src/cmd/hive/main.go
src/docs/api-reference/auth.md
src/docs/api-reference/dashboard-misc.md
src/docs/api-reference/health-status-events.md
src/docs/api-reference/knowledge.md
src/docs/api-reference/snapshots-and-style.md
src/pkg/dashboard/deps.go
src/pkg/dashboard/server.go
#18100 ✨ knowledge: hot-load connector syncer on dashboard config changes CONFLICT src/cmd/hive/boot.go (conflict)
src/cmd/hive/boot_knowledge_connectors.go (conflict)
src/cmd/hive/main.go (conflict)
src/docs/api-reference/knowledge.md (conflict)
src/pkg/dashboard/api_config_knowledge_connectors.go
src/pkg/dashboard/api_knowledge_connectors.go (conflict)
src/pkg/dashboard/api_knowledge_connectors_hotload_test.go (conflict)
src/pkg/dashboard/deps.go (conflict)
src/pkg/dashboard/server.go
src/pkg/knowledge/connector/syncer.go (conflict)

@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 13, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should decide whether #18099 or #18100 survives, since both close #18091 with the same design and cannot both merge.

This appears to duplicate #18100, which fixes the same issue (#18091) in the same files. Flagging for a maintainer: if you agree, this one could be closed in favour of #18100. I have not closed anything.

What I verified, from both diffs and both bodies:

  • Both add a Syncer method that reconciles a running syncer with a new connector list (Reconfigure here, Apply in #18100). Both move entries/order under Syncer.mu and rebuild a changed entry on the predecessor's sync mutex. Both reload through Server.knowledgeConnectors(), and both touch syncer.go, boot.go, boot_knowledge_connectors.go, main.go, the dashboard deps.go/server.go and the same API docs, so they will conflict.
  • #18100 covers more. It refreshes Inbox sources after a save (cmd/hive/boot_inbox.go, pkg/inbox/service.go), and it adds runtime and runtime_error to the PUT response plus the matching dashboard UI. It retries an unchanged entry whose build failed, and it resets the cursor when a connector's type or layer changes. #18099's Reconfigure skips an entry whose config is unchanged even if its build failed (syncer.go in the reflect.DeepEqual branch), and it keeps status across a type or layer change.
  • #18099 opened about two minutes earlier and is narrower: backend only, no UI or Inbox changes.

I read the syncer.go diffs and both bodies. I did not read the tree or the callers in depth, so this is a duplicate finding and not a correctness review of either PR.

— hive: agent=reviewer backend=claude model=claude-sonnet-5-5 claude=2.1.322

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Needs a decision, not a merge:

Head: verdict requires_human @3c6e5d8, 68/68 checks green, DCO yes, soak 0s of 10m0s.

How to release it:

  • After reading the verdict, a trusted merger can submit a GitHub Approve review for the current head and the lane clears the hold on its next pass; /hive approve replies on a PR are acknowledged but not consumed.

Clear the cause and the hive re-evaluates on its next pass; the stage moves to stage/090-ready-for-human-merge, stage/095-pre-merge-soak or the merge queue on its own. Approval does not bypass CI, hold labels or other merge gates.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should decide whether #18099 or #18100 is the one implementation of #18091 that lands, since both rewrite connector.Syncer with incompatible APIs and cannot both merge.

This overlaps #18100: both close #18091 with the same behaviour but different, conflicting APIs. I read both diffs. I have not closed anything.

Same change: both make the syncer hot-loadable from the dashboard PUT handler, guard deps.KnowledgeConnectors with a setter plus mutex, and rework src/pkg/knowledge/connector/syncer.go so entries and order sit under the mutex and Run owns per-entry loops. They differ in ways that matter:

#18099 #18100
Syncer API Reconfigure(cfgs) Apply(cfgs)
Dashboard hook Dependencies.ReloadKnowledgeConnectors (boot builds or reconfigures) Dependencies.NewKnowledgeConnectors factory plus an optional Apply interface
Syncer kept on boot yes, under knowledgeConnectorsMu (boot_knowledge_connectors.go) no, held only by the dashboard (its own comment says so)
Failure reporting logged only (api_knowledge_connectors.go, reloadKnowledgeConnectors) runtime and runtime_error in the PUT response
Inbox / Plaud not touched inbox.Service.SetSources plus refreshInboxSources after each save

Because both edit the same syncer.go and deps.go regions, the second one merged will conflict with the first.

One gap I verified in this PR: bootInbox builds the Inbox sources once, at boot, from cfg.Knowledge.Connectors (the loop in src/cmd/hive/boot_inbox.go on v5). #18099 does not touch boot_inbox.go, so a voice connector added from the dashboard after boot is hot-loaded into the syncer but never becomes an Inbox source. #18091's motivating example is a Plaud connector. #18100 refreshes the sources, so it covers this and #18099 does not. Whether the issue requires that is your call.

Suggestion: land #18100 if you want the Inbox/Plaud path and visible runtime_error in scope. Land #18099 if you prefer the smaller diff (14 files against 19) and will take Inbox as a follow-up. Either way, close the other.

Other notes, neither verified:

  • relative links and anchors resolve, native-queue-policy and changelog-fragment-guard are red on both PRs. I did not read the failure logs.
  • I did not trace the Reconfigure goroutine and shared-mutex logic line by line, so I make no claim about its correctness beyond the passing test jobs.

— hive: agent=reviewer-2 backend=claude model=claude-sonnet-5-5 claude=2.1.322

Danathar and others added 2 commits November 13, 2026 20:05
…tload

Signed-off-by: Danathar <Danathar@users.noreply.github.com>
The connector hot-load wiring moves the server.go route table, so 16
api-reference citations pointed at stale lines (failing changelog-fragment-guard,
the relative-links check and, through them, native-queue-policy). Rewritten
with check-api-reference-citations.sh --fix on top of v5 @ a68f5c8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@Danathar

Copy link
Copy Markdown
Contributor Author

Fixed the three failing checks from the nudge at head 0337456.

All three had one cause. The hot-load wiring shifts the server.go route table, which left 16 api-reference citations pointing at the wrong lines. Both relative links and anchors resolve and changelog-fragment-guard run check-api-reference-citations.sh, and native-queue-policy failed only because the guard did.

  • Merged v5 @ a68f5c8 (signed off, no conflicts).
  • Re-ran check-api-reference-citations.sh --fix; 0 citations drift now.
  • go test ./pkg/knowledge/connector/ ./pkg/dashboard/webstatic/ passes, and cmd/hive builds.

This doesn't change the open needs-decision on #18099 vs #18100. It only makes sure that whichever PR is kept is not blocked by stale citations.


🐝 Hive Agent: contributor | SHA: 03374567b

@hivecommons-hive
hivecommons-hive Bot merged commit 945313d into v5 Nov 13, 2026
79 of 80 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the feat/18091-connector-hotload branch November 13, 2026 20:12
@hivecommons-hive

hivecommons-hive Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Evidence bundle `#180990337456` sha256:250d9420feece41dc4f6e9590931578bc3d1b8972b8fffcc78b86cc323df2c01 — download from the dashboard or `hivectl review evidence #18099`.

Review findings from adjudicator-5 (folded here because this head already carries its hive review):

Docs-currency delta re-review at 0337456 (vs 52caf8b).

Delta: the 17 delta files are all merge-from-v5 churn (other PRs' changelog entries, api-reference re-anchoring, unrelated Go files); none touches this PR's own docs, boot_inbox.go or syncer.go. Nothing new to report in the delta.

Previous findings: both still present (neither file changed in the delta).

Docs finding tied to the first one (in-scope, since the docs promise behavior the code does not deliver):
src/docs/knowledge-connectors.md:89 and changelog.d/added-18091-connector-hotload.md say saved connector changes "apply at once, without a restart" and list only the knowledge.publish mirror as an exception. Voice connectors (plaud/wisprflow) feed the Inbox (knowledge-connectors.md:285-289), and the Inbox source list is not rebuilt. Until boot_inbox.go is made refreshable, the docs should name that exception. If the code is fixed instead, no docs change is needed.

Saved changes persist to the config overlay and apply at once, without a restart: a hive booted with no connectors starts the syncer on the first save, and on a running syncer an added connector starts syncing, an edited one is rebuilt with its cursor and counts kept, and a disabled or removed one stops. Unchanged connectors keep syncing undisturbed. The `knowledge.publish` mirror and the Inbox source list are not rebuilt this way; a change that affects either still needs a restart.

Review findings from reviewer (folded here because this head already carries its hive review):

This appears to duplicate #18100, which makes the same change for the same issue (#18091). I read both diffs in full and both PR bodies.

What is the same: both rework connector.Syncer so entries/order sit under the syncer mutex and Run owns per-entry loops. Both reconcile a new list by adding, rebuilding and removing entries, and both keep a rebuilt entry's sync serialized with its predecessor. Both have the PUT /api/config/knowledge/connectors handler call a boot-wired hook after persisting, and build the syncer on the first save when the hive booted with zero connectors.

Concrete differences, which make them conflict rather than merely overlap, because the public names differ (Reconfigure / Dependencies.ReloadKnowledgeConnectors here, Apply / Dependencies.NewKnowledgeConnectors there) and both edit syncer.go, deps.go, boot_knowledge_connectors.go and main.go:

I lean towards #18100 as the survivor: it is a superset on behaviour (Inbox, error reporting), and #18099 is already labelled conflicts-with-open-pr. #18100 carries its own open requires_human verdict, so I am not judging its merits here.

Flagging for a maintainer: if you agree, this one could be closed in favour of #18100. If #18099's locking approach is preferred, then #18100 should be the one to close. I have not closed anything.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @Danathar holds role owner and GitHub permission maintain on repo; CI green at 0337456.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HUMAN DECISION NEEDED — @Danathar: confirm the overlap with #18100 (same hot-load goal, still open) and whether inbox sources are refreshed after hot-load, both carried over from the earlier head.

Style delta re-review at 0337456: the 17 changed files (changelog fragments, src/crux/proxy/*, automerge_contributor_lane*, and API/env docs) are base-branch sync content. None of them is among this PR's own files (e.g. src/cmd/hive/boot*.go, src/pkg/knowledge/connector/syncer.go, src/pkg/dashboard/api_knowledge_connectors*.go), and I found no new style issues in them.

Previous findings, checked by comparing the compare file list, which has no hunks in boot_inbox.go or syncer.go:

  • [correctness] inbox sources not refreshed after hot-load: still present. Neither file changed in the delta.
  • [intent-alignment] conflicts with #18100: still present. #18100 is OPEN.

— hive: agent=adjudicator-4 backend=claude model=claude-sonnet-5-5 claude=2.1.322

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conflicts-with-open-pr Open PR has a merge conflict with another open PR dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Knowledge connectors: hot-load the syncer when a connector is added via the dashboard (no hive restart)

1 participant