[clubanderson]

Fixes #17862.

Problem

Config.Save wrote /etc/hive/hive.yaml and /data/hive.yaml.runtime in place with O_TRUNC. On hosted-available-vllmd-03 (liveness-killed ~14x/hr since Nov 7) a kill between truncate and write left a 0-byte runtime file; the entrypoint's -s guard fell back to /data/hive.yaml.bak - frozen at the Aug 2026 rename on every fleet hive - and the hive booted acmm_level 3 / 6 agents instead of the owner's level 6. No log line said so.

Fix

• writeFileAtomic (pkg/config/atomic_write.go): temp file in the target's directory -> fsync -> rename onto the EvalSymlinks-resolved target, so the 🐛 /etc/hive/hive.yaml is an emptyDir on hosted spokes — edits don't survive restart #17468 /etc/hive/hive.yaml -> runtime link survives (TestSaveThroughSymlinkKeepsLink stays green). If rename is refused (single-file Docker bind mount, EBUSY) it falls back to the previous in-place write, so those deployments behave exactly as before.
• Both Save writes use it; runtime stays 0600, the source path keeps its existing mode.
• entrypoint: hive_refresh_legacy_fallback copies a YAML-valid (has project:) runtime config over hive.yaml.bak via tmp+mv on every clean boot, only when the new name is the boot source - so the fallback is at most one boot stale and the Docker/LXC "only copy" worry never applies. hive_runtime_config_read now WARNs loudly on stderr when it falls back (names the 0-byte truncation, fallback age, and flags a fallback older than the damaged runtime file with pointers to .dashboard/audit.jsonl).
• Hub copy-config template comment and RuntimeConfigFileLegacy doc updated.

Tests

• atomic_write_test.go: content+mode replacement, missing-file create, symlink target preserved with no temp in the link dir, and TestSaveSurvivesKillMidWrite (reader racing 200 Saves never observes an empty/partial file - fails against the old O_TRUNC write).
• deploy/test_entrypoint_runtime_config.sh: fallback WARN emitted / steady state silent; refresh happens from a valid runtime, not from an invalid or project-less one, not when booting from legacy; legacy is never removed and only moved-over inside the refresh helper. 18/18 pass locally.

Evidence

Incident write-up for vllmd-03 in the issue; Deployment/PVC//data were intact, only the runtime config reverted.

[clubanderson]

/lgtm
/approve

[kubestellar-prow]

@clubanderson: you cannot LGTM your own PR.

Details

In response to this:

/lgtm
/approve

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[kubestellar-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: clubanderson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [clubanderson]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

Open PR overlap check

✅ Resolved: no conflicting open PRs or same-file overlaps remain against v5.

[github-actions]

Mutation testing

ℹ️ 1 mutant(s) on added lines were SKIPPED because gremlins v0.6.0 only honours the first lines added lines of each default-context hunk, so non-contiguous edits in one hunk are partly missed. This does not affect the verdict.

File	Line	Mutation
src/pkg/config/config.go	7808	CONDITIONALS_NEGATION

All 18 mutant(s) in changed files were killed.

Verdict: pass (#17349).

[hivecommons-hive]

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

• coverage

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Read atomic_write.go, the saveLocked changes, the entrypoint helpers, and the copy-config template at v5. The core change looks correct to me; one gap in the fallback worth a look.

Verified

• writeFileAtomic resolves the link with EvalSymlinks before choosing the rename target (src/pkg/config/atomic_write.go:37-42), so on hosted spokes the /etc/hive/hive.yaml -> /data/hive.yaml.runtime link from saas_provision.go:3472 survives — the rename lands on the PVC file, not on the emptyDir link. The template only creates that link when the runtime file is -s (non-empty), so the dangling-link case where EvalSymlinks returns ErrNotExist and the rename would replace the link itself is not produced by the hosted path.
• Temp file is created in the target's directory, chmod'd before any data is written, fsync'd, then renamed; syncDir is best-effort after. Rename failure (EBUSY on a single-file bind mount) falls back to the old O_TRUNC write (atomic_write.go:71-79), which is what those deployments had before.
• sourceConfigMode uses os.Stat (follows the link) so a tightened source-file mode is preserved; the explicit post-write Chmod(0600) on the runtime file is correctly dropped since the mode is now applied to the fresh inode.
• hive_refresh_legacy_fallback is gated on $1 = $HIVE_CONFIG_RUNTIME and -s (entrypoint.sh, new function), so it only runs when the new name is this boot's source — the legacy file is never the only copy when it is overwritten, which is the Docker/LXC concern the old never-write rule protected. The copy is tmp + mv -f in the same directory, so the fallback cannot itself be truncated by a kill. hive_harden_runtime_config exists at entrypoint.sh:617.

One gap (low): the in-place fallback does not cover a failed CreateTemp

atomic_write.go:44-48: if the target's directory is not writable but the file is (root-owned /etc/hive with a user-owned hive.yaml, or a read-only dir with a writable file), os.CreateTemp fails and the function returns that error — writeFileInPlace is only reached on a failed rename. The previous OpenFile(O_WRONLY|O_TRUNC) would have succeeded there. On Kubernetes this is absorbed by the existing tolerance at config.go:7836-7847 (source failure is logged and Save returns nil when runtime + overlay were written), so it is a behaviour change only where the runtime layer also fails. Since the PR already accepts in-place as an acceptable fallback, extending it to an EACCES/EROFS from CreateTemp would close the gap cheaply. Suggestion, not a blocker.

Minor: TestSaveSurvivesKillMidWrite races a reader against concurrent Saves rather than simulating a kill — it does pin "no reader observes a partial file", which is the property that matters, just noting the name over-promises slightly.

stage 080-awaiting-merge-admission → 071-needs-author: required-check-failing:coverage

[hivecommons-hive]

1273a7e

ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @clubanderson, head 1273a7e37d86) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

• sensitive_path — changes 2 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
  • src/deploy/entrypoint.sh
  • src/deploy/test_entrypoint_runtime_config.sh

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

[hivecommons-hive]

Read writeFileAtomic, both saveLocked call sites, the hub copy-config init command and the entrypoint resolver against v5. This looks correct to me; no defects found.

What I verified

• Symlink case (#17468): the init container only creates /etc/hive/hive.yaml -> /data/hive.yaml.runtime when the target is non-empty (src/pkg/hub/saas_provision.go:3472, [ -s … ] guard), so EvalSymlinks in atomic_write.go:41-46 always resolves to an existing target and the rename lands on the PVC file, never on the link. The ErrNotExist branch (dangling link replaced by a regular file) is therefore unreachable from that template.
• Bind-mount fallback: rename(2) onto a mount point fails with EBUSY; atomic_write.go:76-84 then performs the old O_TRUNC write, so single-file Docker mounts keep their inode exactly as before.
• Mode handling: sourceConfigMode (config.go:7834) uses os.Stat, which follows the link, so on hosted spokes the source write preserves the PVC file's 0600 rather than widening it to 0644. The explicit post-create Chmod that was removed from the runtime branch is now covered by tmp.Chmod(mode) before the rename (atomic_write.go:54).
• Entrypoint: hive_refresh_legacy_fallback only fires when $1 == $HIVE_CONFIG_RUNTIME and that file is non-empty and parses with a project map, so the legacy file is never the sole copy when it is overwritten; the copy goes through ${…}.tmp.$$ + mv -f. Cleanup registered in the shell test matches this.

Info (not a defect): the YAML parse guard depends on import yaml. src/Dockerfile:712-714 installs PyYAML best-effort (|| echo WARN), so an image where that install failed would log "does not parse as a hive config" on every boot and never refresh the fallback — a graceful degradation back to today's behaviour, but worth knowing when reading boot logs.

The PR body's claims match the diff (atomic write on both Save targets, loud WARN on fallback, refresh only from the new name). coverage is the only red check; nothing in the diff looks responsible for that beyond the new file needing its tests counted.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

[hivecommons-hive]

CI: coverage is the only red check — pkg/config landed at 89.9% against a 90% floor (run 43167441822). PR-local: the drop comes from the error/fallback branches in the new atomic_write.go, which the four tests there never reach.

Not pushing to your branch (human-authored PR); suggested additions to src/pkg/config/atomic_write_test.go that cover the unexercised branches with no os.Rename faking:

// EvalSymlinks error that is NOT ErrNotExist (ENOTDIR) must surface, not be
// treated as "missing target".
func TestWriteFileAtomicRejectsPathThroughRegularFile(t *testing.T) {
	dir := t.TempDir()
	file := filepath.Join(dir, "plain")
	if err := os.WriteFile(file, []byte("x"), 0o644); err != nil {
		t.Fatal(err)
	}
	if err := writeFileAtomic(filepath.Join(file, "child.yaml"), []byte("y"), 0o600); err == nil {
		t.Fatal("expected error for path through a regular file")
	}
}

// Unwritable target directory: CreateTemp fails and the error is reported.
func TestWriteFileAtomicUnwritableDir(t *testing.T) {
	if os.Geteuid() == 0 {
		t.Skip("root ignores directory modes")
	}
	dir := t.TempDir()
	if err := os.Chmod(dir, 0o500); err != nil {
		t.Fatal(err)
	}
	t.Cleanup(func() { _ = os.Chmod(dir, 0o700) })
	if err := writeFileAtomic(filepath.Join(dir, "hive.yaml"), []byte("y"), 0o600); err == nil {
		t.Fatal("expected error when temp file cannot be created")
	}
}

// The rename fallback is kept for bind mounts; pin it directly.
func TestWriteFileInPlace(t *testing.T) {
	dir := t.TempDir()
	path := filepath.Join(dir, "hive.yaml")
	if err := os.WriteFile(path, []byte("old-longer-content"), 0o644); err != nil {
		t.Fatal(err)
	}
	if err := writeFileInPlace(path, []byte("new"), 0o600); err != nil {
		t.Fatal(err)
	}
	got, _ := os.ReadFile(path)
	if string(got) != "new" {
		t.Fatalf("content = %q, want new", got)
	}
	if fi, _ := os.Stat(path); fi.Mode().Perm() != 0o600 {
		t.Fatalf("mode = %v, want 0600", fi.Mode().Perm())
	}
	if err := writeFileInPlace(filepath.Join(dir, "missing", "hive.yaml"), []byte("x"), 0o600); err == nil {
		t.Fatal("expected error for missing parent dir")
	}
}

sourceConfigMode for a missing path (0o644 default) is also worth one assertion if the floor is still tight after these.

---

🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

[hivecommons-hive]

CI triage (hive scanner, comment-only on a contributor-authored branch): the only red check is coverage (v2 Tests) — COVERAGE GATE FAILED: config: 89.9% (floor 90%). Overall ./pkg is 89.7%, ./crux 92.9%, both fine; it is just the pkg/config package landing 0.1pt under its floor after the new atomic-save / legacy-fallback branches. One small table-driven test covering the error paths of the new save helper (rename failure, tmp-file write failure, fresh-fallback when the legacy file is absent) should clear it. Baseline check: v5's own coverage job is not red, so this is PR-local.

---

🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88