test(k3s): use direct ClusterIP access in tmachine

[matthewgrossman]

Summary

Connect tmachine K3s conformance clients directly to the gateway's default ClusterIP Service, removing the long-lived kubectl port-forward process. The CLI runs on the K3s node inside the guest, so no NodePort exposure is needed. Discover and register the Service IP once during installation, then reuse the installed registration on test boots.

Related Issue

Closes #4254. Replaces the closed NodePort draft #4255 with a fresh branch from current main.

Changes

• Keep the chart's existing ClusterIP Service and use the existing openshell_client role with its discovered IP on port 8080. No chart change, node address discovery or registration reconciliation.
• Gate automated K3s test boots on API, node, StatefulSet and functional CLI readiness; preserve interactive shell access after boot failures.
• Collect bounded K3s diagnostics on installation, preparation and conformance failure and upload them from integration jobs.
• Document fixture lifetime: the Service and registration persist in the installed disk; recreating the Service requires reprovisioning the fixture.

Testing

• Checks appropriate to the affected code and behavior pass: three Ansible syntax checks, shellcheck, Markdown lint, tmachine formatting, two unit tests, Nix runner/config build and diff checks.
• Unit tests cover optional boot preparation and backward-compatible installer configuration.
• Real local tmachine K3s cold install and cached boot: 8/8 conformance tests passed, 0 skipped, on the first execution (nextest a742182d-c4e0-45a4-b7e4-4da28c869693, 321.663s).
• A second fresh overlay replaced the gateway Pod: ClusterIP 10.43.206.224 and the full gateway registration stayed unchanged, no forwarding unit/process existed, and the targeted sandbox smoke scenario passed (nextest cd65f391-3691-48e9-9646-265a2f742aa6, 74.384s). Other scenarios were intentionally excluded by the smoke filter.
• A separate disposable fixture with zero gateway replicas exhausted bounded CLI readiness, preserved 93,276 bytes of diagnostics, and stopped before conformance. A fresh interactive boot of the same broken fixture still opened SSH and printed a verification marker while the gateway remained unavailable.

These were real Ubuntu 24.04 ARM64 QEMU/HVF guests with K3s v1.36.3+k3s1 and Agent Sandbox v0.5.0. The matching current-main runtime source is 9fd41e6bfd0a7e3e8ba090b2dd3be9fa8534039b (Release Dev run 37525799815); chart, harness and conformance archive came from this branch. No runtime binary source changed. Local logs, provenance and the validation report are retained under plans/ in the managed worktree. This is local validation; a full CI E2E campaign has not been run for this draft.

Scoped native ARM64 Clippy passes with the existing clippy::option_env_unwrap lint allowed for the unchanged VM firmware lookup. This removes a transport failure mode; it does not claim to fix the separate sandbox-provisioning race or establish that flakes cannot occur.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Relevant CI and troubleshooting guidance updated

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.