docs: add the Talos Director Upgrade Hosts page

director.yaml

@@ -24,6 +24,7 @@ navigation:
           pages:
             - "clusters.mdx"
             - "hosts.mdx"
+            - "upgrade-hosts.mdx"
             - "virtual-machines.mdx"
             - "migrate-from-vmware.mdx"
             - "what-happens-during-a-migration.mdx"

public/director/compute/clusters.mdx

@@ -13,7 +13,7 @@ Use the **Clusters** page to answer fleet-level questions — is every cluster h
 
 ## The All Clusters view
 
-Navigate to **Compute** > **Clusters** (`/clusters`) to see every cluster in one table.
+Navigate to **Compute** > **Host Clusters** (`/clusters`) to see every cluster in one table.
 
 ### Columns
 

public/director/compute/hosts.mdx

@@ -145,7 +145,7 @@ The history of system updates run on the host. Select **Run Update** to update t
 Updating a host reboots it, which stops every VM running on it. Put the host into maintenance mode and migrate its VMs first, and confirm the remaining hosts have capacity to take them.
 </Warning>
 
-To upgrade every host in a cluster without taking workloads down, use a rolling upgrade from the cluster's **Updates** tab instead. See [Clusters](/director/compute/clusters#updates).
+To upgrade every host in a cluster without taking workloads down, use a rolling upgrade from the cluster's **Updates** tab instead. See [Upgrade hosts](/director/compute/upgrade-hosts).
 
 ### Services
 

public/director/compute/upgrade-hosts.mdx

@@ -0,0 +1,216 @@
+---
+title: Upgrade Hosts
+description: Upgrade the Talos Linux operating system on one host or a whole cluster, upgrade the host agent and its witness, and roll back or troubleshoot a failed upgrade on Talos Director hosts.
+---
+
+import DirectorLimitedAvailability from '/snippets/director-limited-availability.mdx';
+
+<DirectorLimitedAvailability />
+
+A Talos Director host runs three components: the operating system, the host agent, and the witness. You upgrade them in two operations: the operating system on its own, and the host agent together with the witness. You upgrade from the **Updates** tab of a cluster, or of a single host.
+
+An operating system (OS) upgrade replaces the whole image rather than individual packages. Nothing can half-apply, and a host can be rolled back to its previous image.
+
+## Components
+
+Each component has a different upgrade impact. The witness has no upgrade of its own.
+
+| Component | What it is | Upgrade impact | Where to upgrade |
+| --- | --- | --- | --- |
+| **Operating system** | Talos Linux with KVM, QEMU, and libvirt, delivered as one immutable install image. | The host is power-cycled into the new image and is offline for several minutes. A rolling upgrade moves its VMs to other hosts first. For a single-host upgrade, you evacuate or stop the VMs yourself. | Cluster **Updates** tab (rolling), or host **Updates** tab (single host) |
+| **Host agent** | The container that carries out control plane instructions on the host. | Agent containers restart in place. VMs keep running. | Cluster **Updates** tab |
+| **Witness** | The heartbeat process that writes the host's datastore heartbeat for High Availability (HA). | Upgraded together with the agent. Containers restart in place. | Cluster **Updates** tab, as part of the agent upgrade |
+
+## Before you begin
+
+Check the following before you start an OS upgrade:
+
+- Confirm the cluster has spare capacity. Each host is taken out of service in turn, and its VMs must fit on the remaining hosts within the cluster's overcommit limits.
+- Check that every host shows **Connected** and **Agent tunnel: Up** on its **Overview** tab. The upgrade preflight fails if the control plane cannot reach a host.
+- Read the release notes for the target version in the host's **Available OS Versions** table, or in the **Target version** list when you start a rolling upgrade.
+- For a two-host cluster, expect one host to carry every VM during the upgrade.
+
+## Image lanes
+
+An image lane is the set of images a cluster's hosts are built from. The cluster's machine config profile names the lane, and every host in the cluster renders its configuration from it.
+
+To view the lane, navigate to **Compute** > **Host Clusters** > *cluster name* > **Updates**, and find the **Machine config profile** panel.
+
+| Field | Description |
+| --- | --- |
+| **Lane** | The lane name: `upstream` (stock Talos Linux, with the agent and witness as containers) or `fork` (a forked image, with the agent built in). |
+| **Install image** | The Talos Linux installer image, pinned by digest. This is the OS version hosts install. |
+| **Agent image** | The host agent container image, pinned by digest. |
+| **Witness image** | The datastore heartbeat image, pinned by digest. |
+| **Registry mirrors** | Registries hosts pull images through, or **none**. |
+
+To change the lane:
+
+1. Click **Edit lane**. Only the lane, its images, and the registry mirrors change here. The rest of the profile is kept as stored.
+2. Select the **Lane**.
+3. Under **Install image**, pick a version from the catalog, or enter a full image reference that includes the digest, in the form `<image_ref>@sha256:…`. Picking a catalog row fills in the reference.
+4. For the `upstream` lane, set the **Agent image** and **Witness image**. Both are container images and must be pinned by digest. They become the host's container documents after the lane change. The `fork` lane has no agent or witness image fields.
+5. Optional: click **Add mirror** to add a registry mirror. Enter the registry host, for example `docker.io`, and the mirror URL. Separate multiple URLs with commas.
+6. Click **Save profile**.
+
+<Note>
+Saving the profile does not reboot anything. Hosts move onto a new lane one at a time through an OS upgrade.
+</Note>
+
+## Upgrade the OS
+
+Upgrade a single host to test a new version, then run a rolling upgrade to roll it out across the cluster.
+
+### Upgrade a single host
+
+Upgrade one host to test a new version before rolling it out, or to bring a host that fell behind back in line. To upgrade every host in a cluster, see [Run a rolling upgrade](#run-a-rolling-upgrade).
+
+Navigate to **Compute** > **Hosts** > **All Hosts** > *host name* > **Updates**. The tab has three panels.
+
+#### Operating system
+
+This panel shows what the host is running now.
+
+| Field | Description |
+| --- | --- |
+| **Version** | The running Talos Linux version, for example `v1.14.0-upstream-9abd05af4`. |
+| **Stage** | The host's machine stage, for example `running`. |
+| **Ready** | Whether the host is ready. |
+| **Schematic** | The Image Factory schematic the image was built from, if any. |
+
+#### Last OS upgrade
+
+This panel shows the result of the most recent OS upgrade on the host.
+
+| Field | Description |
+| --- | --- |
+| **Status** | The result, for example **Done**. |
+| **Phase** | The final step reached, for example that the agent registered and the host was left in maintenance for the operator. |
+| **Target** | The target image digest and the version upgraded from. |
+| **Health** | Boot confirmation, for example that the new version booted over the overlay 11m45s after the handoff. |
+
+#### Available OS versions
+
+The catalog of versions this host can move to.
+
+| Column | Description |
+| --- | --- |
+| **Version** | The Talos version and build, for example `v1.14.1-upstream-2f86b9d2a-hw5`. |
+| **Channel** | The release channel, for example **stable**. |
+| **Installer Digest** | The short digest of the install image. |
+| **Release Notes** | What the build contains, such as QEMU, libvirt, microcode, and firmware versions. |
+
+A single-host upgrade does not evacuate the host's VMs for you. The upgrade drains storage and hard power-cycles the host, which kills any VMs still running on it. Before you start, put the host into maintenance mode and evacuate or stop its VMs. See [Manage hosts](/director/compute/hosts#manage-hosts).
+
+To upgrade the host:
+
+1. Put the host into maintenance mode, and evacuate or stop its VMs.
+2. Find the target version in **Available OS Versions**.
+3. Click **Upgrade…** on its row. The **Confirm OS Upgrade** dialog opens. It shows the host, the target version, and the host's etcd role, for example **client** or **seed**.
+4. Review the dialog. Talos Director stages the new image, hard power-cycles the host into it, and health-checks the result. Expect the host to be offline for several minutes.
+5. Optional: turn on **Auto-rollback on failed health check**. If the host comes back unhealthy, Talos Director stages the previous image and power-cycles the host again. This is off by default.
+6. Click **Start Upgrade**.
+7. Follow progress in **Last OS Upgrade**.
+8. When the upgrade finishes, take the host out of maintenance mode.
+
+<Note>
+After a single-host upgrade, the host is left in maintenance mode for the operator. It does not take new VMs until you take it out of maintenance mode.
+</Note>
+
+#### Move a host to the cluster's lane
+
+An `upstream`-lane image needs a lane change if the host is still running the `fork` image. A host already on the `upstream` line does not need one. To move the host to the profile's lane, select **Move this host to the profile's lane (allow_lane_change)** in the **Confirm OS Upgrade** dialog. A lane change requires all of the following:
+
+- The cluster's machine config profile is on the `upstream` lane, with the agent and witness images set. See [Image lanes](#image-lanes).
+- The deployment has a SideroLink hub, and the host is finished over the overlay.
+- The host is in maintenance mode with no running VMs, and every stored VM can be defined on the `upstream` image.
+
+**Force** does not override these requirements.
+
+#### Force an upgrade
+
+If the host is not in maintenance mode, or its live state can't be read, the dialog warns you. Turn on **Force: override blocking pre-flight checks** to proceed anyway. Force overrides two checks:
+
+- **Maintenance-mode requirement:** any VMs still running on the host are killed by the hard power-cycle.
+- **Pre-flight OS state read:** the upgrade proceeds even if the host's live state can't be read, and boot-assessment revert detection is disabled for the job.
+
+Force never skips the digest-pinned image target or the post-upgrade health verification. Each overridden check is recorded on the job and audit-logged with your username.
+
+<Warning>
+Use **Force** only when you have accepted that running VMs on the host will be killed.
+</Warning>
+
+### Run a rolling upgrade
+
+A rolling upgrade moves every host in the cluster to a new Talos Linux image, one host at a time, so VMs keep running throughout. To test a version on one host first, see [Upgrade a single host](#upgrade-a-single-host).
+
+1. Navigate to **Compute** > **Host Clusters** > *cluster name* > **Updates**.
+2. In the **Operating System — Rolling Upgrade** panel, confirm the host count under **Applies to all *N* hosts**.
+3. Click **Start Rolling Upgrade**. The **Start Rolling OS Upgrade** dialog opens and lists the hosts in upgrade order. The host marked **seed** is the etcd seed, which the cluster bootstraps from, so it upgrades first while the cluster is transitional. The remaining hosts follow.
+4. Under **Target version**, select a catalog version. Its release notes appear below the list.
+5. Optional: turn on **Auto-rollback a host on failed health check** so that if a host comes back unhealthy, Talos Director stages the previous image and power-cycles the host again. This is off by default.
+6. Click **Start**.
+7. Click **Refresh** to follow progress. The panel shows the status, the target image digest, hosts completed (for example `1/2 hosts`), and the current phase.
+
+For each host, Talos Director:
+
+1. Runs a preflight check that reads the host's OS status through the agent tunnel.
+2. Drains the host into maintenance mode, which evacuates its VMs to other hosts.
+3. Stages the new install image and hard power-cycles the host into it.
+4. Waits for the host to boot and the host agent to register again.
+5. Health-checks the host and brings it back out of maintenance mode.
+6. Moves on to the next host.
+
+<Warning>
+If any host fails, the upgrade stops. The failed host stays in maintenance mode for inspection, and hosts not yet reached stay on the old version. See [Troubleshoot a failed upgrade](#troubleshoot-a-failed-upgrade).
+</Warning>
+
+### Roll back a host
+
+Click **Roll Back** in the **Operating System** panel to return the host to its previous image.
+
+## Upgrade the host agent
+
+An agent upgrade restarts the host agent and witness containers in place. VMs keep running, and hosts do not reboot or enter maintenance mode. The witness ships with the agent, so there is no separate witness upgrade: you choose its image in the same run.
+
+### Run a rolling upgrade
+
+1. Navigate to **Compute** > **Host Clusters** > *cluster name* > **Updates**.
+2. In the **Agents — Rolling Upgrade** panel, check the host table for each host's current agent version and image.
+3. Complete the fields below.
+4. Click **Start**. The button stays disabled until you select a target agent image.
+
+| Field | Default | Description |
+| --- | --- | --- |
+| **Target agent image** | None | The agent image to roll out, listed by version, channel, and digest. The current pin is labeled **(current pin)**. |
+| **Witness image** | Keep current | The witness image to roll out with the agent. |
+| **Hosts at a time** | 1 | How many hosts upgrade in parallel. The minimum is 1. |
+
+Each run appears as a job, with who started it, hosts completed, and the agent and witness digests before and after. Each host moves through these phases:
+
+| Phase | What happens |
+| --- | --- |
+| **Queued** | The host waits for its turn. |
+| **Applying** | The new image pin is applied to the host. |
+| **Containers** | The agent and witness containers restart on the new image. |
+| **Registering** | The agent reconnects to the control plane. |
+| **Verifying** | The control plane confirms the host runs the target image. |
+| **Done** | The host is on the new version. |
+
+<Warning>
+Increasing the **Hosts at a time** field finishes sooner, but more hosts lose their management connection at once. High Availability treats a host whose agent is down but whose datastore heartbeat is fresh as isolated, not failed, so VMs are not restarted. Even so, keep it at 1 on production clusters so that one bad image affects only one host.
+</Warning>
+
+### Revert an agent upgrade
+
+Click **Revert** in the **Agents — Rolling Upgrade** panel to return hosts to the previously pinned agent image. The panel header shows the current **Pinned agent image** and **Witness** digests.
+
+## Troubleshoot a failed upgrade
+
+When a rolling OS upgrade fails, the panel shows **Failed**, the hosts completed, the host it stopped on, and the full error.
+
+| Symptom | Likely cause | What to do |
+| --- | --- | --- |
+| `pre-flight status read: ... tunnel unavailable` | The control plane cannot reach the host agent through the reverse tunnel. | Check **Agent tunnel** on the host's **Overview** tab. Wait for it to show **Up**, then start the upgrade again. |
+| `dial tcp <address>:50000: i/o timeout` or `network is unreachable` | The Talos API on port 50000 is unreachable over both the overlay and the management (underlay) address. | Check the host's management network and BMC. Confirm the host is powered on and booted. |
+| Phase ends with `host left in maintenance mode for inspection` | The upgrade stopped on this host on purpose. | Fix the cause, then take the host out of maintenance mode or start the upgrade again. |

public/docs.json

@@ -2885,6 +2885,7 @@
             "pages": [
               "director/compute/clusters",
               "director/compute/hosts",
+              "director/compute/upgrade-hosts",
               "director/compute/virtual-machines",
               "director/compute/migrate-from-vmware",
               "director/compute/what-happens-during-a-migration",