Pulse 6.2.1: Agent Doctor Repair Authentication created additional agent identities on a healthy Proxmox agent

[behez0n]

Sharing another field finding from a Pulse 6.2.1 migration.

This came up during the same migration where I documented the NFS/statfs issue in Discussion #1747, but this appears to be a separate agent identity / credential lifecycle issue.

This is not a support request; the environment is healthy now. I'm documenting the behavior because it was reproducible and the final state gave some useful clues about what was happening.

Environment

• Pulse Server: 6.2.1
• Pulse Unified Agent: 6.2.1
• Proxmox VE 8.x
• Proxmox cluster member
• Linux systemd agent
• Single pulse-agent process on the host

For clarity, I'll call the different identities agent-A, agent-B, etc. rather than posting the actual UUIDs.

Initial symptom

The host agent was operational and actively reporting.

The canonical connection state showed:

state: active
agentVersion: v6.2.1
credentialStatus: verified
credentialHealth.status: verified
versionDrift: current

Using the token stored locally by the agent to query its own ID also returned:

HTTP 200
connected: true
status: online

However, Agent Doctor reported the same agent as Critical:

The credential last used by this agent no longer exists on this Pulse server.
Credential status: missing or revoked

and offered:

Repair authentication

So there was already a contradiction between the active connection state and Agent Doctor:

/api/connections -> credential verified
Agent Doctor     -> credential missing/revoked

What happened when using Repair Authentication

I generated the fresh credential from Agent Doctor and ran the repair command it provided.

The generated installer invocation was effectively:

--token-file <fresh-token>
--update
--enable-proxmox
--proxmox-type pve
--non-interactive

In the command generated for this host, there was no explicit --agent-id.

The installer completed successfully:

Existing installation detected: v6.2.1
Upgrading to: v6.2.1
Token stored securely
Clearing Proxmox state for fresh registration
Proxmox pve node registered with Pulse
Upgrade complete

But instead of converging on the existing identity, another suffixed host-agent identity appeared.

Agent Doctor then reported:

agent_credential_missing
duplicate_host_agent_installation

even though there was still only one pulse-agent process running on the machine.

Across the authentication repair attempts during this incident, multiple generations of suffixed agent IDs were created for the same physical host.

At that point I stopped using Repair Authentication.

An interesting detail

During recovery I compared the active connection with the stale identities.

The active agent was still:

state: active
credentialStatus: verified
status: online

and reports were arriving normally.

Looking at the persisted host continuity state showed that the active identity and one of the stale identities had:

same machineId
same tokenId
different hostId / agent ID

For example, conceptually:

agent-A (stale)
  machineId: machine-X
  tokenId:   token-Y

agent-C (active)
  machineId: machine-X
  tokenId:   token-Y

This also explained another observation: using the current local token to perform an explicit lookup of both the active ID and the stale ID returned HTTP 200.

So I did not want to blindly DELETE the stale agent through the API, since the stale and active identities were still associated with the same credential.

Recovery

The identity Pulse was actively receiving reports from via /api/connections was selected as canonical.

The local host state was normalized so these all agreed:

/var/lib/pulse-agent/agent-id
PULSE_AGENT_ID in connection.env
systemd --agent-id

The host then consistently reported under that single identity:

HTTP 200
connected: true
status: online
credentialStatus: verified

After backing up the Pulse data, I also removed the stale continuity entry while Pulse was stopped.

I want to emphasize that I treated this as recovery of an already inconsistent installation, not as a general recommended maintenance procedure.

Once the stale continuity identity was gone, Agent Doctor immediately changed to:

Status: Healthy
rawStatus: online

and both of these disappeared:

agent_credential_missing
duplicate_host_agent_installation

The final active connection remained:

state: active
credentialStatus: verified
agentVersion: v6.2.1

Takeaway

My main takeaway is that on 6.2.1, if Agent Doctor reports:

agent_credential_missing

while /api/connections simultaneously reports the same live agent as:

credentialStatus: verified

I would verify the current identity and credential mapping before repeatedly running Repair Authentication.

In this case, using the repair workflow while those two views disagreed resulted in additional host-agent identities rather than converging on the existing one.

A couple of possible safeguards that might be worth considering:

1. Preserve/pass the already-known canonical agent-id in the Repair Authentication installer command.

2. If Agent Doctor believes the credential is missing but the active connection reports a recently verified credential, surface that inconsistency before offering an identity-changing repair action.

Related discussion

Discussion #1661 covers some related re-enrollment / removed-agent identity behavior.

My case appears somewhat different: this was the same physical Proxmox host with a stable machine identity, and the live agent remained online and credential-verified while Agent Doctor reported the credential as missing.

Sharing this in case the details are useful for the agent lifecycle / Agent Doctor work.

[pulse-triage[bot]]

Good writeup, both halves check out against source. The false Critical came from the doctor judging credentials against a registry view captured at server startup, so a token minted after startup never entered it, repair could not clear the verdict and only minted more tokens the stale view had never seen. That is fixed in v6.3.0-rc.3, and main additionally refuses to call a credential that just authenticated an outage.

The duplicate identities were helped along by the generated Unix repair command carrying no --agent-id, unlike the Windows one which pins PULSE_AGENT_ID and PULSE_HOSTNAME. Fixed in ad91258, the repair command now pins the existing identity and hostname, landing in the next release after rc.3. Your recovery order, normalising agent-id, connection.env, and the unit flag to the identity /api/connections reports, was the right one.

How Pulse handles triage