fix(deploy): release a settled idempotency key so redeploys aren't permanently blocked

Redeploying a workflow could fail forever with "Idempotency key was already used
for a different deployment request", with no way to clear it from the workflow.

The deploy orchestrator scopes one idempotency key to a whole request
(`idempotencyKey: params.requestId`) while the request hash is derived from
workflow state. That is fine when one request performs one deployment, but the
agent path issues several deployments per chat request: deploy, edit, redeploy.
The second deployment presents the same key with a new hash, so
prepareWorkflowDeployment took the conflict branch — and kept taking it, because
the key was never released.

Two problems, both in the same block:

- The request-hash comparison ran BEFORE the terminal-status check, so the
  release path was unreachable whenever the hash differed. Its own comment
  ("a terminally failed or superseded attempt releases its idempotency key")
  therefore never applied to the case that needed it.
- A completed (`active`) operation was not treated as settled at all, so a
  successful deployment reserved its key permanently.

Reordered so the key is released whenever the prior attempt is settled
(`active`, `failed`, or `superseded`) and the request genuinely differs. A
conflict is now raised only when a *different* deployment is still in flight
(`pending`/`preparing`/`activating`), which is the ambiguous case idempotency
is meant to guard. True duplicate submissions (same key, same hash) still reuse
the existing operation and never redeploy.

Observed on staging: one chat request (58a47006) deployed three workflows, then
its redeploy of the first failed five times in a row with this error.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018asmKsWQ5Vi7T7wD9uHofz

Author: TheodoreSpeaks

apps/sim/lib/workflows/persistence/deployment-operations.test.ts

@@ -308,6 +308,126 @@ describe('deployment operation persistence', () => {
     expect(dbChainMockFns.insert).toHaveBeenCalled()
   })
 
+  /**
+   * Callers scope one idempotency key to a whole request (the deploy orchestrator
+   * passes `requestId`), so a single request that deploys the same workflow twice —
+   * e.g. the agent redeploying after edits within one chat request — presents the
+   * same key with a new request hash. Once the first deployment settled, the key
+   * must be reassignable or that caller is locked out permanently.
+   */
+  it('releases the key of a completed (active) operation when the request differs', async () => {
+    const completed = operationRow({
+      id: 'operation-active',
+      idempotencyKey: 'deploy-1',
+      requestHash: 'original-hash',
+      status: 'active',
+      generation: 2,
+    })
+    const fresh = operationRow({ id: 'operation-fresh', generation: 3 })
+    mockGenerateId.mockReturnValueOnce('version-fresh').mockReturnValueOnce('operation-fresh')
+    dbChainMockFns.for.mockResolvedValueOnce([{ id: WORKFLOW_ID, archivedAt: null }])
+    dbChainMockFns.limit
+      .mockResolvedValueOnce([completed])
+      .mockResolvedValueOnce([])
+      .mockResolvedValueOnce([{ maxVersion: 2 }])
+      .mockResolvedValueOnce([{ maxGeneration: 2 }])
+    dbChainMockFns.returning.mockResolvedValueOnce([fresh])
+
+    const result = await prepareWorkflowDeployment({
+      workflowId: WORKFLOW_ID,
+      actorId: 'user-1',
+      requestHash: 'different-hash',
+      idempotencyKey: 'deploy-1',
+      workflowState: workflowState(),
+      readinessComponents: ['webhooks'],
+    })
+
+    expect(result).toEqual({ success: true, operation: fresh, reused: false })
+    expect(dbChainMockFns.set).toHaveBeenCalledWith(
+      expect.objectContaining({ idempotencyKey: null })
+    )
+    expect(dbChainMockFns.insert).toHaveBeenCalled()
+  })
+
+  it('releases the key of a failed operation even when the request differs', async () => {
+    const failed = operationRow({
+      id: 'operation-failed',
+      idempotencyKey: 'deploy-1',
+      requestHash: 'original-hash',
+      status: 'failed',
+      generation: 2,
+    })
+    const fresh = operationRow({ id: 'operation-fresh', generation: 3 })
+    mockGenerateId.mockReturnValueOnce('version-fresh').mockReturnValueOnce('operation-fresh')
+    dbChainMockFns.for.mockResolvedValueOnce([{ id: WORKFLOW_ID, archivedAt: null }])
+    dbChainMockFns.limit
+      .mockResolvedValueOnce([failed])
+      .mockResolvedValueOnce([])
+      .mockResolvedValueOnce([{ maxVersion: 2 }])
+      .mockResolvedValueOnce([{ maxGeneration: 2 }])
+    dbChainMockFns.returning.mockResolvedValueOnce([fresh])
+
+    const result = await prepareWorkflowDeployment({
+      workflowId: WORKFLOW_ID,
+      actorId: 'user-1',
+      requestHash: 'different-hash',
+      idempotencyKey: 'deploy-1',
+      workflowState: workflowState(),
+      readinessComponents: ['webhooks'],
+    })
+
+    expect(result).toEqual({ success: true, operation: fresh, reused: false })
+    expect(dbChainMockFns.insert).toHaveBeenCalled()
+  })
+
+  it('still rejects a different request while an operation is mid-activation', async () => {
+    dbChainMockFns.for.mockResolvedValueOnce([{ id: WORKFLOW_ID, archivedAt: null }])
+    dbChainMockFns.limit.mockResolvedValueOnce([
+      operationRow({
+        idempotencyKey: 'deploy-1',
+        requestHash: 'original-hash',
+        status: 'activating',
+      }),
+    ])
+
+    const result = await prepareWorkflowDeployment({
+      workflowId: WORKFLOW_ID,
+      actorId: 'user-1',
+      requestHash: 'different-hash',
+      idempotencyKey: 'deploy-1',
+      workflowState: workflowState(),
+    })
+
+    expect(result).toEqual({
+      success: false,
+      reason: 'idempotency_conflict',
+      error: 'Idempotency key was already used for a different deployment request',
+    })
+    expect(dbChainMockFns.insert).not.toHaveBeenCalled()
+    expect(dbChainMockFns.update).not.toHaveBeenCalled()
+  })
+
+  it('reuses a completed (active) operation for a true duplicate request', async () => {
+    const completed = operationRow({
+      idempotencyKey: 'deploy-1',
+      requestHash: 'hash-1',
+      status: 'active',
+    })
+    dbChainMockFns.for.mockResolvedValueOnce([{ id: WORKFLOW_ID, archivedAt: null }])
+    dbChainMockFns.limit.mockResolvedValueOnce([completed])
+
+    const result = await prepareWorkflowDeployment({
+      workflowId: WORKFLOW_ID,
+      actorId: 'user-1',
+      requestHash: 'hash-1',
+      idempotencyKey: 'deploy-1',
+      workflowState: workflowState(),
+    })
+
+    expect(result).toEqual({ success: true, operation: completed, reused: true })
+    expect(dbChainMockFns.insert).not.toHaveBeenCalled()
+  })
+
   it('rejects stale generation callbacks before mutating legacy deployment state', async () => {
     dbChainMockFns.for
       .mockResolvedValueOnce([{ id: WORKFLOW_ID }])

apps/sim/lib/workflows/persistence/deployment-operations.ts

@@ -751,22 +751,44 @@ async function prepareOperation(
         )
         .limit(1)
       if (existing) {
-        if (existing.requestHash !== requestHash) {
+        /**
+         * `spent` — the attempt ended without deploying, so it can never be
+         * handed back as a successful duplicate.
+         * `settled` — the attempt reached any terminal state (including a
+         * completed `active` deploy), so no work is still in flight under this
+         * key.
+         */
+        const spent = existing.status === 'failed' || existing.status === 'superseded'
+        const settled = spent || existing.status === 'active'
+
+        if (existing.requestHash === requestHash) {
+          // A genuine duplicate submission: hand back the operation that is
+          // already doing (or has already done) the work.
+          if (!spent) {
+            return { success: true, operation: existing, reused: true }
+          }
+        } else if (!settled) {
+          /**
+           * A *different* deployment is still in flight under this key. Which
+           * one the caller meant is genuinely ambiguous, so refuse rather than
+           * race two payloads through the same key.
+           */
           return {
             success: false,
             reason: 'idempotency_conflict',
             error: 'Idempotency key was already used for a different deployment request',
           }
         }
-        if (existing.status !== 'failed' && existing.status !== 'superseded') {
-          return { success: true, operation: existing, reused: true }
-        }
         /**
-         * A terminally failed or superseded attempt releases its idempotency
-         * key: duplicate-submission protection must never pin a retry to a
-         * spent attempt — the caller would get success with no live work
-         * behind it. The key moves to the fresh operation created below so
-         * later duplicates of the same request reuse that one instead.
+         * The key belongs to a settled attempt, so it is free to reassign:
+         * duplicate-submission protection must never pin a retry to a spent
+         * attempt (the caller would get success with no live work behind it),
+         * nor permanently reserve a key that a later, genuinely different
+         * deployment needs. Callers that scope one key to a whole request —
+         * e.g. the agent redeploying an edited workflow within a single chat
+         * request — would otherwise be locked out for good once the first
+         * deployment settled. The key moves to the fresh operation created
+         * below so later duplicates of the same request reuse that one instead.
          */
         await tx
           .update(workflowDeploymentOperation)