Merge remote-tracking branch 'origin/staging' into feature/pi-exa

# Conflicts:
#	packages/db/migrations/meta/0268_snapshot.json
#	packages/db/migrations/meta/_journal.json
#	scripts/check-api-validation-contracts.ts

Author: Bill Leoutsakos

apps/docs/content/docs/en/platform/enterprise/data-retention.mdx

@@ -1,34 +1,50 @@
 ---
 title: Data Retention
-description: Control how long execution logs, deleted resources, and copilot data are kept before permanent deletion
+description: Control how long execution logs, deleted resources, and Chat data are kept before permanent deletion — and redact PII from workflow data
 ---
 
+import { Callout } from 'fumadocs-ui/components/callout'
 import { FAQ } from '@/components/ui/faq'
 import { Image } from '@/components/ui/image'
 
-Data Retention lets organization owners and admins on Enterprise plans configure how long three categories of data are kept before they are permanently deleted. The configuration applies to every workspace in the organization.
+Data Retention lets organization owners and admins on Enterprise plans control two things:
+
+1. **Retention periods** — how long execution logs, soft-deleted resources, and Chat data are kept before they are permanently deleted.
+2. **PII redaction** — masking personally identifiable information out of your workflow data at up to three points: workflow input, block outputs, and logs.
+
+Both are configured once at the **organization level** and apply to every workspace, with optional **per-workspace overrides** for workspaces that need different rules.
 
 ---
 
 ## Setup
 
 Go to **Settings → Enterprise → Data Retention** in your workspace.
 
-<Image src="/static/enterprise/data-retention.png" alt="Data Retention settings showing three dropdowns — Log retention, Soft deletion cleanup, and Task cleanup — each set to Forever" width={900} height={500} />
+<Image src="/static/enterprise/data-retention.png" alt="Data Retention settings showing the Retention policies list with the Organization default row and its summary of retention periods and PII stages" width={900} height={537} />
+
+The page shows your **retention policies** as a list:
 
-You will see three independent settings, each with the same set of options: **1 day, 3 days, 7 days, 14 days, 30 days, 60 days, 90 days, 180 days, 1 year, 5 years,** or **Forever**.
+- The **Organization** row (tagged *Default*) holds the settings that apply to every workspace without its own override.
+- Each **workspace override** row below it applies to one or more specific workspaces.
 
-Setting a period to **Forever** means that category of data is never automatically deleted.
+Open a row to edit it, or click **Add override** to create a workspace override. Each policy has a **Retention** section, and — when PII redaction is enabled — a **PII redaction** section.
 
 ---
 
-## Settings
+## Retention periods
+
+Each policy has three independent retention settings, each with the same set of options: **1 day, 3 days, 7 days, 14 days, 30 days, 60 days, 90 days, 180 days, 1 year, 5 years,** or **Forever**.
+
+Setting a period to **Forever** means that category of data is never automatically deleted. On a workspace override, each field can also be set to **Inherit from organization** to fall back to the organization default for just that field.
 
 ### Log retention
 
-Controls how long **workflow execution logs** are kept.
+Controls how long **execution logs** are kept.
 
-When the retention period expires, execution log records are permanently deleted, along with any files associated with those executions stored in cloud storage.
+When the retention period expires, log records are permanently deleted, along with any files associated with those executions in cloud storage. This covers:
+
+- Workflow execution logs
+- Background job logs (deployed APIs, schedules, and webhooks)
 
 ### Soft deletion cleanup
 
@@ -40,33 +56,83 @@ Resources covered:
 
 - Workflows
 - Workflow folders
-- Knowledge bases
+- Knowledge bases (and their documents)
 - Tables
 - Files
 - MCP server configurations
 - Agent memory
+- Chat conversations
 
 ### Task cleanup
 
-Controls how long **Mothership data** is kept, including:
+Controls how long **Chat data** is kept, including:
 
-- Copilot chats and run history
+- Chat conversations and run history
 - Run checkpoints and async tool calls
-- Inbox tasks (Sim Mailer)
+- Inbox tasks
 
 Each setting is independent. You can configure a short log retention period alongside a long soft deletion cleanup period, or any combination that fits your compliance requirements.
 
 ---
 
-## Organization-wide configuration
+## PII redaction
+
+When PII redaction is enabled for your organization, each policy gains a **PII redaction** section that masks personally identifiable information — names, emails, phone numbers, credit-card numbers, national IDs, and more — from your workflow data. Sim detects and masks PII with [Microsoft Presidio](https://microsoft.github.io/presidio/); each match is replaced with a placeholder token such as `<EMAIL_ADDRESS>`.
+
+Redaction is configured per **stage** — the point in a run where masking is applied. Select a stage, then choose which entity types and custom patterns to redact for it:
+
+| Stage | What it does |
+|---|---|
+| **Logs** | Redacts workflow logs when they are persisted. Observability-only — the workflow still runs on the original data. |
+| **Workflow input** | Redacts the workflow input **before execution**. The workflow runs on the masked data, which may change its output. |
+| **Block outputs** | Masks every block output **before the next block reads it**. Runs in-flight and may change output and execution performance. |
+
+<Image src="/static/enterprise/data-retention-pii.png" alt="PII redaction section with the Block outputs stage selected, showing the entity type grid grouped by Common, United States, United Kingdom, and Other regions" width={900} height={537} />
+
+<Callout type="warn">
+The **Workflow input** and **Block outputs** stages alter what the workflow computes on, not just what is stored. Redacted data is masked during the run and may affect workflow output. Enable them only where that trade-off is acceptable.
+</Callout>
+
+### Entity types and language
+
+For each stage, choose the **entity types** to redact from the searchable grid. They are grouped as:
+
+- **Common** — person name, email, phone, credit card, IP address, URL, IBAN, crypto wallet, medical license, VIN
+- **United States** — SSN, passport, driver's license, bank account, ITIN
+- **United Kingdom** — NHS number, National Insurance number
+- **Other regions** — Singapore, Australian, and Indian identifiers
+
+The **Block outputs** stage is restricted to regex- and checksum-based recognizers, so it can run in-flight over large payloads without a performance penalty. Types that need name-model detection — person name, location, date or time — are not offered for that stage.
+
+Detection is language-aware: pick the **language** whose recognizers should apply. English, Spanish, Italian, Polish, and Finnish are supported, and the grid filters to the identifiers available for the selected language.
+
+<Callout type="warn">
+Some recognizers match loosely and over-redact — US Social Security Number, US bank account number, and Date or time have no checksum and match aggressively. Enable these only where false positives are acceptable.
+</Callout>
+
+### Custom patterns
 
-Retention is configured at the **organization level**. A single configuration applies to every workspace in the organization — there are no per-workspace overrides.
+Beyond the built-in entity types, each stage can redact anything a **regular expression** matches — employee IDs, internal URLs, ticket numbers. Give each pattern a name, a regex, and a replacement token; every match is replaced with the replacement text wrapped in angle brackets (e.g. `EMPLOYEE_ID` → `<EMPLOYEE_ID>`).
+
+---
+
+## Per-workspace overrides
+
+Retention and PII redaction are configured at the **organization level** and apply to every workspace by default. When a workspace needs different rules, add a **workspace override**.
+
+<Image src="/static/enterprise/data-retention-override.png" alt="Add workspace override panel showing the workspace picker, retention fields set to Inherit from organization, and the PII redaction Inherit or Override switch" width={900} height={537} />
+
+- An override targets one or more workspaces (a workspace can belong to only one override).
+- Each **retention** field either sets its own period or **inherits** the organization value.
+- **PII redaction** on an override is either **Inherit** (use the organization's redaction) or **Override** (workspace-specific). Choosing Override replaces the organization's redaction rules entirely for that workspace — it is not merged stage-by-stage.
+
+Removing an override returns its workspaces to the organization defaults.
 
 ---
 
 ## Defaults
 
-By default, all three settings are unconfigured — no data is automatically deleted in any category until you configure it. Setting a period to **Forever** has the same effect as leaving it unconfigured, but makes the intent explicit and allows you to change it later without saving from scratch.
+By default, retention settings are unconfigured — no data is automatically deleted in any category, and no PII is redacted, until you configure it. Setting a retention period to **Forever** has the same effect as leaving it unconfigured, but makes the intent explicit and lets you change it later without configuring from scratch.
 
 ---
 
@@ -84,13 +150,17 @@ By default, all three settings are unconfigured — no data is automatically del
     answer: "No. Once the soft deletion cleanup period expires and the cleanup job runs, resources are permanently deleted and cannot be recovered."
   },
   {
-    question: "Does the retention period apply to all workspaces in my organization?",
-    answer: "Yes. Retention is configured once per organization and applies to every workspace in the organization."
+    question: "Does a policy apply to all workspaces?",
+    answer: "The organization policy applies to every workspace that does not have its own override. Add a per-workspace override to give specific workspaces different retention periods or PII redaction rules."
   },
   {
     question: "What happens if I shorten the retention period?",
     answer: "The next cleanup job will delete any data that is older than the new, shorter period — including data that would have been kept under the previous setting. Shortening the period is irreversible for data that falls outside the new window."
   },
+  {
+    question: "Does PII redaction change how my workflows run?",
+    answer: "The Logs stage does not — it only masks data as it is persisted, so the workflow runs on the original data. The Workflow input and Block outputs stages do: they mask data in-flight, so the workflow computes on the redacted values, which can change its output."
+  },
   {
     question: "What is the minimum retention period?",
     answer: "1 day (24 hours)."
@@ -105,10 +175,28 @@ By default, all three settings are unconfigured — no data is automatically del
 
 ## Self-hosted setup
 
-### Environment variables
+### Retention periods
 
 ```bash
 NEXT_PUBLIC_DATA_RETENTION_ENABLED=true
+DATA_RETENTION_ENABLED=true
+```
+
+Once enabled, retention settings are configurable through **Settings → Enterprise → Data Retention** the same way as Sim Cloud.
+
+### PII redaction
+
+PII redaction runs against a standalone [Presidio](https://microsoft.github.io/presidio/) service. Deploy it (see `apps/pii`) and point Sim at it, then enable the redaction surfaces:
+
+```bash
+# The Presidio service exposing /analyze and /anonymize
+PII_URL=http://localhost:5001
+
+# Expose the log-redaction stage and the Data Retention PII section
+PII_REDACTION=true
+
+# Additionally expose the execution-altering stages (Workflow input, Block outputs)
+PII_GRANULAR_REDACTION=true
 ```
 
-Once enabled, data retention settings are configurable through **Settings → Enterprise → Data Retention** the same way as Sim Cloud.
+`PII_GRANULAR_REDACTION` layers on top of `PII_REDACTION` — with only `PII_REDACTION` enabled, just the **Logs** stage is configurable.

apps/docs/content/docs/en/platform/enterprise/meta.json

@@ -3,6 +3,7 @@
   "pages": [
     "index",
     "sso",
+    "verified-domains",
     "session-policies",
     "access-control",
     "custom-blocks",

apps/docs/content/docs/en/platform/enterprise/verified-domains.mdx

@@ -0,0 +1,57 @@
+---
+title: Verified Domains
+description: Prove ownership of your email domains before configuring single sign-on
+---
+
+import { Callout } from 'fumadocs-ui/components/callout'
+import { FAQ } from '@/components/ui/faq'
+
+Verified Domains let organization owners and admins on Enterprise plans prove they control an email domain (like `acme.com`) with a DNS TXT record. Verifying a domain is the security precondition for configuring single sign-on for it.
+
+<Callout type="warning">
+  Configuring SSO for a domain requires it to be verified first. Verifying proves your organization controls the domain — without it, anyone could point another company's domain at their own identity provider. Domains you had already configured for SSO are automatically treated as verified.
+</Callout>
+
+---
+
+## Verify a domain
+
+Go to **Settings → Security → Verified domains** in your organization settings.
+
+1. Enter the domain, for example `acme.com`, and click **Add domain**.
+2. Sim shows a DNS **TXT record** to publish — a host (`_sim-challenge.acme.com`) and a unique value (`sim-domain-verification=…`).
+3. Add that TXT record at your DNS provider.
+4. Click **Verify**. Sim looks up the record; on success the domain is marked **Verified**.
+
+DNS changes can take up to 48 hours to propagate — if verification does not succeed immediately, wait and retry. You can remove the TXT record after the domain is verified; the verification persists.
+
+Add each domain you own separately. Subdomains (`eng.acme.com`) are verified independently of the apex.
+
+---
+
+## FAQ
+
+<FAQ
+  items={[
+    {
+      question: 'Where does the TXT record go?',
+      answer:
+        'On a dedicated host, _sim-challenge.<your-domain>, rather than the root of your domain — this avoids colliding with your SPF, DMARC, or other root TXT records.',
+    },
+    {
+      question: 'What happens to domains we already use for SSO?',
+      answer:
+        'They are automatically treated as verified, so existing single sign-on keeps working with no action needed.',
+    },
+    {
+      question: 'Can two organizations verify the same domain?',
+      answer:
+        'No. A verified domain belongs to exactly one organization. Once verified, another organization cannot claim it.',
+    },
+    {
+      question: 'What if I remove a verified domain?',
+      answer:
+        'You lose the ownership proof, so you cannot configure SSO for that domain until you re-add and re-verify it. Removing it does not sign anyone out — an already-configured SSO provider keeps working.',
+    },
+  ]}
+/>