Skip to content

Commit 78dff13

Browse files
committed
Add docs for qurantining from cli / mcp
1 parent 1a3fde0 commit 78dff13

3 files changed

Lines changed: 122 additions & 1 deletion

File tree

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
---
2+
title: Modify quarantine rules with AI
3+
description: Ask an AI coding agent to create and deactivate Endform quarantine rules
4+
sidebar:
5+
order: 8
6+
---
7+
8+
import Conversation from "../../../components/conversation/Conversation.astro";
9+
import UserMessage from "../../../components/conversation/UserMessage.astro";
10+
import ToolCall from "../../../components/conversation/ToolCall.astro";
11+
import AssistantMessage from "../../../components/conversation/AssistantMessage.astro";
12+
13+
After you [connect the Endform MCP server](/docs/reference/mcp-server), your coding agent can inspect recent test runs and manage quarantine rules. The agent uses `query_samples` to retrieve the exact persisted suite and test IDs needed to create a rule.
14+
15+
<Conversation title="Quarantine a failing test" open>
16+
<UserMessage>
17+
Quarantine the failing saved-card checkout test on the `main` branch while we investigate. Set it to expire on September 23, 2026 and explain exactly what the rule will match.
18+
</UserMessage>
19+
20+
<ToolCall name="query_samples" summary="find recent failures for the saved-card checkout test" />
21+
<ToolCall name="list_quarantine_rules" summary="check the current quarantine rules" />
22+
<ToolCall name="create_quarantine_rule" summary="quarantine the exact test and branch until September 23" />
23+
24+
<AssistantMessage>
25+
I created a rule for `checkout.spec.ts › saved card › completes payment` in the sampled suite. It matches that exact persisted test ID on the `main` branch across all Playwright projects and expires on September 23, 2026.
26+
</AssistantMessage>
27+
</Conversation>
28+
29+
<Conversation title="Remove quarantine after a fix">
30+
<UserMessage>
31+
Check whether the saved-card checkout test has recovered over the last seven days. If it has, deactivate its quarantine rule.
32+
</UserMessage>
33+
34+
<ToolCall name="query_samples" summary="inspect the test's recent outcomes and retries" />
35+
<ToolCall name="list_quarantine_rules" summary="find the active rule for the test" />
36+
<ToolCall name="deactivate_quarantine_rule" summary="deactivate the exact quarantine rule" />
37+
38+
<AssistantMessage>
39+
The test passed its last 18 runs without a retry, including six runs after the fix. I deactivated its quarantine rule, so it will return to normal failure behavior on new suite runs.
40+
</AssistantMessage>
41+
</Conversation>
42+
43+
See [Quarantine tests](/docs/guides/quarantine-tests) for how rules affect test execution and suite results. The complete tool reference is in [Endform MCP server](/docs/reference/mcp-server#manage-quarantine-rules).

‎src/content/docs/guides/quarantine-tests.mdx‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,34 @@ counts each matching test once across projects and does not consider other
4545
quarantine rules. Future runs may contain different tests, so the preview is a
4646
guide to the rule's scope.
4747

48+
## Manage rules from the CLI or AI
49+
50+
Use `endform samples` to find the exact persisted suite and test IDs, then use
51+
them to create a rule:
52+
53+
```bash
54+
npx endform@latest samples test-runs --where 'testRun.outcome = fail' --limit 10
55+
npx endform@latest quarantine create \
56+
--suite-id suite_123 \
57+
--test-id test_456 \
58+
--reason 'Investigating checkout failures' \
59+
--ends-at 2026-09-23T12:00:00Z
60+
```
61+
62+
Creation requires `--suite-id` and at least one of `--test-id`, `--project`, or
63+
`--branch`. List active rules, include inactive rules, or deactivate a rule by
64+
its exact ID:
65+
66+
```bash
67+
npx endform@latest quarantine list
68+
npx endform@latest quarantine list --all
69+
npx endform@latest quarantine deactivate rule_789
70+
```
71+
72+
An AI coding agent can follow the same sample-driven workflow through the
73+
[Modify quarantine rules with AI](/docs/guides/modify-quarantine-rules-with-ai)
74+
guide.
75+
4876
## Example: quarantine a failing Firefox test
4977

5078
This test, `launches firefox in lambda on arm64`, fails because its assertion

‎src/content/docs/reference/mcp-server.mdx‎

Lines changed: 51 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ sidebar:
88
import { Aside, Tabs, TabItem } from "@astrojs/starlight/components";
99
import ParameterTable from "../../../components/ParameterTable.astro";
1010

11-
The Endform CLI includes a local [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) server. It gives AI coding agents tools for analyzing historical test data and debugging a Playwright test on Endform's remote infrastructure.
11+
The Endform CLI includes a local [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) server. It gives AI coding agents tools for analyzing historical test data, managing quarantine rules, and debugging a Playwright test on Endform's remote infrastructure.
1212

1313
## Set up the MCP server
1414

@@ -154,6 +154,56 @@ Returns raw OpenTelemetry trace data for a test attempt. The trace can show Play
154154

155155
</ParameterTable>
156156

157+
## Manage quarantine rules
158+
159+
These tools let an AI agent list, create, and deactivate quarantine rules. See [Modify quarantine rules with AI](/docs/guides/modify-quarantine-rules-with-ai) for example conversations.
160+
161+
### `list_quarantine_rules`
162+
163+
Lists active quarantine rules and their exact rule IDs, scope, reason, and expiry.
164+
165+
<ParameterTable>
166+
167+
| Parameter | Type | Description |
168+
| --- | --- | --- |
169+
| `organizationId` | string | Optional organization. Omit it to use the organization from the active login. |
170+
| `includeInactive` | boolean | Include expired and deactivated rules. Defaults to `false`. |
171+
172+
</ParameterTable>
173+
174+
### `create_quarantine_rule`
175+
176+
Creates a quarantine rule for an exact suite and scope. Use `query_samples` to retrieve the persisted `suite.id` and, when selecting a test, `test.id`. Test names, file paths, and Playwright worker IDs are not test IDs.
177+
178+
<ParameterTable>
179+
180+
| Parameter | Type | Description |
181+
| --- | --- | --- |
182+
| `suiteId` | string | Exact persisted `suite.id` returned by `query_samples`. |
183+
| `testId` | string | Optional exact persisted `test.id`. Omit it to match all tests in the suite. |
184+
| `projectName` | string | Optional exact Playwright project name. |
185+
| `branch` | string | Optional exact branch name. |
186+
| `reason` | string | Optional reason for quarantining the matching tests. |
187+
| `endsAt` | RFC 3339 string | Optional future expiry timestamp. Omit it for no expiry. |
188+
| `organizationId` | string | Optional organization. Omit it to use the organization from the active login. |
189+
190+
</ParameterTable>
191+
192+
In addition to `suiteId`, provide at least one of `testId`, `projectName`, or `branch`. Omitted scope fields match any value within the suite.
193+
194+
### `deactivate_quarantine_rule`
195+
196+
Deactivates one quarantine rule by its exact ID. Use `list_quarantine_rules` to retrieve the ID.
197+
198+
<ParameterTable>
199+
200+
| Parameter | Type | Description |
201+
| --- | --- | --- |
202+
| `ruleId` | string | Exact quarantine rule ID returned by `list_quarantine_rules`. |
203+
| `organizationId` | string | Optional organization. Omit it to use the organization from the active login. |
204+
205+
</ParameterTable>
206+
157207
## Debug tests
158208

159209
These tools let an AI agent pause inside one Playwright test and inspect or interact with its live browser state. See [Debug tests with AI](/docs/guides/debug-tests-with-ai) for example conversations.

0 commit comments

Comments
 (0)