Skip to content

Commit d4b0b5d

Browse files
augustoccesaraugustoccesar
authored
docs: add quarantine feature documentation (#14)
1 parent 7bb071c commit d4b0b5d

7 files changed

Lines changed: 215 additions & 0 deletions

File tree

‎src/assets/quarantine-add-rule.png‎

167 KB
Loading

‎src/assets/quarantine-suite-results.png‎

50.7 KB
Loading

‎src/assets/quarantine-test-after.png‎

125 KB
Loading

‎src/assets/quarantine-test-before.png‎

119 KB
Loading

‎src/content/docs/explanation/custom-reporters.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,3 +120,41 @@ onTestEnd(test: TestCase, result: TestResult) {
120120
```
121121
122122
This annotation is only present when the remote runner produced a Playwright OpenTelemetry trace for that test attempt.
123+
124+
## Quarantined results
125+
126+
[Quarantine rules](/docs/guides/quarantine-tests) allow selected tests to keep running without their failures normally failing the suite. Endform's dashboard and analytics preserve the actual outcomes. The merged blob reports consumed by Playwright reporters use a passing representation for quarantined attempts whose original status is `passed`, `failed`, or `timedOut`.
127+
128+
For those attempts, the reported status and expected status are `passed`, and the reported errors are cleared. This means an HTML, JUnit, or custom reporter can show a passing result where Endform's dashboard shows a quarantined failure. Skipped, interrupted, and attempts that did not run are not converted to passing results by quarantine.
129+
130+
The original information is available in two places:
131+
132+
- An `endform-quarantined` annotation identifies the quarantine rule and original status.
133+
- An `application/json` attachment named `endform-quarantined-original-result` contains `quarantineRuleId`, `status`, `expectedStatus`, `errors`, and `annotations` from the original attempt.
134+
135+
If your reporter needs the original outcome, read the attachment rather than relying only on `result.status`. Attachments can be supplied as an inline body or a file path:
136+
137+
```typescript
138+
import { readFileSync } from "node:fs";
139+
import type { Reporter, TestCase, TestResult } from "@playwright/test/reporter";
140+
141+
export default class QuarantineReporter implements Reporter {
142+
onTestEnd(test: TestCase, result: TestResult) {
143+
const attachment = result.attachments.find(
144+
(item) => item.name === "endform-quarantined-original-result",
145+
);
146+
if (!attachment) return;
147+
148+
const body =
149+
attachment.body ??
150+
(attachment.path ? readFileSync(attachment.path) : null);
151+
if (!body) return;
152+
153+
const original = JSON.parse(body.toString("utf8"));
154+
console.log(`${test.title}: ${original.status} (quarantined)`);
155+
console.log("Quarantine rule:", original.quarantineRuleId);
156+
}
157+
}
158+
```
159+
160+
Endform's built-in terminal reporter reads these diagnostics to display original outcomes with quarantine labels and counts. Using the original failure to fail a separate reporting check can make that check fail even when Endform considers the suite passing.

‎src/content/docs/explanation/dashboard-analytics.mdx‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,16 @@ testRun.outcome = fail
102102

103103
From a failed test run, inspect the individual attempts. Attempts can include error messages, error locations, Playwright trace links, and OpenTelemetry trace links when available.
104104

105+
## Quarantined test results
106+
107+
A suite can pass even when it contains failed quarantined tests. Quarantine normally prevents those failures from failing the suite, while Endform preserves each test's actual outcome, attempts, and metrics.
108+
109+
Quarantined tests have a **Quarantined** badge in test run tables and on the test run page. Select the badge to view the rule that applied to that run. The association remains visible after the rule expires or is deactivated.
110+
111+
For example, `testRun.outcome = fail` still includes quarantined failures. When investigating a quarantined test, use its individual outcomes and retry history to judge whether it has recovered; a passing suite alone does not show that the test is healthy.
112+
113+
See [Quarantine tests](/docs/guides/quarantine-tests) to create, expire, or deactivate rules and understand which failures still block a suite.
114+
105115
## Use branch and commit metadata
106116

107117
Endform associates runs with git metadata when possible. This lets you compare behavior across branches and commits.

‎src/content/docs/guides/quarantine-tests.mdx‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
---
2+
title: Quarantine tests
3+
description: Keep tests running while temporarily preventing their failures from blocking your Endform suite
4+
sidebar:
5+
order: 7
6+
---
7+
8+
import { Image } from "astro:assets";
9+
import QuarantineTestBefore from "../../../assets/quarantine-test-before.png";
10+
import QuarantineAddRule from "../../../assets/quarantine-add-rule.png";
11+
import QuarantineTestAfter from "../../../assets/quarantine-test-after.png";
12+
import QuarantineSuiteResults from "../../../assets/quarantine-suite-results.png";
13+
14+
Quarantine lets you keep selected tests running while you investigate a known
15+
failure or flaky test. When a quarantined test fails, the suite does not fail.
16+
Endform still records their actual results, retries, and metrics.
17+
18+
Use a quarantine rule when you want to keep collecting evidence while allowing
19+
the rest of your suite to determine whether CI passes. Tests still execute and
20+
use runner time.
21+
22+
## Create a quarantine rule
23+
24+
Run your suite in Endform first so you can select it in the dashboard.
25+
26+
1. Open your organization's **Settings → Quarantine rules**.
27+
2. Select **Add quarantine rule** and choose a **Suite**.
28+
3. Fill in at least one matching field:
29+
- **Test:** Search for and select a test, or paste its test ID.
30+
- **Playwright project:** Match a project such as `chromium`.
31+
- **Branch:** Match a branch such as `main`.
32+
4. Select **Preview matches** to check the scope.
33+
5. Under **Reason & expiry**, optionally add context for your team and an expiry
34+
date. The expiry input uses your local time.
35+
6. Select **Create rule**. The rule applies to matching tests in new suite runs.
36+
37+
All filled-in fields must match exactly. Empty fields match any value within the
38+
selected suite. For example, selecting only a branch quarantines every test in
39+
that suite on that branch. Add a test or project condition when you need a
40+
narrower rule.
41+
42+
The preview uses the latest completed run for the selected suite and branch. It
43+
counts each matching test once across projects and does not consider other
44+
quarantine rules. Future runs may contain different tests, so the preview is a
45+
guide to the rule's scope.
46+
47+
## Example: quarantine a failing Firefox test
48+
49+
This test, `launches firefox in lambda on arm64`, fails because its assertion
50+
expects `firefox` but receives `chrome`. Before quarantine, the dashboard shows
51+
the failed result and the assertion error.
52+
53+
<figure>
54+
<Image
55+
src={QuarantineTestBefore}
56+
alt="Failed Firefox test with one failed attempt: expected firefox, received chrome. No quarantine badge is present."
57+
width={1200}
58+
/>
59+
<figcaption>Before quarantine: the test records a failure.</figcaption>
60+
</figure>
61+
62+
To keep this test running while investigating, select the `endform · e2e` suite
63+
and the test in **Add quarantine rule**. In this example, **Playwright project**
64+
and **Branch** are empty, so the rule matches this test across all projects and
65+
branches in the selected suite.
66+
67+
<figure>
68+
<Image
69+
src={QuarantineAddRule}
70+
alt="Add quarantine rule dialog with the endform e2e suite and Firefox test selected, with project and branch left empty."
71+
width={580}
72+
/>
73+
<figcaption>
74+
Select the test, preview the matches, and choose Create rule. Reason and
75+
expiry are optional.
76+
</figcaption>
77+
</figure>
78+
79+
The selected test is matched by its ID. Renaming or moving it, or changing its
80+
describe blocks, creates a different test ID that this rule will not match.
81+
82+
On a new run, the test still executes with its configured retries. If it fails
83+
after exhausting retries, the suite can still pass when the other tests pass
84+
and there are no blocking errors.
85+
86+
<figure>
87+
<Image
88+
src={QuarantineTestAfter}
89+
alt="The Firefox test on a later run has both Failed and Quarantined badges, with the original assertion error and trace links still visible."
90+
width={1200}
91+
/>
92+
<figcaption>
93+
After quarantine: the test still shows Failed, alongside a Quarantined
94+
badge. Its error and trace links remain available.
95+
</figcaption>
96+
</figure>
97+
98+
Select the **Quarantined** badge to see the rule that applied. The CLI also
99+
marks the test as quarantined and includes quarantine counts in the summary.
100+
101+
The suite result shows the effect of the rule. The earlier run failed; the new
102+
run passes even though its one test still fails, because that test is now
103+
quarantined.
104+
105+
<figure>
106+
<Image
107+
src={QuarantineSuiteResults}
108+
alt="Two runs of the endform e2e suite, each containing one test: the newer run at the top passed, while the earlier run below failed."
109+
width={1200}
110+
/>
111+
<figcaption>
112+
The newer suite run passes after quarantine. The earlier run keeps its
113+
failed outcome.
114+
</figcaption>
115+
</figure>
116+
117+
## What quarantine changes
118+
119+
Quarantine changes whether a test failure blocks the suite. It does not change
120+
how the test executes:
121+
122+
- Configured retries still run, including immediate and isolated retries.
123+
- Quarantined test failures do not count toward `maxFailures`.
124+
- Actual test outcomes and metrics remain in Endform's history, so you can see
125+
whether the test is recovering.
126+
127+
:::caution[Some failures still block the suite]
128+
Infrastructure errors still fail
129+
the suite. Quarantine also preserves Playwright project dependencies: if a
130+
quarantined test fails in a setup project, dependent projects cannot run, and
131+
the suite fails.
132+
:::
133+
134+
A passing suite can therefore contain failed quarantined tests. See
135+
[Dashboard analytics](/docs/explanation/dashboard-analytics#quarantined-test-results)
136+
for how to interpret those results.
137+
138+
Merged Playwright reports present quarantined attempts differently from
139+
Endform's stored outcomes. If you use HTML, JUnit, or custom reporters, read
140+
[Quarantined results in custom reporters](/docs/explanation/custom-reporters#quarantined-results).
141+
142+
## Rule timing and overlapping rules
143+
144+
Endform determines which rule applies when a suite run starts. Creating,
145+
expiring, or deactivating a rule does not change an ongoing run or rewrite
146+
historical results.
147+
148+
When several active rules match a test, the oldest matching rule applies. After
149+
it expires or is deactivated, the next matching active rule can apply to future
150+
runs.
151+
152+
A rule without an expiry stays active until you deactivate it. Expired and
153+
deactivated rules remain visible for reference.
154+
155+
## Remove or change quarantine
156+
157+
Once you have fixed the test, inspect its recent outcomes and retries in the
158+
dashboard. To restore normal failure behavior, open **Settings → Quarantine
159+
rules** and choose **Deactivate rule** from the rule's actions menu.
160+
161+
Check for other matching active rules: deactivating one rule does not remove
162+
quarantine if another still matches. Start a new suite run to verify the test is
163+
no longer quarantined.
164+
165+
To change a rule's conditions, reason, or expiry, deactivate it and create a
166+
replacement. Historical runs keep their association with the rule that
167+
originally applied.

0 commit comments

Comments
 (0)