Skip to content

Commit 8af6f7b

Browse files
pavelfeldmanpavelfeldman
committed
test: fix what the run of #3282 showed, and use the 64-core runners
- The golden comparison is pixelmatch at the threshold 0.2 again, as the matcher of @playwright/test compares; #3282 switched it to ssim-cie94, which upstream uses only in comparePNGs() and utils.compare(). SS-069, SS-072, SS-082 and SS-025 on Windows Firefox failed by a few pixels. - The static HTTPS server of the Firefox policies test does not look up the name of its host: Python's HTTPServer does, and the reverse lookup hung on the macOS runner for the whole timeout of FF-003. - TR-039 counts the console messages with a listener and a future, as upstream does. expect_event() is an action of its own in the trace, which flushed the console events that the test asserts are not flushed yet. - The ports of the upstream tests that call it.slow() get the slow mark: BC-059, DC-025, HA-058, HE-002, LC-008, SS-039 and VI-031 lacked it. DC-025 timed out on every try on Windows Edge, HE-002 on Windows Firefox. - CI runs Linux and Windows on the 64-core runners of the organization with 16 workers, as upstream's tests_primary.yml does. macOS runs two workers, upstream's default of half the cores, where three ran the video and remote server tests into the timeout. The Stable job gets 120 minutes. - The Build matrix runs the three browsers on Python 3.12, the most used version, and Chromium on 3.10, 3.11, 3.13 and 3.14 and on macos-15-xlarge. The Stable matrix drops Edge on macOS, which upstream does not test. - The stacks of a test that hits its timeout show the frames of this repository only, the test and the client, and leave out the threads and tasks that every run has: the frames of pytest, asyncio and execnet around them were most of the dump and said nothing about the hang.
1 parent e25e4fc commit 8af6f7b

21 files changed

Lines changed: 186 additions & 78 deletions

‎.github/workflows/ci.yml‎

Lines changed: 46 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -43,58 +43,57 @@ jobs:
4343
env:
4444
# The dot reporter of the test steps (scripts/pytest_dot_reporter.py).
4545
PYTHONPATH: scripts
46+
# 16 workers on the 64-core runners, as upstream runs its tests there; two on
47+
# the 3-core macOS runner, upstream's default of half the cores.
48+
WORKERS: ${{ startsWith(matrix.os, 'macos') && '2' || '16' }}
4649
strategy:
4750
fail-fast: false
4851
matrix:
49-
# ubuntu-22.04, not ubuntu-latest: the 24.04 image has no WebGL for headless
50-
# Firefox (CA-008, CA-009).
51-
os: [ubuntu-22.04, windows-latest, macos-latest]
52-
python-version: ['3.14']
52+
# Linux and Windows run on the 64-core runners of the organization, as
53+
# upstream's tests_primary.yml; macOS on the hosted runner.
54+
# 3.12 is our most used Python version (base)
55+
os: [playwright-x64-ubuntu24-64-core, playwright-x64-server2025-64-core, macos-latest]
56+
python-version: ['3.12']
5357
browser: [chromium, firefox, webkit]
54-
exclude:
55-
# WebKit on standard macOS-latest (currently macos-15-arm64) is unstable;
56-
# upstream pins paid macos-15-xlarge for cross-browser webkit too.
57-
- os: macos-latest
58-
browser: webkit
5958
include:
60-
- os: windows-latest
59+
- os: playwright-x64-server2025-64-core
6160
python-version: '3.10'
6261
browser: chromium
6362
- os: macos-latest
6463
python-version: '3.10'
6564
browser: chromium
66-
- os: ubuntu-22.04
65+
- os: playwright-x64-ubuntu24-64-core
6766
python-version: '3.10'
6867
browser: chromium
69-
- os: macos-15-xlarge
70-
python-version: '3.14'
71-
browser: webkit
72-
- os: windows-latest
68+
- os: playwright-x64-server2025-64-core
7369
python-version: '3.11'
7470
browser: chromium
7571
- os: macos-latest
7672
python-version: '3.11'
7773
browser: chromium
78-
- os: ubuntu-22.04
74+
- os: playwright-x64-ubuntu24-64-core
7975
python-version: '3.11'
8076
browser: chromium
81-
- os: windows-latest
82-
python-version: '3.12'
77+
- os: playwright-x64-server2025-64-core
78+
python-version: '3.13'
8379
browser: chromium
8480
- os: macos-latest
85-
python-version: '3.12'
86-
browser: chromium
87-
- os: ubuntu-22.04
88-
python-version: '3.12'
81+
python-version: '3.13'
8982
browser: chromium
90-
- os: windows-latest
83+
- os: playwright-x64-ubuntu24-64-core
9184
python-version: '3.13'
9285
browser: chromium
86+
- os: playwright-x64-server2025-64-core
87+
python-version: '3.14'
88+
browser: chromium
9389
- os: macos-latest
94-
python-version: '3.13'
90+
python-version: '3.14'
9591
browser: chromium
96-
- os: ubuntu-22.04
97-
python-version: '3.13'
92+
- os: playwright-x64-ubuntu24-64-core
93+
python-version: '3.14'
94+
browser: chromium
95+
- os: macos-15-xlarge
96+
python-version: '3.12'
9897
browser: chromium
9998
runs-on: ${{ matrix.os }}
10099
steps:
@@ -119,34 +118,34 @@ jobs:
119118
- name: Test Wheel Installation
120119
run: pytest tests/test_installation.py --browser=${{ matrix.browser }}
121120
- name: Test Sync API
122-
if: matrix.os != 'ubuntu-22.04'
123-
run: pytest tests/sync --browser=${{ matrix.browser }} -n auto -p pytest_dot_reporter -rfE --color=yes
121+
if: runner.os != 'Linux'
122+
run: pytest tests/sync --browser=${{ matrix.browser }} -n ${{ env.WORKERS }} -p pytest_dot_reporter -rfE --color=yes
124123
- name: Test Sync API
125-
if: matrix.os == 'ubuntu-22.04'
126-
run: xvfb-run pytest tests/sync --browser=${{ matrix.browser }} -n auto -p pytest_dot_reporter -rfE --color=yes
124+
if: runner.os == 'Linux'
125+
run: xvfb-run --auto-servernum --server-args="-screen 0 1280x960x24" pytest tests/sync --browser=${{ matrix.browser }} -n ${{ env.WORKERS }} -p pytest_dot_reporter -rfE --color=yes
127126
- name: Test Async API
128127
# Also after a failed sync step: the two suites report on their own.
129-
if: ${{ !cancelled() && steps.install.outcome == 'success' && matrix.os != 'ubuntu-22.04' }}
130-
run: pytest tests/async --browser=${{ matrix.browser }} -n auto -p pytest_dot_reporter -rfE --color=yes
128+
if: ${{ !cancelled() && steps.install.outcome == 'success' && runner.os != 'Linux' }}
129+
run: pytest tests/async --browser=${{ matrix.browser }} -n ${{ env.WORKERS }} -p pytest_dot_reporter -rfE --color=yes
131130
- name: Test Async API
132-
if: ${{ !cancelled() && steps.install.outcome == 'success' && matrix.os == 'ubuntu-22.04' }}
133-
run: xvfb-run pytest tests/async --browser=${{ matrix.browser }} -n auto -p pytest_dot_reporter -rfE --color=yes
131+
if: ${{ !cancelled() && steps.install.outcome == 'success' && runner.os == 'Linux' }}
132+
run: xvfb-run --auto-servernum --server-args="-screen 0 1280x960x24" pytest tests/async --browser=${{ matrix.browser }} -n ${{ env.WORKERS }} -p pytest_dot_reporter -rfE --color=yes
134133

135134
test-stable:
136135
name: Stable
137-
timeout-minutes: 90
136+
# The branded browsers on the 3-core macOS runner take longer than the bundled ones.
137+
timeout-minutes: 120
138138
env:
139139
# The dot reporter of the test steps (scripts/pytest_dot_reporter.py).
140140
PYTHONPATH: scripts
141+
WORKERS: ${{ startsWith(matrix.os, 'macos') && '2' || '16' }}
141142
strategy:
142143
fail-fast: false
143144
matrix:
144-
os: [ubuntu-22.04, windows-latest, macos-latest]
145+
os: [playwright-x64-ubuntu24-64-core, playwright-x64-server2025-64-core, macos-latest]
145146
browser-channel: [chrome]
146147
include:
147-
- os: windows-latest
148-
browser-channel: msedge
149-
- os: macos-latest
148+
- os: playwright-x64-server2025-64-core
150149
browser-channel: msedge
151150
runs-on: ${{ matrix.os }}
152151
steps:
@@ -167,18 +166,18 @@ jobs:
167166
- name: Common Tests
168167
run: pytest tests/common --browser=chromium --browser-channel=${{ matrix.browser-channel }}
169168
- name: Test Sync API
170-
if: matrix.os != 'ubuntu-22.04'
171-
run: pytest tests/sync --browser=chromium --browser-channel=${{ matrix.browser-channel }} -n auto -p pytest_dot_reporter -rfE --color=yes
169+
if: runner.os != 'Linux'
170+
run: pytest tests/sync --browser=chromium --browser-channel=${{ matrix.browser-channel }} -n ${{ env.WORKERS }} -p pytest_dot_reporter -rfE --color=yes
172171
- name: Test Sync API
173-
if: matrix.os == 'ubuntu-22.04'
174-
run: xvfb-run pytest tests/sync --browser=chromium --browser-channel=${{ matrix.browser-channel }} -n auto -p pytest_dot_reporter -rfE --color=yes
172+
if: runner.os == 'Linux'
173+
run: xvfb-run --auto-servernum --server-args="-screen 0 1280x960x24" pytest tests/sync --browser=chromium --browser-channel=${{ matrix.browser-channel }} -n ${{ env.WORKERS }} -p pytest_dot_reporter -rfE --color=yes
175174
- name: Test Async API
176175
# Also after a failed sync step: the two suites report on their own.
177-
if: ${{ !cancelled() && steps.install.outcome == 'success' && matrix.os != 'ubuntu-22.04' }}
178-
run: pytest tests/async --browser=chromium --browser-channel=${{ matrix.browser-channel }} -n auto -p pytest_dot_reporter -rfE --color=yes
176+
if: ${{ !cancelled() && steps.install.outcome == 'success' && runner.os != 'Linux' }}
177+
run: pytest tests/async --browser=chromium --browser-channel=${{ matrix.browser-channel }} -n ${{ env.WORKERS }} -p pytest_dot_reporter -rfE --color=yes
179178
- name: Test Async API
180-
if: ${{ !cancelled() && steps.install.outcome == 'success' && matrix.os == 'ubuntu-22.04' }}
181-
run: xvfb-run pytest tests/async --browser=chromium --browser-channel=${{ matrix.browser-channel }} -n auto -p pytest_dot_reporter -rfE --color=yes
179+
if: ${{ !cancelled() && steps.install.outcome == 'success' && runner.os == 'Linux' }}
180+
run: xvfb-run --auto-servernum --server-args="-screen 0 1280x960x24" pytest tests/async --browser=chromium --browser-channel=${{ matrix.browser-channel }} -n ${{ env.WORKERS }} -p pytest_dot_reporter -rfE --color=yes
182181

183182
test_examples:
184183
name: Examples

‎local-requirements.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ build==1.6.1
44
mypy==2.4.0
55
objgraph==3.6.2
66
Pillow==12.3.0
7+
pixelmatch==0.3.0
78
pre-commit==3.5.0
89
pyOpenSSL==26.0.0
910
pytest==9.0.3

‎tests/async/library/test_browsercontext_locale.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,7 @@ async def test_LC_007_should_work_for_multiple_pages_sharing_same_process(
130130
await context.close()
131131

132132

133+
@pytest.mark.slow
133134
async def test_LC_008_should_be_isolated_between_contexts(
134135
browser: Browser, server: Server
135136
) -> None:

‎tests/async/library/test_browsercontext_pages.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,7 @@ async def test_BC_058_should_return_bounding_box_with_page_scale(
166166
await context.close()
167167

168168

169+
@pytest.mark.slow
169170
async def test_BC_059_should_not_leak_listeners_during_navigation_of_20_pages(
170171
context_factory: Callable[..., Awaitable[BrowserContext]], server: Server
171172
) -> None:

‎tests/async/library/test_defaultbrowsercontext_2.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -210,6 +210,7 @@ async def test_DC_024_should_accept_relative_user_data_dir(
210210
await context.close()
211211

212212

213+
@pytest.mark.slow
213214
async def test_DC_025_should_restore_state_from_user_data_dir(
214215
browser_type: BrowserType,
215216
server: Server,

‎tests/async/library/test_har.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1326,6 +1326,7 @@ async def test_HA_057_should_close_the_context_when_saving_the_har_fails(
13261326
await browser.close()
13271327

13281328

1329+
@pytest.mark.slow
13291330
async def test_HA_058_should_support_har_larger_than_512_mb(
13301331
context_factory: Callable[..., Awaitable[BrowserContext]],
13311332
server: Server,

‎tests/async/library/test_headful.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,7 @@ async def test_HE_001_should_have_default_url_when_launching_browser_smoke(
121121
assert urls == ["about:blank"]
122122

123123

124+
@pytest.mark.slow
124125
async def test_HE_002_should_close_browser_with_beforeunload_page(
125126
launch_persistent: LaunchPersistent, server: Server
126127
) -> None:

‎tests/async/library/test_tracing.py‎

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1145,23 +1145,30 @@ async def test_TR_039_should_not_flush_console_events(
11451145
page = await context.new_page()
11461146
test_id = "TR-039"
11471147
await context.tracing.start(name=test_id)
1148+
promise: "asyncio.Future[None]" = page._loop.create_future()
11481149
counter = 0
11491150

1150-
def predicate(_: Any) -> bool:
1151+
def on_console(_: Any) -> None:
11511152
nonlocal counter
11521153
counter += 1
1153-
return counter == 100
1154+
if counter == 100:
1155+
promise.set_result(None)
11541156

1155-
async with page.expect_event("console", predicate=predicate):
1156-
await page.evaluate(
1157-
"""() => {
1157+
# Python: a listener and a future, as upstream: expect_event() would be an
1158+
# action of its own in the trace, which flushes the console events.
1159+
page.on("console", on_console)
1160+
1161+
await page.evaluate(
1162+
"""() => {
11581163
window.builtins.setTimeout(() => {
11591164
for (let i = 0; i < 100; ++i)
11601165
console.log('hello ' + i);
11611166
}, 10);
11621167
return 31415926;
11631168
}"""
1164-
)
1169+
)
1170+
1171+
await promise
11651172

11661173
def read_trace() -> str:
11671174
trace_name = next(

‎tests/async/library/test_video.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -819,6 +819,7 @@ async def test_VI_030_screencast_should_work_with_video_trace(
819819
assert is_almost_red(pixel) is True
820820

821821

822+
@pytest.mark.slow
822823
async def test_VI_031_should_save_as_video(browser: Browser, tmp_path: Path) -> None:
823824
videos_path = tmp_path
824825
size: ViewportSize = {"width": 320, "height": 240}

‎tests/async/page/test_page_screenshot.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -654,6 +654,7 @@ async def test_SS_038_page_screenshot_should_work_with_array_deleted(
654654

655655

656656
@pytest.mark.usefixtures("skip_firefox_headed")
657+
@pytest.mark.slow
657658
async def test_SS_039_page_screenshot_should_take_full_page_screenshots_during_navigation(
658659
page: Page, server: Server
659660
) -> None:

0 commit comments

Comments
 (0)