Repository navigation
test: fix what the run of #3282 showed, and use the 64-core runners - #3284
Merged
Pavel Feldman (pavelfeldman) merged 1 commit intoOct 21, 2026
Merged
Conversation
Pavel Feldman (pavelfeldman)
force-pushed
the
large-runners
branch
from
October 21, 2026 22:18
2c8af33 to
8af6f7b
Compare
…runners - The golden comparison is pixelmatch at the threshold 0.2 again, as the matcher of @playwright/test compares; microsoft#3282 switched it to ssim-cie94, which upstream uses only in comparePNGs() and utils.compare(). SS-069, SS-072, SS-082 and SS-025 on Windows Firefox failed by a few pixels. - The static HTTPS server of the Firefox policies test does not look up the name of its host: Python's HTTPServer does, and the reverse lookup hung on the macOS runner for the whole timeout of FF-003. - TR-039 counts the console messages with a listener and a future, as upstream does. expect_event() is an action of its own in the trace, which flushed the console events that the test asserts are not flushed yet. - The ports of the upstream tests that call it.slow() get the slow mark: BC-059, DC-025, HA-058, HE-002, LC-008, SS-039 and VI-031 lacked it. DC-025 timed out on every try on Windows Edge, HE-002 on Windows Firefox. - CI runs Linux and Windows on the 64-core runners of the organization with 16 workers, as upstream's tests_primary.yml does. macOS runs two workers, upstream's default of half the cores, where three ran the video and remote server tests into the timeout. The Stable job gets 120 minutes. - The Build matrix runs the three browsers on Python 3.12, the most used version, and Chromium on 3.10, 3.11, 3.13 and 3.14 and on macos-15-xlarge. The Stable matrix drops Edge on macOS, which upstream does not test. - The stacks of a test that hits its timeout show the frames of this repository only, the test and the client, and leave out the threads and tasks that every run has: the frames of pytest, asyncio and execnet around them were most of the dump and said nothing about the hang.
Pavel Feldman (pavelfeldman)
force-pushed
the
large-runners
branch
from
October 21, 2026 22:24
8af6f7b to
57b222e
Compare
Yury Semikhatsky (yury-s)
approved these changes
Oct 21, 2026
Pavel Feldman (pavelfeldman)
merged commit Oct 21, 2026
a65b9fc
into
microsoft:main
23 of 29 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
comparePNGs()andutils.compare(). SS-069, SS-072, SS-082 and SS-025 on Windows Firefox failed by a few pixels.HTTPServerdoes, and the reverse lookup hung on the macOS runner for the whole timeout of FF-003.expect_event()is an action of its own in the trace and flushed the console events the test asserts are not flushed yet.it.slow()get the slow mark (BC-059, DC-025, HA-058, HE-002, LC-008, SS-039, VI-031); DC-025 timed out on every try on Windows Edge, HE-002 on Windows Firefox.tests_primary.yml; macOS runs two workers. The Build matrix runs the three browsers on Python 3.12 and Chromium on 3.10, 3.11, 3.13 and 3.14 and on macos-15-xlarge; the Stable matrix drops Edge on macOS.