Skip to content

Make product performance traversal tests deterministic under parallel load - #111

Merged
Xueyang-Song merged 1 commit into
mainfrom
claude/charming-feistel-262338
Oct 31, 2026
Merged

Xueyang-Song merged 1 commit into
mainfrom
claude/charming-feistel-262338

Conversation

@Xueyang-Song

Copy link
Copy Markdown
Collaborator

Problem

ProductPerformanceObservationTests.TraversalCommitEndsGenerationAndRejectsLaterStages failed once in a full JitHub.WinUI.Tests run on the MarkdownRenderer stack (3,125 of 3,126). The test and the product code it covers haven't changed since #89.

Root cause

The traversal tests set the process-wide JITHUB_PERFORMANCE_FIXTURE variable and drive the static ProductPerformanceReadiness traversal. While the variable is set, product code under test in parallel classes records stages into the same static traversal. For example, RepoCodePageViewModel.SelectFileAsync calls RecordTraversalStage("repo_code.selection.started") from thread-pool threads. Those calls reach the test's List<T> handler at the same time as the test's own calls. The class-local ReadinessGate lock only serialized the class against itself.

The flake did not show up naturally on main: 150 full-suite runs and 300 targeted runs (this class next to RepoCodePageViewModelTests) all passed. The race window is only two calls wide. To confirm the mechanism, a temporary harness ran the real test method 20,000 times while 3 threads made that same RecordTraversalStage call:

Before Count
Assert.Single() Failure: The collection did not contain any matching items (a foreign Add overwrote before.commit) ~1,239
NullReferenceException (corrupted list) 186
InvalidOperationException: Collection was modified (during Assert.Single) 144
Total 1,569 / 20,000

After the fix, the same harness gave 0 / 20,000. The harness is not committed.

Fix

  • Move the two traversal tests into a new ProductPerformanceReadinessTraversalTests class, in a DisableParallelization collection, so xUnit runs them alone after the parallel collections finish. This also stops the environment variable from leaking IsEnabled == true into other tests' product code. It follows the RequestTransportTimingCollection precedent from Make Phase0 service tests deterministic under parallel load #107.
  • Collect stages in a ConcurrentQueue and assert on one snapshot. The static event can be raised from any thread, so a late continuation from an earlier test cannot corrupt the list.
  • Count the measurement events with Interlocked.
  • Remove the redundant ReadinessGate lock. The tracker tests that don't touch static state stay in ProductPerformanceObservationTests and keep running in parallel.

No assertion changed, and no product code changed.

Validation

  • dotnet test JitHub.WinUI.Tests -c Debug -p:Platform=x64: 10 of 10 full runs passed (2,888 tests, same count as before).
  • No packages.lock.json drift.

🤖 Generated with Claude Code

The traversal tests set the process-wide JITHUB_PERFORMANCE_FIXTURE
variable and drive the static ProductPerformanceReadiness traversal.
While the variable is set, product code under test in parallel classes
(RepoCodePageViewModel.SelectFileAsync) records stages into the same
static traversal from thread-pool threads. Those calls reached the
test's List<T> handler concurrently, which lost "before.commit",
threw "Collection was modified" inside Assert.Single, or corrupted the
list. The class-local ReadinessGate lock only serialized the class
against itself.

Move the two traversal tests into a DisableParallelization collection
so they run alone, and collect stages in a ConcurrentQueue so a late
stage from another thread cannot corrupt the snapshot the test asserts
on. Assertions are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Xueyang-Song
Xueyang-Song merged commit dcfb4af into main Oct 31, 2026
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants