Skip to content

Make Phase0 service tests deterministic under parallel load - #107

Merged
Xueyang-Song merged 3 commits into
mainfrom
claude/serene-napier-f42b74
Oct 28, 2026
Merged

Xueyang-Song merged 3 commits into
mainfrom
claude/serene-napier-f42b74

Conversation

@Xueyang-Song

Copy link
Copy Markdown
Collaborator

Why

Two app tests kept timing out when the full JitHub.WinUI.Tests suite ran in parallel on a busy machine, which blocked unrelated PRs (for example the MarkdownRenderer stack, #106). Both passed every time when run alone. The cause was short wall-clock timeouts used as hang guards: they measured machine load, not correctness. This PR changes only test code. I found no race in the code under test.

Changes

New TestSupport/AsyncTestWait

  • HangGuard (30 s): used only to stop a hung test. It never decides whether timing-sensitive behavior passed.
  • UntilAsync(condition): polls until a condition holds and fails with a message after HangGuard. It replaces the per-class WaitForAsync (1 s budget) and WaitUntilAsync (2 s budget) helpers.

Phase0QueryServiceTests.RefreshAsync_AccountCleanupCancelsDelayedCachePutBeforePartitionClear

  • Failure: TimeoutException at line 361 (PutStarted.WaitAsync(2s)) in run 40635407090, attempt 1.
  • Cause: before the blocked cache write is reached, the refresh reads the cache. That read runs on Task.Run, and the first call creates the SQLite schema on disk. On a loaded 2-core runner, that path alone can take more than 2 s.
  • Fix: the test now reads the empty partition before starting the refresh. That asserts the precondition the final Assert.Null relies on, and it moves cold schema creation out of the window being raced. The waits use HangGuard.
  • Added guard: refresh is now awaited with HangGuard. If cleanup ever stopped cancelling the write, the test fails with "expected OperationCanceledException, got TimeoutException" instead of hanging. Checked by pointing the shared work at _sharedWorkCancellation.Token instead of the account lease token: the test fails as described.

Phase0RequestTransportTests.RequestQueue_LastPrefetchSubscriberCancellationStopsActiveWork

  • Failure: once locally under the full suite, after about 10 s.
  • Cause: the fake work registered transportCancelled.SetResult with token.Register inside a using, then awaited Task.Delay(token). Cancellation callbacks run newest-first, so the delay's callback runs before the signal's. The delayed work resumes on another thread and disposes the registration. If that happens before the token reaches the older callback, the signal is lost and the test waits out its timeout. The window is small: 0 lost in 200,000 tries in a standalone repro. The 10 s guard could also trip on thread-pool starvation alone.
  • Fix: the work now signals from its own catch (OperationCanceledException) when (token.IsCancellationRequested), which cannot be lost.
  • New deterministic check: after the caller sees cancellation, the test asserts workToken.IsCancellationRequested. This holds by construction: the caller's task only completes after the queue drops its last subscriber, and dropping the last subscriber cancels the shared token. EnqueueCoreAsync disposes its registration, which waits for a running RemoveSubscriber callback.
  • Checked by removing _sharedWorkCancellation.Cancel() from RemoveSubscriber: the test fails in 18 ms on that assertion. Before this PR, the same mutation would have meant a 10 s timeout.

Other Phase0 waits with the same shape

  • RequestQueue_ForegroundLaneIsNeverBlockedByQueuedPrefetchWork: Task.WhenAny(foreground, Task.Delay(2s)) is now foreground.WaitAsync(HangGuard). The background lane stays held until the test releases it, so a foreground read stuck behind background work still never completes and still fails the test.
  • Every other 2/5/10 s WaitAsync, Wait and WaitForIdleAsync in Phase0QueryServiceTests, Phase0RequestTransportTests and Phase0TelemetryTests is a wait for a signal the code must raise. Each now uses HangGuard.
  • Left as they are: the telemetry pacing check (gaps of at least 40 ms) and retention check (40 ms retention, read after 80 ms) assert only lower bounds, which a slow machine makes more true. The same goes for the cache-store Task.Delay(5) ordering gaps.

All [Trait("Category", "ReleaseSecurity")] attributes are unchanged.

Verification

  • Full dotnet test JitHub.WinUI.Tests/JitHub.WinUI.Tests.csproj -c Debug -p:Platform=x64, 20 runs in a row: every Phase0 test passed in all 20, with a concurrent Release build adding load for part of the loop. In 2 of the 20 runs, a different, pre-existing test failed: RepoCodePageViewModelTests.Initialize_CachedTreeRefreshReloadsReadmeWhenShaChanges, with Expected "readme-new", Actual "readme-old". This PR doesn't touch it.
  • The Release gate's test step, dotnet test ... -c Release --filter Category=ReleaseSecurity, passed (93 tests).
  • Full dotnet build JitHub.WinUI -c Release -p:Platform=x64 could not complete locally. The dev machine only has SDK 10.0.402, which brings ILCompiler 10.0.13. The lock files pin 10.0.12 (from the global.json 10.0.2xx band), so the gate's Verify-DependencySecurity.ps1 locked restore fails here, and would on main too. With locked restore relaxed, the app compiles in Release. CI's publish job runs the full gate with the pinned SDK.

Follow-ups (not in this PR)

  • RepoCodePageViewModelTests.Initialize_CachedTreeRefreshReloadsReadmeWhenShaChanges is flaky under the full suite (2 in 20 runs), as described above.
  • LatestWinsHoverStorm_DoesNotAccumulateQueuedNetworkRequests relies on a real 30 ms debounce. If the test thread stalls for longer than that while scheduling, an earlier item starts. It runs in the non-parallel collection, so the risk is low. A deterministic fix needs a TimeProvider seam in LatestWinsPrefetchScheduler, which is a production change.

🤖 Generated with Claude Code

The Phase0 query, request-queue and telemetry tests used 1-10 s wall-clock
timeouts as hang guards. On a loaded CI runner running the whole suite in
parallel, those guards tripped even though the code under test was correct.

- Add AsyncTestWait with a 30 s HangGuard and a polling UntilAsync helper;
  replace the per-class WaitForAsync/WaitUntilAsync helpers and the short
  WaitAsync/Wait timeouts in Phase0 tests with it.
- RefreshAsync_AccountCleanupCancelsDelayedCachePutBeforePartitionClear:
  read the empty partition first, which checks the precondition the final
  assertion relies on and moves cold SQLite schema creation out of the
  window being raced. Guard the refresh await so a cancellation regression
  fails instead of hanging.
- RequestQueue_LastPrefetchSubscriberCancellationStopsActiveWork: signal
  from the work's own cancellation path instead of a token.Register callback
  scoped to the work, which the resumed work could unregister before the
  token ran it. Assert the work token is cancelled once the caller observes
  cancellation, which the queue guarantees.
- RequestQueue_ForegroundLaneIsNeverBlockedByQueuedPrefetchWork: replace
  WhenAny with a 2 s Delay by awaiting the foreground with the hang guard;
  the background lane stays held, so a blocked foreground still fails.

No production code changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Xueyang-Song
Xueyang-Song merged commit c9b9d3c into main Oct 28, 2026
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants