[basvandijk]

Root cause

core::test::test_startup starts Pebble (Let's Encrypt's ACME test server) and then ic-boundary with an ACME TLS-ALPN-01 certificate resolver. It then polls https://127.0.0.1:5001/health for up to 240 s, until ic-boundary serves the certificate it got from Pebble. Both flaky runs of the last week (merge queue on 2026-10-15, #11881 on 2026-10-19) failed their first attempt with Unable to query ic-boundary: timed out. The Pebble log interleaved in the test log shows ic-boundary abandoning 8 ACME orders in a row.

ic-boundary only gets its certificate by luck:

1. rustls-acme 0.15.4 (used through ic-bn-lib's AcmeAlpn) POSTs the challenge, waits 1 s and polls the authorization. If it is still pending, it POSTs the challenge again (state.rs, unchanged on upstream main).

2. By default, Pebble's validation authority runs 3 validations per challenge, each after a random 0-4 s sleep (rand.Intn(5)). Meanwhile the challenge is processing, so Pebble answers the second POST with 400 malformed: Cannot update challenge with status processing, only status pending.

3. rustls-acme treats the 400 as an order error. It abandons the order and starts a new one after an exponential backoff of 1, 2, 4, 8, … s.

4. An order succeeds only in two cases:

   • all 3 sleeps are short enough (max 0-1 s, at most 6.4% of orders) for validation to finish before the 1 s poll;
   • Pebble reuses a valid authorization left behind by an abandoned order. It does this for 51% of new orders (PEBBLE_AUTHZREUSE defaults to 50, applied as rand.Intn(100) > 50).

   Abandoned authorizations become valid because rustls-acme leaves the challenge certificate in its single challenge slot when an order errors. When the next order replaces it before Pebble's delayed validation runs, that validation fails. This causes the authz ... set INVALID lines in the logs, which fix: deflake //rs/boundary_node/ic_boundary:ic_boundary_test #9197 took for the cause.

With the backoff, 240 s fits 8 orders, starting about 0, 2, 6, 12, 21, 38, 71 and 137 s in; a 9th would start at about 267 s. #9197 raised the budget from 120 s to 240 s, which only added the 8th order.

Evidence:

• CI history. Since fix: deflake //rs/boundary_node/ic_boundary:ic_boundary_test #9197, 19 of about 1,450 CI executions of this target (remote-cache hits excluded) were FLAKY. Their BuildBuddy console logs show:
  • 16 runs with exactly this pattern: 8 orders, each abandoned;
  • 3 runs with a different cause (see "Not addressed here");
  • only test_startup failed in all of them.
• Model. A Monte Carlo model of the flow, fitted to the logs' timings, gives a 1.7% first-attempt failure rate per execution, and 3.5% with the old 120 s budget.
• Local runs. Runs of core::test::test_startup with Pebble's settings varied through --test_env:

Pebble environment	passed	orders per run, mean (max)	time to certificate, median (max)
defaults (master)	30/30	4.6 (8)	22 s (139 s)
PEBBLE_AUTHZREUSE=0	3/10	6.9 (8)	the 7 failures timed out after 8 orders, as in CI
PEBBLE_VA_NOSLEEP=1 + PEBBLE_AUTHZREUSE=0	30/30	1.1 (2)	3 s (6 s)
PEBBLE_VA_NOSLEEP=1	30/30	1.0 (1)	3 s (4 s)

Fix

Set PEBBLE_VA_NOSLEEP=1 in the test's environment. ic-bn-lib spawns Pebble as a child process, so Pebble inherits it and validates the challenge right away. rustls-acme's first poll then finds the authorization valid. Pebble still performs the real TLS-ALPN-01 validation against ic-boundary's resolver (unlike PEBBLE_VA_ALWAYS_VALID), so the test keeps covering it.

One source of randomness remains. ic-bn-lib makes Pebble reject 1% of nonces, and rustls-acme doesn't retry a badNonce, so an order of 9 signed requests still fails about 9% of the time. The next order starts 1 s later.

Verification on this branch:

• bazel test --runs_per_test=3 --jobs=3: 3/3 passed, about 4 s each.
• --runs_per_test=50 of the whole target: 50/50 passed, at most 7.5 s; the certificate arrived after a median of 3 s and at most 7 s.
• Every log has Pebble's Disabling random VA sleeps line and no Sleeping for lines.

Not addressed here

• Port 34000. 3 of the 19 flaky runs since fix: deflake //rs/boundary_node/ic_boundary:ic_boundary_test #9197 (2026-05-08, 07-10 and 07-24) failed because Pebble could not bind its port: listen tcp 127.0.0.1:34000: bind: address already in use. ic-bn-lib's Pebble helper hard-codes ports 34000, 38053 and 38055, all inside Linux's ephemeral port range (32768-60999). Fixing it needs an ic-bn-lib change: ports below 32768, or retrying startup.
• rustls-acme upstream. It shouldn't POST a challenge again while it is being validated, and it should retry badNonce. Let's Encrypt's Boulder rejects such a repeated challenge POST with Authorization is already being validated when its SetAuthzProcessing feature is enabled. Whether Let's Encrypt runs with that enabled in production is unknown.

---

This PR was created following the steps in .claude/skills/fix-flaky-tests/SKILL.md.

🤖 Generated with Claude Code

[Copilot]

🟢 Approval recommended

The narrow test-only change addresses the identified timing issue without changing production behavior or bypassing validation.

No open issues 🎉

What changed in this PR

Reduces startup-test flakiness by disabling Pebble’s random validation delays, addressing the timing issue left unresolved by #9197.

Changes:

• Sets PEBBLE_VA_NOSLEEP=1 while preserving real ACME challenge validation.
• Documents why immediate validation is needed.

File	Description
rs/​boundary_node/​ic_boundary/​BUILD.bazel	Disables random Pebble validation delays in the test environment.

_{Review effort: Balanced}

[zeropath-ai]

✅ No security or compliance issues detected. Reviewed everything up to 666f76f.

Security Overview

• 🔎 Scanned files: 1 changed file(s)
• 🔗 Scan Link: https://zeropath.com/app/repositories/6992424c-495e-4621-b441-efd6821a4be6?scanId=72b16a1e-aa44-4371-98b4-689a13ec1ed8&codeScanTypes=PrScan&tab=issues&detectionTypes=traditional,businessLogic,customRule,IAC,SECRETS,CICD,EOL,uncategorized

Detected Code Changes

Change Type	Relevant files
Enhancement	► rs/boundary_node/ic_boundary/BUILD.bazel     Add environment variable PEBBLE_VA_NOSLEEP to Rust test environment to validate challenges immediately