Repository navigation
chore: Replace infogetty cpp with a Rust version - #11881
Merged
Merged
Conversation
Bownairo
force-pushed
the
eero/remove-cpp
branch
from
October 16, 2026 23:26
46d986a to
56bad0b
Compare
This reverts commit cea9c9f.
Bownairo
force-pushed
the
eero/remove-cpp
branch
2 times, most recently
from
October 16, 2026 23:49
2f896f5 to
b87313c
Compare
Bownairo
changed the title
Bownairo
force-pushed
the
eero/remove-cpp
branch
2 times, most recently
from
October 16, 2026 23:57
def23cb to
7c465bf
Compare
Bownairo
force-pushed
the
eero/remove-cpp
branch
from
October 19, 2026 19:08
7c465bf to
299f89d
Compare
Bownairo
commented
Oct 19, 2026
|
✅ No security or compliance issues detected. Reviewed everything up to 3c1021d. Security OverviewDetected Code Changes
|
pierugo-dfinity
approved these changes
Oct 20, 2026
basvandijk
approved these changes
Oct 20, 2026
q-uint
pushed a commit
to Swiss-Subnet/ic
that referenced
this pull request
Oct 20, 2026
…#11899) ## Root cause `core::test::test_startup` starts Pebble (Let's Encrypt's ACME test server) and then ic-boundary with an ACME TLS-ALPN-01 certificate resolver. It then polls `https://127.0.0.1:5001/health` for up to 240 s, until ic-boundary serves the certificate it got from Pebble. Both flaky runs of the last week (merge queue on 2026-10-15, dfinity#11881 on 2026-10-19) failed their first attempt with `Unable to query ic-boundary: timed out`. The Pebble log interleaved in the test log shows ic-boundary abandoning 8 ACME orders in a row. ic-boundary only gets its certificate by luck: 1. rustls-acme 0.15.4 (used through ic-bn-lib's `AcmeAlpn`) POSTs the challenge, waits 1 s and polls the authorization. If it is still `pending`, it POSTs the challenge again ([`state.rs`](https://github.com/FlorianUekermann/rustls-acme/blob/main/src/state.rs), unchanged on upstream `main`). 2. By default, Pebble's validation authority runs 3 validations per challenge, each after a random 0-4 s sleep (`rand.Intn(5)`). Meanwhile the challenge is `processing`, so Pebble answers the second POST with `400 malformed: Cannot update challenge with status processing, only status pending`. 3. rustls-acme treats the 400 as an order error. It abandons the order and starts a new one after an exponential backoff of 1, 2, 4, 8, … s. 4. An order succeeds only in two cases: - all 3 sleeps are short enough (max 0-1 s, at most 6.4% of orders) for validation to finish before the 1 s poll; - Pebble reuses a valid authorization left behind by an abandoned order. It does this for 51% of new orders (`PEBBLE_AUTHZREUSE` defaults to 50, applied as `rand.Intn(100) > 50`). Abandoned authorizations become valid because rustls-acme leaves the challenge certificate in its single challenge slot when an order errors. When the next order replaces it before Pebble's delayed validation runs, that validation fails. This causes the `authz ... set INVALID` lines in the logs, which dfinity#9197 took for the cause. With the backoff, 240 s fits 8 orders, starting about 0, 2, 6, 12, 21, 38, 71 and 137 s in; a 9th would start at about 267 s. dfinity#9197 raised the budget from 120 s to 240 s, which only added the 8th order. Evidence: - **CI history.** Since dfinity#9197, 19 of about 1,450 CI executions of this target (remote-cache hits excluded) were FLAKY. Their BuildBuddy console logs show: - 16 runs with exactly this pattern: 8 orders, each abandoned; - 3 runs with a different cause (see "Not addressed here"); - only `test_startup` failed in all of them. - **Model.** A Monte Carlo model of the flow, fitted to the logs' timings, gives a 1.7% first-attempt failure rate per execution, and 3.5% with the old 120 s budget. - **Local runs.** Runs of `core::test::test_startup` with Pebble's settings varied through `--test_env`: | Pebble environment | passed | orders per run, mean (max) | time to certificate, median (max) | |---|---|---|---| | defaults (master) | 30/30 | 4.6 (8) | 22 s (139 s) | | `PEBBLE_AUTHZREUSE=0` | **3/10** | 6.9 (8) | the 7 failures timed out after 8 orders, as in CI | | `PEBBLE_VA_NOSLEEP=1` + `PEBBLE_AUTHZREUSE=0` | 30/30 | 1.1 (2) | 3 s (6 s) | | `PEBBLE_VA_NOSLEEP=1` | 30/30 | 1.0 (1) | 3 s (4 s) | ## Fix Set `PEBBLE_VA_NOSLEEP=1` in the test's environment. ic-bn-lib spawns Pebble as a child process, so Pebble inherits it and validates the challenge right away. rustls-acme's first poll then finds the authorization valid. Pebble still performs the real TLS-ALPN-01 validation against ic-boundary's resolver (unlike `PEBBLE_VA_ALWAYS_VALID`), so the test keeps covering it. One source of randomness remains. ic-bn-lib makes Pebble reject 1% of nonces, and rustls-acme doesn't retry a `badNonce`, so an order of 9 signed requests still fails about 9% of the time. The next order starts 1 s later. Verification on this branch: - `bazel test --runs_per_test=3 --jobs=3`: 3/3 passed, about 4 s each. - `--runs_per_test=50` of the whole target: 50/50 passed, at most 7.5 s; the certificate arrived after a median of 3 s and at most 7 s. - Every log has Pebble's `Disabling random VA sleeps` line and no `Sleeping for` lines. ## Not addressed here - **Port 34000.** 3 of the 19 flaky runs since dfinity#9197 (2026-05-08, 07-10 and 07-24) failed because Pebble could not bind its port: `listen tcp 127.0.0.1:34000: bind: address already in use`. ic-bn-lib's Pebble helper hard-codes ports 34000, 38053 and 38055, all inside Linux's ephemeral port range (32768-60999). Fixing it needs an ic-bn-lib change: ports below 32768, or retrying startup. - **rustls-acme upstream.** It shouldn't POST a challenge again while it is being validated, and it should retry `badNonce`. Let's Encrypt's Boulder rejects such a repeated challenge POST with `Authorization is already being validated` when its `SetAuthzProcessing` feature is enabled. Whether Let's Encrypt runs with that enabled in production is unknown. --- This PR was created following the steps in `.claude/skills/fix-flaky-tests/SKILL.md`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Bownairo
enabled auto-merge
October 20, 2026 16:30
dfinityianblenke
added a commit
that referenced
this pull request
Oct 29, 2026
dfinityianblenke
added a commit
that referenced
this pull request
Nov 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Translate the cpp
infogettyinto Rust, so we can remove the last remaining cpp builds from the repo.