Build recipes that exceed the 6h runner limit (e.g. envoy) via a Bazel remote cache

[tannevaled]

Problem

github.com/envoyproxy/envoy cannot be bottled on the standard GitHub-hosted
runners: a from-scratch Bazel build exceeds the 6 h job time limit. This is
not a recipe defect — every recipe-level lever has been exhausted:

Lever tried	Result
--jobs = HOST_CPUS * 0.5	x86-64 times out at 6 h
--jobs = HOST_CPUS * 1.0	ARM64 OOM-killed
--jobs = HOST_CPUS * 0.66	both fail (x64 timeout, ARM64 flaky Rust genrule)
--fission=no, -Wno-error, --disable-new-dtags, rustc wrapper	needed to compile at all, but do not reduce wall-clock below 6 h

A single job cannot finish, and the recipe cannot change the runner's time limit.

Key insight (empirically verified)

Bazel uploads each action's result to a remote cache as soon as that action
completes — not at the end of the build. Consequently a fresh, ephemeral
runner pointed at the same remote cache hits the cache for every action a
previous run already completed, and only executes what remains. Splitting one

6 h build into several sequential ≤6 h windows becomes viable: window K
uploads the actions it finishes before timing out; window K+1 starts from a
clean checkout, hits the cache for those actions, and continues.

Proof

Reference cache: bazel-remote, built
pure-Go, CGO=0. Test project: 6 × sleep 4 genrules + an aggregator.

PHASE 1  COLD  (cache empty)        -> 7 processes executed        10 s
         bazel clean --expunge      -> wipes the ENTIRE output_base (= a brand-new runner)
PHASE 2  WARM  (same remote_cache)  -> 7 remote cache hit, 0 exec   2 s
         cache store survived the --expunge: 28 files / 112 K

--expunge deletes the whole local build state, which is exactly what an
ephemeral runner starts from. The warm run reconstructed the action graph and
resumed entirely from the cache — no re-execution. This is the multi-window
mechanism, demonstrated end to end.

Proposal

1. Stand up a persistent Bazel remote cache reachable from the build workflow —
   e.g. bazel-remote (recipe proposed in github.com/buchgr/bazel-remote: new package #13957), backed by disk or S3, with
   GC/quota. One small always-on host, or an S3-proxy cache, suffices.
2. Add to the envoy build the flags:

   --remote_cache=$CACHE_URL --remote_upload_local_results=true
   

   The cache URL / credentials are a workflow-level secret, not part of the
   public recipe (a public recipe must not carry secrets). This is why the
   change belongs at the workflow layer, not in package.yml.
3. Run the build as N sequential jobs (a small needs:-ordered chain), each
   with the same --remote_cache. Each job may time out; the next resumes from
   the cache. After ≤N windows the final job produces a fully-populated tree and
   the bottle is packaged.

Alternatives considered

• --disk_cache on a persistent runner volume — the local analogue; works
  only if the same runner volume survives between windows. Ephemeral hosted
  runners don't keep the volume, so a remote cache is the portable answer.
  (For a self-hosted runner with a persistent workspace, --disk_cache alone
  is enough and needs no server.)
• A self-hosted runner without the 6 h cap — solves it directly but adds
  standing infra + trust surface; the remote-cache approach keeps the hosted
  runners.
• RBE (BuildBuddy / NativeLink) — remote execution, not just caching;
  bigger lift, out of scope for "just get envoy to bottle".

Ask

Is pkgxdev open to (a) a persistent remote cache in the build infra and (b) a
chained multi-window job for the few recipes that exceed 6 h? If so, envoy
(otherwise code-complete) can land. Happy to wire the workflow side.

[jhheider]

It sounds like building envoy isn't a good match. This is likely a case where vendored binaries (I see they ship both linuxen) is preferred. Bazel is sufficiently rare (and build-minutes heavy) that I'm not sure paying for a persistent cache server is worthwhile solution, though I appreciate the pitch.

In particular, it looks like it's getting 1/3 built in 6 hours. That's roughly an 18-hour build. I was going to confirm that you were using clang explicitly instead of gcc, since both are required apparently, but I don't see if helping sufficiently.

I think this is an instance where vendored binaries are the path of sanity.

[tannevaled]

Thanks @jhheider — the build-minutes/cost angle is the right thing to push on, so let me reframe, because it may dissolve that part.

We can provide the infrastructure. I have access to academic OpenStack research-computing capacity and can contribute persistent Linux amd64 and arm64 VMs at no cost to pkgx. The point isn't "run one giant job" — it's the cache-and-resume mechanism:

• A persistent Bazel remote cache (bazel-remote, github.com/buchgr/bazel-remote: new package #13957) hosted on that academic cloud. Bazel uploads each action as it finishes, so an ~18h build completes across several ordinary ≤6h windows — each window resumes from the cache instead of restarting. I verified this end-to-end (cold build → wipe the entire output_base → warm build = 100% cache hits), so it's not hand-waving.
• Self-hosted runners are an option too, but even there the resume comes from a persistent --disk_cache, not from a longer timeout.

So "paying for a persistent cache server" becomes "someone contributes the box," and the 6h cap stops mattering because the build is resumable rather than long.

On vendored binaries I'd gently push back: shipping upstream's prebuilt binaries gives up the one guarantee the from-source model provides — that a bottle is what its recipe says, buildable and inspectable. For a security-sensitive network proxy like envoy that's especially uncomfortable, and it runs opposite to the signing/SBOM/provenance direction elsewhere in the ecosystem. Building from source is rather the point.

On the toolchain — good question; I'll confirm envoy is compiling with clang explicitly (not gcc) and pin it, and measure rather than assert whether a clean clang toolchain shaves real time off the build.

If a contributed amd64+arm64 cache host (or runners) would make from-source envoy viable for you, I'm glad to stand it up and hand you the details. Either way, thanks for engaging with the pitch.

[jhheider]

That's a very generous infrastructure offer. I know your prior offer to Tim is still open. My reasoning is this:

• 18 hours x 2 architectures per release is a lot of build time for a single package. Even things like node strain credulity due to their release frequency.
• Envoy publishes binaries, so, I'd be more interested in devoting resources to a shim that confirms a hash against the release binary (maybe; we'd have to deal with tgz and the like) than replicating such a massive build.
• At present, I suspect you'll be the sole consumer of the output. So, I'd rather get the cost-effective method running, and if there's general need for something heavier, then we can explore that.

As always, your efforts are extremely appreciated, @tannevaled .

[tannevaled]

Fair on the cost, @jhheider — and let me gently correct one thing: I'm not really the consumer here. I build and package for others (that's rather the point of contributing recipes), and building from source is how a recipe gets validated — a package that only ever ships upstream's prebuilt binary is a recipe that's never actually exercised, so you can't really know it builds. That validation is what I'm chasing more than the envoy binary itself.

That said, your cost-effectiveness call for the pantry is the right one, and the hash-against-release-binary shim is a good middle ground — I'll prototype it. Envoy publishes per-arch standalone binaries plus SHA256s, so a recipe can fetch the release artifact, verify it against the published checksum, and package that. I can even carry SBOM/provenance through it, so a vendored bottle still records exactly which upstream artifact and hash it came from (buildType: verified-vendor rather than from-source). I'll put a draft up for you to look at.

The amd64+arm64 infra offer (and the one to Tim) stays open for anything that genuinely needs the heavier from-source path later. Thanks as always for thinking it through.