Per-step resource limits are silently ignored in rootless mode, even with cgroup v2 delegated #7287

Contributing guidelines and issue reporting guide

• I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.

Well-formed report checklist

• I have found a bug that the documentation does not mention anything about my problem
• I have found a bug that there are no open or closed issues that are related to my problem
• I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

Bug description

Per-step resource limits (docker buildx build --resource ..., Bake resources, LLB MemoryLimit and friends, added in #6569) have no effect in rootless BuildKit, and nothing tells the user.

ToRootless() in util/rootless/specconv/specconv_linux.go always sets spec.Linux.Resources = nil and spec.Linux.CgroupsPath = "". Both the runc and containerd executors call it in rootless mode. The comment there refers to a cgroup v1 error (mkdir /sys/fs/cgroup/cpuset/buildkit: permission denied). On cgroup v2, a rootless process can manage a cgroup that systemd delegated to it, and rootless runc applies limits there fine. Rootless Docker does exactly this for docker run --memory.

The daemon still advertises exec.meta.linux.resources, so clients send the limits and the build succeeds without any error or warning. testLinuxResources skips rootless sandboxes, so this is not covered by tests.

Clearing CgroupsPath also turns off resource usage sampling and OOM detection for rootless steps.

Observed: a step with --resource memory=256m --resource memory-swap=256m uses 600MB and succeeds.
Expected: the step is OOM killed like docker run --memory 256m on the same host. If limits can't be applied, the user should at least get a warning.

Reproduction

On a host with cgroup v2 and rootless Docker (the docker driver, so the daemon's embedded BuildKit):

FROM alpine
# tail keeps its whole input in memory when there is no newline, so this
# step needs about 600MB.
RUN head -c 600m /dev/zero | tail > /dev/null && echo "step used 600MB and was not killed"

$ docker buildx build --builder rootless --no-cache --progress=plain \
    --resource memory=256m --resource memory-swap=256m .
#5 [2/2] RUN head -c 600m /dev/zero | tail > /dev/null && echo "step used 600MB and was not killed"
#5 3.043 step used 600MB and was not killed
#5 DONE 3.2s

$ docker run --rm --memory 256m --memory-swap 256m alpine sh -c 'head -c 600m /dev/zero | tail > /dev/null'
$ echo $?
137

The same happens with standalone rootless buildkitd (OCI worker), in a systemd user service with Delegate=yes, and in a Kubernetes pod running moby/buildkit:rootless with --oci-worker-no-process-sandbox (v0.33.1). Inside the step, /sys/fs/cgroup does not exist.

Possible fix

Two PRs, stacked:

• executor: warn when rootless drops resource limits #7288 logs a warning and adds a build warning when a rootless executor ignores limits. No behaviour change otherwise.
• rootless: apply resource limits when cgroup v2 is delegated #7289 applies the limits in the rootless OCI worker when the cgroup buildkitd runs in is cgroup v2, writable and has the cpu and memory controllers. Otherwise it falls back to today's behaviour plus the warning.

Neither PR covers the rest:

• The containerd executor still drops the limits in rootless mode. Docker Engine's embedded BuildKit uses this executor when the containerd image store is on (worker label executor: containerd in the output below).
• Docker Engine would also need to pass a delegated cgroup to runcexecutor.New in daemon/internal/builder-next for its runc path.
• In Kubernetes, unprivileged pods get a read-only /sys/fs/cgroup, so detection fails there and limits stay off (KEP-5714).

Version information

$ docker buildx version
github.com/docker/buildx v0.38.0 0dbe87f36ed472fe36b6796c3f70b4210a731fc5

$ docker buildx inspect rootless
Name:          rootless
Driver:        docker
Nodes:
Name:             rootless
Endpoint:         rootless
Status:           running
BuildKit version: v0.34.0
Platforms:        linux/amd64, linux/amd64/v2, linux/amd64/v3
Labels:
 org.mobyproject.buildkit.worker.executor:             containerd
 org.mobyproject.buildkit.worker.network:              host
 org.mobyproject.buildkit.worker.snapshotter:          overlayfs

$ docker version --format '{{.Server.Version}}'
29.9.1

$ docker info --format 'cgroup v{{.CgroupVersion}} driver={{.CgroupDriver}} kernel={{.KernelVersion}} os={{.OperatingSystem}} {{.SecurityOptions}}'
cgroup v2 driver=systemd kernel=7.0.0-38-generic os=Ubuntu 24.04.5 LTS [name=seccomp,profile=builtin name=rootless name=cgroupns]

$ cat /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/cgroup.subtree_control
cpu memory pids