Merge pull request #22 from ci-sourcerer/feat/docker-in-docker

feat(docker): add Docker-in-Docker support for container image

Author: ci-sourcerer

docs/tasks/container-images.md

@@ -77,7 +77,7 @@ COPY . /tmp/build/
 # Export debug requirements when the project defines a debug dependency group
 RUN --mount=type=cache,target=/root/.cache/uv,id=uv-cache{{ CACHE_ID_SUFFIX }}{% for mount in UV_INDEX_SECRET_MOUNTS %} \
     --mount={{ mount }}{% endfor %} \
-    uv export --group debug --no-hashes --format requirements-txt --output-file requirements-debug.txt
+    uv export --frozen --only-group debug --no-hashes --format requirements-txt --output-file requirements-debug.txt
 {% endif %}
 
 # Build the wheel, caching the uv cache directory to speed up subsequent builds.
@@ -172,10 +172,11 @@ LABEL git.commit=${GIT_COMMIT}
 # current package does not provide a console script, the entrypoint will default to `python`
 RUN echo "#!/bin/sh
 
-{{ ENTRYPOINT_COMMAND|default('python') }} \"\$@\"" >/pkg/entrypoint.sh \
+exec {{ ENTRYPOINT_COMMAND|default('python') }} \"\$@\"" >/pkg/entrypoint.sh \
     && chmod +x /pkg/entrypoint.sh
 
-USER py
+ARG RUNTIME_USER=py
+USER ${RUNTIME_USER}
 
 {% if EXTENSION_CONTENT %}
 {{ EXTENSION_CONTENT }}
@@ -193,13 +194,13 @@ COPY --from=builder /tmp/build /tmp/build
 
 RUN --mount=type=cache,target=/root/.cache/pip,id=pip-cache{{ CACHE_ID_SUFFIX }} pip install --root-user-action=ignore -r /tmp/build/requirements-debug.txt
 
-USER py
+USER ${RUNTIME_USER}
 {% endif %}
 
 # Final (default) image: explicitly use runtime as the final target so debug is not used unless requested
 FROM runtime AS final
 
-USER py
+USER ${RUNTIME_USER}
 ```
 
 ### Dependency image template
@@ -310,12 +311,14 @@ RUN apt-get update \
 USER py
 ```
 
-Extension files are concatenated in their configured order and inserted near the end of `runtime`, after the entrypoint is created and after `USER py`. An extension that needs elevated permissions must switch to `USER root`; it should normally restore `USER py` for the instructions that follow. `COPY` paths remain relative to the project-root build context.
+Extension files are concatenated in their configured order and inserted near the end of `runtime`, after the entrypoint is created and after `USER ${RUNTIME_USER}` (which defaults to `py`). An extension that needs elevated permissions must switch to `USER root`; it should normally restore `USER ${RUNTIME_USER}` for the instructions that follow. Extensions that need to start as another user can redeclare `ARG RUNTIME_USER=root`; both the final and debug stages honor this argument. `COPY` paths remain relative to the project-root build context.
 
 Extension content is treated as raw Dockerfile syntax, not as a Jinja template. This keeps project extensions independent of private template variables used by `common-python-tasks`.
 
 `CONTAINER_EXTENSIONS` selects extension bundles shipped in the installed package's `data/dockerfile_extensions/` directory. Bundle names are colon-delimited and are applied after local extension files. A bundle may accept one value with `bundle=value`; that value is passed to the first `ARG` declared by the bundle that has not already been assigned to another extension. Arguments are ignored with a warning when the bundle declares no `ARG`.
 
+A bundled extension can keep scripts and other supporting files beside its `Dockerfile`. The image builder exposes that directory as a BuildKit named context called `cpt-extension-<bundle-name>`, with unsupported characters normalized to hyphens. The bundle can copy an asset with `COPY --from=cpt-extension-example script.sh /usr/local/bin/script` without embedding it in a heredoc. These managed contexts are added only to the application image build.
+
 Use an extension for additive runtime instructions. Use `CONTAINER_DOCKERFILE_HOOK_PATH` only when a change must rewrite another part of the selected Dockerfile. The hook must be an executable host-side script; it receives a temporary copy of the selected Dockerfile as its first argument and must edit that file in place. The hook also receives the following context variables.
 
 | Variable | Meaning |
@@ -327,6 +330,37 @@ Use an extension for additive runtime instructions. Use `CONTAINER_DOCKERFILE_HO
 | `COMMON_PYTHON_TASKS_DOCKER_PLAIN` | `1` when plain progress output is active, otherwise `0` |
 | `COMMON_PYTHON_TASKS_DOCKER_SINGLE_ARCH` | `1` for a single-architecture build, otherwise `0` |
 
+### Docker-in-Docker
+
+The bundled `docker-in-docker` extension installs Docker CE, containerd, Buildx, and Compose from Docker's signed APT repository. Its installation and supervisor scripts are separate packaged files copied through the extension's named build context; image builds do not download scripts from the devcontainer feature repository.
+
+Initial support is limited to Debian Bookworm variants (`slim-bookworm` and `bookworm`) on `amd64` and `arm64`. The extension checks the actual distribution and architecture during the build and rejects everything else, including Alpine and Trixie. Select extensions independently for each image build; other images do not need to enable Docker-in-Docker.
+
+```sh
+CONTAINER_PYTHON_VARIANT=slim-bookworm \
+    CONTAINER_EXTENSIONS=docker-in-docker poe build-image
+poe run-container --privileged
+```
+
+The image starts as root through Tini and the DinD supervisor. The supervisor prepares nested cgroups, starts a dedicated Docker daemon, and waits for `docker info` to succeed before launching the existing `/pkg/entrypoint.sh` as `py`. Application arguments and exit status are preserved. Signals reach the application, and shutdown stops the application before stopping Docker. Startup failure or loss of the daemon terminates the container with a nonzero status. `DIND_STARTUP_TIMEOUT` controls the readiness deadline in seconds (default `60`, accepted range `1`–`9999`). Each process group gets up to five seconds to stop before being killed; allow more than ten seconds for the outer container's stop timeout.
+
+Docker listens only on `unix:///var/run/docker.sock`. The image sets `DOCKER_HOST` accordingly, and startup clears Docker context and TLS environment overrides so the application uses its own daemon. Do not mount the host Docker socket. The `py` user belongs to the Docker group and can control the nested daemon; this is a privileged container, not a sandbox for untrusted workloads.
+
+The image declares volumes for `/var/lib/docker` and `/var/lib/containerd`. Docker creates anonymous volumes automatically. Use dedicated named volumes when state should survive container replacement, and never share them between concurrently running daemons. For example, when launching the built image directly, replace `your-image:tag` with its tag.
+
+```sh
+docker run --rm --privileged --stop-timeout 20 \
+    --mount source=my-app-docker,target=/var/lib/docker \
+    --mount source=my-app-containerd,target=/var/lib/containerd \
+    your-image:tag
+```
+
+For Compose, set `privileged: true`, `stop_grace_period: 20s`, and the equivalent volume mounts on the application service. Keep the image's entrypoint and startup user. `container-shell` and an explicit `--entrypoint` override bypass DinD startup; to inspect a running DinD container, use `docker exec -it --user py <container> sh`.
+
+By default, the build installs the current stable packages available in Docker's repository. An optional bundle value pins the exact Docker Engine and CLI APT version. Include literal quotes around the version inside `CONTAINER_EXTENSIONS` so its epoch colon is not parsed as an extension separator. For example, `CONTAINER_EXTENSIONS='docker-in-docker="5:29.8.1-1~debian.12~bookworm"'` selects that version if available. This pins only Engine and CLI; containerd and CLI plugins still use the repository's current versions.
+
+Maintainers can run the opt-in integration tests with `CPT_DIND_INTEGRATION=1 poe test tests/test_docker_in_docker.py`. These tests require a running Docker daemon, privileged Linux containers, and network access for image and package downloads. They build the actual generated application Dockerfile and remove their test images, containers, and anonymous volumes afterward.
+
 ## Supplying external dependencies
 
 Dependency images support artifacts that should be built separately from the application wheel, such as compiled tools or browser binaries. The dependency image must place its exported content under `/tmp/deps`; the generated application Dockerfile copies that directory into its `runtime` stage.

src/common_python_tasks/data/dockerfile_extensions/docker-in-docker/Dockerfile

@@ -0,0 +1,27 @@
+# Docker-in-Docker currently supports Debian Bookworm on amd64 and arm64.
+USER root
+
+# Optional exact APT version, including the epoch and distribution suffix.
+ARG DIND_DOCKER_VERSION
+
+COPY --from=cpt-extension-docker-in-docker --chmod=0755 install.sh \
+    /usr/local/share/cpt-dind-install.sh
+RUN /usr/local/share/cpt-dind-install.sh \
+    && rm /usr/local/share/cpt-dind-install.sh
+
+COPY --from=cpt-extension-docker-in-docker --chmod=0755 supervisor.sh \
+    /usr/local/bin/cpt-dind-supervisor
+
+RUN mv /pkg/entrypoint.sh /pkg/application-entrypoint.sh \
+    && printf '%s\n' '#!/bin/sh' \
+        'exec /usr/bin/tini -- /usr/local/bin/cpt-dind-supervisor "$@"' \
+        >/pkg/entrypoint.sh \
+    && chmod 0755 /pkg/entrypoint.sh
+
+ENV DOCKER_HOST=unix:///var/run/docker.sock
+VOLUME ["/var/lib/docker", "/var/lib/containerd"]
+
+# The generic final and debug stages inherit this startup-user selection.
+# The wrapper drops to py only for the application, after Docker is ready.
+ARG RUNTIME_USER=root
+USER ${RUNTIME_USER}

src/common_python_tasks/data/dockerfile_extensions/docker-in-docker/install.sh

@@ -0,0 +1,39 @@
+#!/bin/sh
+set -eu
+
+# shellcheck source=/dev/null
+. /etc/os-release
+if [ "$ID" != debian ] || [ "${VERSION_CODENAME:-}" != bookworm ]; then
+    echo >&2 'docker-in-docker requires Debian Bookworm; use slim-bookworm.'
+    exit 1
+fi
+case "$(dpkg --print-architecture)" in
+    amd64 | arm64) ;;
+    *)
+        echo >&2 'docker-in-docker supports amd64 and arm64 only.'
+        exit 1
+        ;;
+esac
+
+apt-get update
+apt-get install -y --no-install-recommends \
+    bash ca-certificates curl iptables tini util-linux
+install -m 0755 -d /etc/apt/keyrings
+curl -fsSL https://download.docker.com/linux/debian/gpg \
+    -o /etc/apt/keyrings/docker.asc
+chmod 0644 /etc/apt/keyrings/docker.asc
+cat >/etc/apt/sources.list.d/docker.sources <<SOURCES
+Types: deb
+URIs: https://download.docker.com/linux/debian
+Suites: bookworm
+Components: stable
+Architectures: $(dpkg --print-architecture)
+Signed-By: /etc/apt/keyrings/docker.asc
+SOURCES
+apt-get update
+apt-get install -y --no-install-recommends \
+    "docker-ce${DIND_DOCKER_VERSION:+=$DIND_DOCKER_VERSION}" \
+    "docker-ce-cli${DIND_DOCKER_VERSION:+=$DIND_DOCKER_VERSION}" \
+    containerd.io docker-buildx-plugin docker-compose-plugin
+usermod -aG docker py
+rm -rf /var/lib/apt/lists/*

src/common_python_tasks/data/dockerfile_extensions/docker-in-docker/supervisor.sh

@@ -0,0 +1,116 @@
+#!/bin/bash
+set -euo pipefail
+
+_prepare_host() {
+    if [[ $(id -u) != 0 ]]; then
+        echo >&2 'docker-in-docker must start as root with --privileged.'
+        return 1
+    fi
+    if [[ ! -w /sys/fs/cgroup ]]; then
+        echo >&2 'docker-in-docker needs writable cgroups; use --privileged.'
+        return 1
+    fi
+    if [[ -d /sys/kernel/security ]] &&
+        ! mountpoint -q /sys/kernel/security; then
+        mount -t securityfs none /sys/kernel/security
+    fi
+    _prepare_cgroups
+    if ! iptables -nL >/dev/null 2>&1 &&
+        iptables-legacy -nL >/dev/null 2>&1; then
+        update-alternatives --set iptables /usr/sbin/iptables-legacy
+        update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
+    fi
+    # Only remove our daemon's PID file, never unrelated processes' files.
+    rm -f /var/run/docker.pid
+}
+
+_prepare_cgroups() {
+    [[ -f /sys/fs/cgroup/cgroup.controllers ]] || return 0
+    mkdir -p /sys/fs/cgroup/cpt-init
+    local attempt pid controllers
+    for ((attempt = 0; attempt < 5; attempt++)); do
+        while read -r pid; do
+            # Processes can disappear or become immovable while we move them.
+            printf '%s\n' "$pid" | tee \
+                /sys/fs/cgroup/cpt-init/cgroup.procs >/dev/null 2>&1 || true
+        done </sys/fs/cgroup/cgroup.procs
+        controllers=$(sed 's/[^ ]\+/+&/g' \
+            /sys/fs/cgroup/cgroup.controllers)
+        if printf '%s\n' "$controllers" | tee \
+            /sys/fs/cgroup/cgroup.subtree_control >/dev/null 2>&1; then
+            return 0
+        fi
+        sleep 0.1
+    done
+    echo >&2 'docker-in-docker could not enable nested cgroups.'
+    return 1
+}
+
+_stop_group() {
+    local pid=$1
+    [[ -n "$pid" ]] || return 0
+    kill -TERM -- "-$pid" 2>/dev/null || true
+    local attempt
+    for ((attempt = 0; attempt < 50; attempt++)); do
+        kill -0 -- "-$pid" 2>/dev/null || break
+        sleep 0.1
+    done
+    kill -KILL -- "-$pid" 2>/dev/null || true
+    wait "$pid" 2>/dev/null || true
+}
+
+_shutdown() {
+    trap '' TERM INT
+    _stop_group "$app_pid"
+    _stop_group "$daemon_pid"
+}
+
+_wait_for_docker() {
+    local deadline=$((SECONDS + DIND_STARTUP_TIMEOUT))
+    while ((SECONDS < deadline)); do
+        if ! kill -0 "$daemon_pid" 2>/dev/null; then
+            echo >&2 'docker-in-docker daemon exited during startup.'
+            return 1
+        fi
+        if timeout 1 docker --host "$DOCKER_HOST" info >/dev/null 2>&1; then
+            return 0
+        fi
+        sleep 0.2
+    done
+    echo >&2 'docker-in-docker timed out waiting for the daemon.'
+    return 1
+}
+
+_main() {
+    if [[ ! ${DIND_STARTUP_TIMEOUT:-60} =~ ^[1-9][0-9]{0,3}$ ]]; then
+        echo >&2 'DIND_STARTUP_TIMEOUT must be an integer from 1 to 9999.'
+        return 1
+    fi
+    export DIND_STARTUP_TIMEOUT=${DIND_STARTUP_TIMEOUT:-60}
+    export DOCKER_HOST=unix:///var/run/docker.sock
+    export container=docker
+    unset DOCKER_CONTEXT DOCKER_TLS_VERIFY DOCKER_CERT_PATH
+    _prepare_host
+    daemon_pid=''
+    app_pid=''
+    trap _shutdown EXIT
+    trap 'exit 143' TERM
+    trap 'exit 130' INT
+    # Keep daemon logs on stderr and expose only the local Unix socket.
+    setsid dockerd --host "$DOCKER_HOST" --group docker >&2 &
+    daemon_pid=$!
+    _wait_for_docker
+    setsid setpriv --reuid py --regid py --init-groups \
+        env HOME="$(getent passwd py | cut -d: -f6)" USER=py LOGNAME=py \
+        /pkg/application-entrypoint.sh "$@" &
+    app_pid=$!
+    local finished status=0
+    wait -n -p finished "$daemon_pid" "$app_pid" || status=$?
+    if [[ "$finished" == "$daemon_pid" ]]; then
+        echo >&2 'docker-in-docker daemon exited while the app was running.'
+        return 1
+    fi
+    return "$status"
+}
+
+_main "$@"

src/common_python_tasks/data/generic/Dockerfile.j2

@@ -40,7 +40,7 @@ COPY . /tmp/build/
 # Export debug requirements when the project defines a debug dependency group
 RUN --mount=type=cache,target=/root/.cache/uv,id=uv-cache{{ CACHE_ID_SUFFIX }}{% for mount in UV_INDEX_SECRET_MOUNTS %} \
     --mount={{ mount }}{% endfor %} \
-    uv export --group debug --no-hashes --format requirements-txt --output-file requirements-debug.txt
+    uv export --frozen --only-group debug --no-hashes --format requirements-txt --output-file requirements-debug.txt
 {% endif %}
 
 # Build the wheel, caching the uv cache directory to speed up subsequent builds.
@@ -133,10 +133,11 @@ LABEL git.commit=${GIT_COMMIT}
 # This entrypoint is deliberately not configurable via environment variables in order to
 # ensure that the container always uses the entrypoint selected at build time. If the
 # current package does not provide a console script, the entrypoint will default to `python`
-RUN echo "#!/bin/sh\n\n{{ ENTRYPOINT_COMMAND|default('python') }} \"\$@\"" >/pkg/entrypoint.sh \
+RUN echo "#!/bin/sh\n\nexec {{ ENTRYPOINT_COMMAND|default('python') }} \"\$@\"" >/pkg/entrypoint.sh \
     && chmod +x /pkg/entrypoint.sh
 
-USER py
+ARG RUNTIME_USER=py
+USER ${RUNTIME_USER}
 
 {% if EXTENSION_CONTENT %}
 {{ EXTENSION_CONTENT }}
@@ -154,10 +155,10 @@ COPY --from=builder /tmp/build /tmp/build
 
 RUN --mount=type=cache,target=/root/.cache/pip,id=pip-cache{{ CACHE_ID_SUFFIX }} pip install --root-user-action=ignore -r /tmp/build/requirements-debug.txt
 
-USER py
+USER ${RUNTIME_USER}
 {% endif %}
 
 # Final (default) image: explicitly use runtime as the final target so debug is not used unless requested
 FROM runtime AS final
 
-USER py
+USER ${RUNTIME_USER}

src/common_python_tasks/env.py

@@ -227,6 +227,40 @@ def resolve_extension_content(descriptor: dict[str, str | None]) -> str:
     utils.fatal(f"Unknown extension descriptor source: {descriptor['source']}")
 
 
+def resolve_extension_build_context(
+    descriptor: dict[str, str | None],
+) -> tuple[str, Path] | None:
+    """Return the named build context for a bundled extension's asset directory.
+
+    Args:
+        descriptor: Extension descriptor dictionary containing `source` and
+            `bundle_name` values.
+
+    Returns:
+        A Docker build-context name and its directory, or `None` when the
+        extension has no packaged assets.
+    """
+    if descriptor["source"] != "bundle":
+        return None
+    bundle_name = descriptor["bundle_name"]
+    out = utils.load_data_file(
+        f"{bundle_name}/Dockerfile",
+        type_identifier="dockerfile_extensions",
+        fatal_on_missing=False,
+    )
+    if out is None:
+        utils.fatal(f"Extension bundle not found: {bundle_name}")
+    bundle_directory = out[0].parent
+    if not bundle_directory.is_dir() or not any(
+        path.name != "Dockerfile" for path in bundle_directory.iterdir()
+    ):
+        return None
+    return (
+        f"cpt-extension-{re.sub(r'[^a-z0-9]+', '-', bundle_name.lower()).strip('-')}",
+        bundle_directory,
+    )
+
+
 def get_cache_id_suffix(no_cache: bool) -> str:
     """Return a cache-break suffix for Docker cache mount IDs.
     Args:

src/common_python_tasks/tasks.py

@@ -541,6 +541,7 @@ def build_image(
         resolve_container_docker_build_args,
         resolve_container_dockerfile_hook_path,
         resolve_container_dockerfile_path,
+        resolve_extension_build_context,
         resolve_extension_content,
         uv_index_secret_build_args,
         uv_index_secret_mounts,
@@ -571,6 +572,11 @@ def build_image(
     # Resolve all extension fragments up-front so we fail fast on missing
     # bundles or files and avoid calling resolution logic multiple times.
     resolved_fragments = [resolve_extension_content(desc) for desc in extensions]
+    extension_build_contexts = [
+        context
+        for desc in extensions
+        if (context := resolve_extension_build_context(desc)) is not None
+    ]
 
     resolved_docker_build_args = resolve_container_docker_build_args(
         docker_build_args,
@@ -740,7 +746,14 @@ def build_image(
         plain=plain,
         single_arch=single_arch,
         extra_build_args=merged_build_args or None,
-        docker_build_args=resolved_docker_build_args,
+        docker_build_args=[
+            *resolved_docker_build_args,
+            *(
+                item
+                for name, path in extension_build_contexts
+                for item in ("--build-context", f"{name}={path}")
+            ),
+        ],
         dockerfile_hook_path=resolved_dockerfile_hook_path,
     )