fix(bootstrap): unify executable and zipapp startup

Describe executables and ZIPs with one private application specification,
and share environment preparation, storage ownership and packaging across
ordinary binaries, legacy executable ZIPs and py_zipapp entry points.

Use native POSIX exec to preserve the application PID, signals and terminal
behavior, with a detached watcher for temporary-file cleanup. On Windows,
wait for the application child and its console cleanup before removal.
Publish complete persistent images without replacing concurrent entries.

Preserve public providers, custom templates and stage-two behavior. Keep
historical raw-template adapters available and support Python 3.9 ZIP
loaders. Document cleanup lifetime and legacy compatibility limitations.

Validation: 727 root targets pass; the launcher matrix passes 313 cases
with two skips. Native checks pass on macOS, Windows and Linux arm64/x64,
including final loader/ZIP checks on all four pools. Changed-file hooks,
documentation, distribution and independent implementation audit pass.

Author: jrandolf

.bazelrc.deleted_packages

@@ -28,8 +28,8 @@ common --deleted_packages=gazelle/manifest/hasher
 common --deleted_packages=gazelle/manifest/test
 common --deleted_packages=gazelle/modules_mapping
 common --deleted_packages=gazelle/python
-common --deleted_packages=gazelle/pythonconfig
 common --deleted_packages=gazelle/python/private
+common --deleted_packages=gazelle/pythonconfig
 common --deleted_packages=tests/integration/bzlmod_lockfile
 common --deleted_packages=tests/integration/compile_pip_requirements
 common --deleted_packages=tests/integration/compile_pip_requirements_test_from_external_repo

docs/environment-variables.md

@@ -3,7 +3,7 @@
 ::::{envvar} RULES_PYTHON_ADDITIONAL_INTERPRETER_ARGS
 
 This variable allows for additional arguments to be provided to the Python interpreter
-at bootstrap time when the `bash` bootstrap is used. If
+at bootstrap time. If
 `RULES_PYTHON_ADDITIONAL_INTERPRETER_ARGS` were provided as `-Xaaa`, then the command
 would be:
 
@@ -20,6 +20,12 @@ in the command executed being:
 python /path/to/debugger.py --port 12345 --file /path/to/file.py
 ```
 
+The Bash entry point parses the first line with `read -a` and places these
+arguments before the target's `interpreter_args`. Python entry points use
+`shlex.split` and place them after the target arguments. This preserves each
+entry point's existing precedence. The variable is removed before the
+application runs, so nested launchers do not apply it again.
+
 :::{seealso}
 The {bzl:obj}`interpreter_args` attribute.
 
@@ -31,13 +37,16 @@ The guide on {any}`How to integrate a debugger`
 :::{versionchanged} 1.7.0
 Support added for {bzl:flag}`--bootstrap_impl=system_python`.
 :::
+:::{versionchanged} VERSION_NEXT_PATCH
+Target and additional interpreter arguments also apply to `python app.zip`.
+:::
 
 ::::
 
 :::{envvar} RULES_PYTHON_BOOTSTRAP_VERBOSE
 
 When `1`, debug information about bootstrapping of a program is printed to
-stderr.
+stderr. Temporary runtime directories are retained to help diagnose failures.
 :::
 
 :::{envvar} RULES_PYTHON_BZLMOD_DEBUG
@@ -57,23 +66,61 @@ be removed in a subsequent major `rules_python` version. Defaults to `0` if unse
 Directory to use as the root for creating files necessary for bootstrapping so
 that a binary can run.
 
-Only applicable when {bzl:flag}`--venvs_use_declare_symlink=no` is used.
-
-When set, a binary will attempt to find a unique, reusable, location within this
-directory for the files it needs to create to aid startup. The files may not be
-deleted upon program exit; it is the responsibility of the caller to ensure
-cleanup.
-
-Manually specifying the directory is useful to lower the overhead of
-extracting/creating files on every program execution. By using a location
-outside /tmp, longer lived programs don't have to worry about files in /tmp
-being cleaned up by the OS.
-
-If not set, then a temporary directory will be created and deleted upon program
-exit.
+Applies to runtime-created virtual environments and to `py_zipapp_binary` and
+`py_zipapp_test`. Legacy executable ZIPs always use temporary extraction; their
+virtual environments cannot persist because they refer to that extraction.
+
+When set, a binary reuses files beneath this directory. The caller owns their
+lifetime and must arrange cleanup. ZIP applications prepare a unique staging
+directory and publish it only after setup succeeds. Concurrent launches reuse
+the completed result. Startup refuses an incomplete existing cache rather than
+removing files another process may be using. Use a fresh extract root if an
+existing entry is damaged.
+
+Each new cache entry is a directory symlink to a completed image in a hidden
+backing directory beside it. Publishing the symlink cannot replace another
+entry, including one created concurrently. The backing belongs to the caller
+once published. Removing just the symlink does not reclaim its image; clean
+the extract root when its applications are no longer running. Older cache
+entries stored directly as directories remain readable.
+
+ZIP cache identities include application files, permissions, bootstrap code,
+interpreter options and resolved external-runtime facts. Updating a binary can
+leave older cache entries behind. Shell and Python entry points share the same
+image identity. Published directories follow the caller's umask.
+
+When unset, bootstraps create temporary runtime directories. Bash entry points
+use `TMPDIR` or `/tmp`; Python entry points follow `tempfile`'s directory
+selection. On POSIX, an independent process removes these directories
+asynchronously after the original interpreter PID exits, including across
+exec. The application keeps its native PID, signal delivery and terminal job.
+Windows waits for the application child before removing its runtime. Console
+Ctrl-C is delivered by Windows; the bootstrap waits for the application's own
+cleanup and exit status without forwarding another interrupt.
+
+The temporary lifetime ends with the original interpreter, even if a forked
+child outlives it. Such applications need a persistent extract root. Linux
+namespace PID 1 and child subreapers can adopt the cleanup process; waiting for
+every child can then block until application exit. Persistent extraction avoids
+that process. Namespace or cgroup shutdown can kill it before removal finishes,
+and SIGKILL during setup before registration cannot guarantee cleanup. On
+systems without a native exit watch or suitable Linux procfs, PID reuse can
+delay removal.
+
+The lifetime and publication behavior above applies to the default application
+launchers. Raw templates and older custom rules exposing only `PyExecutableInfo`
+retain their existing behavior. Their Windows ZIP adapter re-extracts a persistent
+cache on every launch; directory links can make a repeated launch fail. Use
+temporary extraction for that compatibility path.
 
 :::{versionadded} 1.2.0
 :::
+
+:::{versionchanged} VERSION_NEXT_PATCH
+Ordinary and ZIP entry points preserve native POSIX execution and share
+failure-safe temporary cleanup. ZIP caches are published after preparation and
+include bootstrap inputs in their identity.
+:::
 ::::
 
 :::{envvar} RULES_PYTHON_GAZELLE_VERBOSE

news/bootstrap-runtime.fixed.md

@@ -0,0 +1,8 @@
+(bootstrap) Fixed cancellation and temporary-runtime cleanup across ordinary
+binaries, legacy ZIPs, and both entry points of `py_zipapp_binary` and
+`py_zipapp_test`. POSIX launchers preserve the application's PID, signal delivery,
+and terminal job; cleanup follows interpreter exit. Ordinary runtime-created
+virtual environments now prepare in Python. ZIP caches publish only completed
+trees, and Python ZIP entry points honor target and additional interpreter
+arguments. Paths containing spaces, partial extraction, concurrent startup, and
+custom startup templates retain their intended behavior.

python/private/BUILD.bazel

@@ -105,6 +105,49 @@ filegroup(
     visibility = NOT_ACTUALLY_PUBLIC,
 )
 
+filegroup(
+    name = "bootstrap_cleanup",
+    srcs = ["bootstrap_cleanup.py"],
+    visibility = NOT_ACTUALLY_PUBLIC,
+)
+
+filegroup(
+    name = "application_sources",
+    srcs = glob(["_rules_python_bootstrap/*.py"]),
+    visibility = NOT_ACTUALLY_PUBLIC,
+)
+
+py_library(
+    name = "application_bootstrap",
+    srcs = glob(
+        ["_rules_python_bootstrap/*.py"],
+        exclude = ["_rules_python_bootstrap/driver.py"],
+    ),
+)
+
+exports_files(
+    [
+        "_rules_python_bootstrap/driver.py",
+        "application_python_template.txt",
+        "application_zip_template.txt",
+        "application_shell_template.sh",
+    ],
+    visibility = NOT_ACTUALLY_PUBLIC,
+)
+
+bzl_library(
+    name = "application",
+    srcs = ["application.bzl"],
+    deps = [
+        ":builders",
+        ":common",
+        ":py_application_info",
+        ":py_internal",
+        "@bazel_skylib//lib:paths",
+        "@bazel_skylib//lib:shell",
+    ],
+)
+
 filegroup(
     name = "stage2_bootstrap_template",
     srcs = ["stage2_bootstrap_template.py"],
@@ -542,6 +585,7 @@ bzl_library(
     name = "py_executable",
     srcs = ["py_executable.bzl"],
     deps = [
+        ":application",
         ":attr_builders",
         ":attributes",
         ":builders",
@@ -562,6 +606,7 @@ bzl_library(
         ":venv_runfiles",
         "@bazel_skylib//lib:dicts",
         "@bazel_skylib//lib:paths",
+        "@bazel_skylib//lib:shell",
         "@bazel_skylib//lib:structs",
         "@bazel_skylib//rules:common_settings",
         "@rules_cc//cc/common",
@@ -969,6 +1014,11 @@ bzl_library(
     srcs = ["platform_info.bzl"],
 )
 
+bzl_library(
+    name = "py_application_info",
+    srcs = ["py_application_info.bzl"],
+)
+
 bzl_library(
     name = "py_cc_toolchain_info",
     srcs = ["py_cc_toolchain_info.bzl"],

python/private/_rules_python_bootstrap/__init__.py

@@ -0,0 +1 @@
+"""Private application preparation; imported only from a declared image path."""

python/private/_rules_python_bootstrap/diagnostics.py

@@ -0,0 +1,9 @@
+"""Opt-in preparation diagnostics without application arguments or environment."""
+
+import os
+import sys
+
+
+def verbose(event, *paths):
+    if os.environ.get("RULES_PYTHON_BOOTSTRAP_VERBOSE"):
+        print("rules_python bootstrap:", event, *paths, file=sys.stderr, flush=True)

python/private/_rules_python_bootstrap/driver.py

@@ -0,0 +1,27 @@
+"""Load the declared private package without using application import paths."""
+
+import sys
+
+if not getattr(sys.flags, "safe_path", False) and not sys.flags.isolated and sys.path:
+    del sys.path[0]
+
+import importlib.machinery
+import importlib.util
+import os
+
+spec = importlib.machinery.PathFinder.find_spec(
+    "_rules_python_bootstrap", [os.path.dirname(os.path.dirname(__file__))]
+)
+package = importlib.util.module_from_spec(spec)
+sys.modules[spec.name] = package
+spec.loader.exec_module(package)
+
+if __name__ == "__main__":
+    from _rules_python_bootstrap import entry
+
+    if sys.argv[1] == "directory":
+        sys.exit(entry.shell_directory_main(sys.argv[2:]))
+    elif sys.argv[1] == "prepare-archive":
+        entry.prepare_archive(sys.argv[2], sys.argv[3], cached=sys.argv[4] == "1")
+    else:
+        raise ValueError("Unknown bootstrap entry: " + sys.argv[1])