julia: derive stdlib compat entries from the versions bundled across the julia compat range

julia/CHANGELOG.md

@@ -17,6 +17,7 @@
 ### Fixed
 
 - Fixed registry lookups on Julia 1.13, where Pkg changed `registry_info` to also take the registry instance
+- Compat entries for standard libraries are now derived from the versions bundled across the project's `julia` compat range (using HistoricalStdlibVersions.jl) instead of the latest registry release, which produced bounds like `Artifacts = "1.3.0"` from a legacy bridge package or `Statistics = "1.11.5"` from an upgradable stdlib release; stdlib entries are only ever widened and their manifest entries are left alone (#16227, #16228)
 - Fixed Julia version requirement parsing to correctly handle caret (^) and tilde (~) semantics according to Julia's official specification
 - Fixed handling julia style compat version spec lists
 - Corrected test expectations for 0.0.x version semantics to match Julia Pkg behavior (0.0.5 satisfies only itself, not 0.0.6+)

julia/README.md

@@ -24,6 +24,7 @@ There are some notable differences:
 - **Lockfile Updates**: Dependabot updates both `Project.toml` and tries to update any `Manifest.toml` files, whereas CompatHelper.jl only updates `Project.toml`.
 - **Workspace Support**: Dependabot handles Julia workspaces where multiple packages share a common manifest file in a parent directory.
 - **Conflict Notifications**: When manifest updates fail due to dependency conflicts (common in workspaces), Dependabot adds warning notices to pull requests explaining the issue.
+- **Standard Libraries**: Dependabot derives the compat entry of a package that ships with Julia from the versions bundled across the project's supported Julia releases, whereas CompatHelper.jl treats its registry releases like any other package (see [Standard Libraries](#standard-libraries)).
 
 Also, a goal of this is to integrate into github's CVE database and alerting systems for vulnerabilities in Julia packages.
 
@@ -72,6 +73,20 @@ Julia workspaces are fully supported. In workspace configurations:
 - When updating workspace packages, Dependabot will attempt to update both the individual `Project.toml` and the shared manifest
 - If manifest updates fail due to conflicting requirements between workspace siblings, a warning notice is added to the pull request explaining the conflict
 
+### Standard Libraries
+
+Some standard libraries also have releases in the General registry: legacy bridge packages for Julia versions that predate the stdlib (`Artifacts` 1.3.0 for Julia 1.0-1.5) and "upgradable" stdlibs that can be updated from the registry (`Statistics` 1.11.x). Pkg pins a stdlib to the version bundled with the running Julia, so a compat entry derived from those registry releases can make a project uninstallable on part of the Julia range its own `julia` compat entry admits.
+
+Dependabot therefore never treats the registry's latest release as the target for a package that ships as a standard library in any Julia release admitted by the project's `julia` compat entry (a project without a `julia` entry is assumed to support every release). Instead the helper computes the versions the compat entry has to admit across that range:
+
+- the bundled version where the package is a pinned stdlib (the Julia version itself for stdlibs that were unversioned before Julia 1.11, which is how Pkg resolves them),
+- the newest installable registry release where it is upgradable or not yet a stdlib,
+- and `0.0.0` while the range reaches releases before Julia 1.10, whose `Pkg.test()` sandbox pinned stdlibs to that version.
+
+Reduced to the lowest version per caret line, this becomes the compat entry: `Statistics = "1.10"` for `julia = "1.10"`, `Statistics = "< 0.0.1, 1"` for `julia = "1"`, `SHA = "0.7, 1"` for `julia = "1.10"`. A missing entry is added, and an existing entry that fails to admit one of those versions is widened, regardless of the configured update strategy; entries are never narrowed and no manifest updates are proposed for stdlibs. Which packages are stdlibs in which release, and at what version, comes from [HistoricalStdlibVersions.jl](https://github.com/JuliaPackaging/HistoricalStdlibVersions.jl), the same data Pkg uses to resolve for a `julia_version` other than the running one. For example, `Artifacts` is only a registry package when the `julia` compat is capped below 1.6, and `StyledStrings` when it is capped below 1.11.
+
+The General registry's AutoMerge guidelines currently exempt stdlibs from the compat requirement; the entries above follow the [stdlib compat PSA](https://discourse.julialang.org/t/psa-compat-requirements-in-the-general-registry-are-changing/104958).
+
 ### Terminology: Julia vs Dependabot
 
 **Important**: There is a terminology difference between Julia and Dependabot ecosystems:

julia/helpers/DependabotHelper.jl/Project.toml

@@ -5,13 +5,15 @@ authors = ["Ian Butterworth", "contributors"]
 
 [deps]
 Downloads = "f43a241f-c20a-4ad4-852c-f6b1247861c6"
+HistoricalStdlibVersions = "6df8b67a-e8a0-4029-b4b7-ac196fe72102"
 JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
 Pkg = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
 PrecompileTools = "aea7be01-6a6a-4083-8856-8a6e6704d82a"
 TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
 
 [compat]
 Downloads = "1"
+HistoricalStdlibVersions = "2"
 JSON = "1"
 Pkg = "1"
 PrecompileTools = "1.2.1"

julia/helpers/DependabotHelper.jl/src/DependabotHelper.jl

@@ -1,13 +1,15 @@
 module DependabotHelper
 
 import Downloads
+import HistoricalStdlibVersions
 import JSON
 import Pkg
 import TOML
 using PrecompileTools
 
 # Include all the logical function modules
 include("utilities.jl")
+include("stdlibs.jl")
 include("project_parsing.jl")
 include("version_constraints.jl")
 include("package_discovery.jl")

julia/helpers/DependabotHelper.jl/src/project_parsing.jl

@@ -60,6 +60,19 @@ function parse_project(project_path::String, manifest_path::Union{String,Nothing
             # updates for them.
             sources = get(project_toml, "sources", Dict{String,Any}())
 
+            # Packages that ship with any Julia release the project supports
+            # must not have their compat entries track registry releases; they
+            # get the versions the project has to accept instead
+            julia_compat = julia_compat_spec(project_info)
+            stdlib_julia_versions = julia_versions_for_compat(julia_compat)
+            function add_stdlib_info!(dep_info, dep_uuid)
+                dep_info["stdlib"] = is_stdlib_for_julia_versions(dep_uuid, stdlib_julia_versions)
+                if dep_info["stdlib"]
+                    dep_info["stdlib_versions"] = [string(v) for v in stdlib_versions_for_julia_compat(dep_uuid, julia_compat)]
+                end
+                return dep_info
+            end
+
             # Get dependencies and add compat requirements
             dependencies = []
             for (dep_name, dep_uuid) in project_info.deps
@@ -69,6 +82,7 @@ function parse_project(project_path::String, manifest_path::Union{String,Nothing
                     "name" => dep_name,
                     "uuid" => string(dep_uuid)
                 )
+                add_stdlib_info!(dep_info, dep_uuid)
 
                 # Add version constraint if available in compat
                 if haskey(project_info.compat, dep_name)
@@ -100,8 +114,11 @@ function parse_project(project_path::String, manifest_path::Union{String,Nothing
 
                     weak_dep_info = Dict{String,Any}(
                         "name" => dep_name,
-                        "uuid" => dep_uuid_str
+                        "uuid" => dep_uuid_str,
+                        "stdlib" => false
                     )
+                    weak_dep_uuid = tryparse(Base.UUID, string(dep_uuid_str))
+                    weak_dep_uuid === nothing || add_stdlib_info!(weak_dep_info, weak_dep_uuid)
 
                     # Add version constraint if available in compat
                     if haskey(project_info.compat, dep_name)

julia/helpers/DependabotHelper.jl/src/stdlibs.jl

@@ -0,0 +1,166 @@
+# Standard library handling for DependabotHelper.jl
+#
+# Pkg pins a standard library to the version bundled with the running Julia,
+# so a compat entry derived from a stdlib's registry releases (a legacy bridge
+# such as Artifacts 1.3 for Julia 1.0-1.5, or an "upgradable" stdlib release
+# such as Statistics 1.11) can make a project uninstallable on part of the
+# Julia range its own `julia` compat entry admits. Pkg can say whether a
+# package is a stdlib at a given `julia_version` (using the per-release data
+# from HistoricalStdlibVersions.jl, as it does when resolving for another
+# Julia), but a compat entry admits a range of releases and the stdlib set
+# changes between them (Artifacts became a stdlib in 1.6, StyledStrings in
+# 1.11), so the range is split into eras of constant stdlib set here and each
+# era is checked with Pkg.
+
+const JULIA_UUID = Base.UUID("1222c4b2-2114-5bfd-aeef-88e4692bbb3e")
+
+# Before this release, Pkg.test() pinned stdlibs to v0.0.0 in the test
+# sandbox, so a stdlib compat entry has to admit 0.0.0 as well. See
+# https://discourse.julialang.org/t/psa-compat-requirements-in-the-general-registry-are-changing/104958
+const STDLIB_TEST_SANDBOX_FIXED = v"1.10.0"
+
+"""
+    stdlib_eras(julia_compat) -> Vector{Tuple{VersionNumber, VersionSpec, Dict}}
+
+Split the Julia releases admitted by `julia_compat` (a `VersionSpec`; `nothing`
+admits every release) into eras with the same stdlib set. Each era is the
+release the historical data records it from, the admitted releases within it,
+and its stdlibs.
+"""
+function stdlib_eras(julia_compat::Union{Nothing, Pkg.Versions.VersionSpec})
+    spec = something(julia_compat, Pkg.Versions.VersionSpec())
+    eras = Tuple{VersionNumber, Pkg.Versions.VersionSpec, Dict{Base.UUID, Pkg.Types.StdlibInfo}}[]
+    isempty(spec) && return eras
+
+    isempty(Pkg.Types.STDLIBS_BY_VERSION) && HistoricalStdlibVersions.register!()
+    by_version = Pkg.Types.STDLIBS_BY_VERSION
+    # Each entry holds the stdlibs of that release and of every release before
+    # the next entry
+    for (i, (julia_version, stdlibs)) in enumerate(by_version)
+        lower = Pkg.Versions.VersionBound(julia_version)
+        upper = i < length(by_version) ? bound_below(first(by_version[i + 1])) : Pkg.Versions.VersionBound()
+        admitted = intersect(spec, Pkg.Versions.VersionSpec(Pkg.Versions.VersionRange(lower, upper)))
+        isempty(admitted) || push!(eras, (julia_version, admitted, stdlibs))
+    end
+
+    # Releases the historical data does not know about yet are assumed to ship
+    # the running Julia's stdlibs
+    from_current = Pkg.Versions.VersionSpec(Pkg.Versions.VersionRange(Pkg.Versions.VersionBound(VERSION), Pkg.Versions.VersionBound()))
+    admitted = intersect(spec, from_current)
+    isempty(admitted) || push!(eras, (VERSION, admitted, Pkg.Types.stdlib_infos()))
+
+    return eras
+end
+
+# Highest version bound strictly below `v`, the inclusive upper end of the
+# release range covered by one STDLIBS_BY_VERSION entry
+function bound_below(v::VersionNumber)
+    v.patch > 0 && return Pkg.Versions.VersionBound(v.major, v.minor, v.patch - 1)
+    v.minor > 0 && return Pkg.Versions.VersionBound(v.major, v.minor - 1)
+    v.major > 0 && return Pkg.Versions.VersionBound(v.major - 1)
+    return Pkg.Versions.VersionBound(0, 0, 0)
+end
+
+"""
+    julia_versions_for_compat(julia_compat) -> Vector{VersionNumber}
+
+Representative Julia releases admitted by `julia_compat`, one per era.
+"""
+julia_versions_for_compat(julia_compat) = VersionNumber[era[1] for era in stdlib_eras(julia_compat)]
+
+"""
+    is_stdlib_for_julia_versions(uuid, julia_versions) -> Bool
+
+Whether Pkg treats `uuid` as a standard library, current or former, in any of
+`julia_versions`.
+"""
+function is_stdlib_for_julia_versions(uuid::Base.UUID, julia_versions::Vector{VersionNumber})
+    # Upgradable stdlibs (DelimitedFiles, Statistics) are reported as ordinary
+    # registry packages by newer Julia releases; Pkg only learns their UUIDs
+    # while loading the running Julia's stdlib list
+    Pkg.Types.stdlib_infos()
+    return any(julia_version -> Pkg.Types.is_or_was_stdlib(uuid, julia_version), julia_versions)
+end
+
+"""
+    stdlib_versions_for_julia_compat(uuid, julia_compat) -> Vector{VersionNumber}
+
+The versions of stdlib `uuid` a project admitting `julia_compat` has to accept,
+reduced to the lowest one per caret-compatible line: the bundled version where
+the package is a pinned stdlib (the Julia version itself when the stdlib is
+unversioned, which is how Pkg resolves it), or the newest installable registry
+release where it is upgradable or not yet a stdlib. Includes v0.0.0 when the
+range reaches releases whose test sandbox pins stdlibs to 0.0.0.
+"""
+function stdlib_versions_for_julia_compat(uuid::Base.UUID, julia_compat::Union{Nothing, Pkg.Versions.VersionSpec})
+    eras = stdlib_eras(julia_compat)
+    isempty(eras) && return VersionNumber[]
+
+    versions = Set{VersionNumber}()
+    registry_versions = registry_versions_with_julia_compat(uuid)
+    for (_, admitted, stdlibs) in eras
+        info = get(stdlibs, uuid, nothing)
+        if info !== nothing
+            push!(versions, something(info.version, lowest_version(admitted)))
+        else
+            installable = [v for (v, julia_spec) in registry_versions if !isempty(intersect(julia_spec, admitted))]
+            isempty(installable) || push!(versions, maximum(installable))
+        end
+    end
+
+    spec = something(julia_compat, Pkg.Versions.VersionSpec())
+    before_fix = Pkg.Versions.VersionSpec(Pkg.Versions.VersionRange(Pkg.Versions.VersionBound(), bound_below(STDLIB_TEST_SANDBOX_FIXED)))
+    isempty(intersect(spec, before_fix)) || push!(versions, v"0.0.0")
+
+    return lowest_per_line(versions)
+end
+
+# Non-yanked registry releases of `uuid` with the Julia versions each admits
+function registry_versions_with_julia_compat(uuid::Base.UUID)
+    result = Pair{VersionNumber, Pkg.Versions.VersionSpec}[]
+    for reg in Pkg.Registry.reachable_registries()
+        entry = get(reg, uuid, nothing)
+        entry === nothing && continue
+        info = registry_info(reg, entry)
+        for (v, version_info) in info.version_info
+            version_info.yanked && continue
+            # The registry stores compat compressed by version range; Pkg 1.13
+            # dropped the per-version `compat_info` view, so intersect the
+            # ranges covering `v` here
+            julia_spec = Pkg.Versions.VersionSpec()
+            for (range, compat) in info.compat
+                v in range || continue
+                julia_spec = intersect(julia_spec, get(compat, JULIA_UUID, Pkg.Versions.VersionSpec()))
+            end
+            push!(result, v => julia_spec)
+        end
+    end
+    return result
+end
+
+function lowest_version(spec::Pkg.Versions.VersionSpec)
+    return minimum(VersionNumber(r.lower[1], r.lower[2], r.lower[3]) for r in spec.ranges if !isempty(r))
+end
+
+# Caret compatibility: versions sharing a leading non-zero component (or the
+# 0.0.x patch) are interchangeable for a compat entry, so only the lowest of
+# each line needs to be listed
+function lowest_per_line(versions)
+    lines = Dict{Tuple{Int, Int, Int}, VersionNumber}()
+    for v in versions
+        key = v.major > 0 ? (Int(v.major), 0, 0) : v.minor > 0 ? (0, Int(v.minor), 0) : (0, 0, Int(v.patch))
+        lines[key] = min(v, get(lines, key, v))
+    end
+    return sort!(collect(values(lines)))
+end
+
+"""
+    julia_compat_spec(project::Pkg.Types.Project)
+
+The project's `julia` compat entry as a `VersionSpec`, or `nothing` when absent.
+"""
+function julia_compat_spec(project::Pkg.Types.Project)
+    haskey(project.compat, "julia") || return nothing
+    compat = project.compat["julia"]
+    return compat isa Pkg.Types.Compat ? compat.val : Pkg.Types.semver_spec(string(compat))
+end