Skip to content

[BUG]: Publish Docker Image fails on main — bullseye-security packages 404 at Dockerfile:3 #119

Description

@gmacario

Description of the bug

The Publish Docker Image workflow fails for every build, on main as well as on pull requests. code/oak-d-lite-module/Dockerfile cannot get past line 3.

Dockerfile:3
   1 |     FROM python:3.10-bullseye
   2 |
   3 | >>> RUN apt-get update && apt-get install -y wget build-essential cmake pkg-config
         libjpeg-dev libtiff5-dev libavcodec-dev ... gfortran git

E: Failed to fetch http://deb.debian.org/debian-security/pool/updates/main/t/tiff/libtiff5-dev_4.2.0-1%2bdeb11u8_amd64.deb  404  Not Found
E: Failed to fetch http://deb.debian.org/debian-security/pool/updates/main/f/ffmpeg/libavcodec58_4.3.9-0%2bdeb11u2_amd64.deb  404  Not Found
E: Failed to fetch http://deb.debian.org/debian-security/pool/updates/main/m/mesa/libgl1-mesa-dri_20.3.5-1%2bdeb11u1_amd64.deb  404  Not Found
... (~13 packages in total)
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
exit code: 100

Steps To Reproduce

Any build triggers it. Confirmed on the base branch itself:

Run Ref Content Result
34765657297 main @ 1a0e71d no PR changes (workflow_dispatch) ❌ same 404s
34765516598 PR #118 docs/Python-only diff ❌ same 404s

What the evidence shows

  • Not caused by any PR. The failing step is Dockerfile:3; the first COPY is Dockerfile:19. No repository file has entered the image when it dies, so no diff can influence it. Confirmed by the main run above.
  • Not a single bad mirror node. The two runs hit different CDN backends (146.75.30.132 and 140.248.146.132) and failed identically.
  • Not a transient network blip. The earlier symptom of that kind was Connection reset by peer; this is a clean 404 Not Found.
  • Partial, not wholesale. apt-get update succeeds, bullseye/main packages all download, and some bullseye-security packages download fine (git, libavutil-dev, libswresample-dev). Only a subset 404. So the bullseye-security index being served references .deb files that are no longer in the pool.

Why now

The last green build was run 166 on 2026-08-07. Nothing has rebuilt the image between then and 2026-09-13, so the breakage window is those five weeks, and the first build afterwards surfaced it.

Debian 11 "bullseye" is at/near end of LTS, and a pool prune or archive transition during that window fits the symptom. I was not able to verify this against deb.debian.org / archive.debian.org directly (network egress from my environment is proxied and returns 403 for those hosts), so treat the cause as a strong hypothesis and the behaviour above as established fact.

Possible fixes

Roughly in order of how durable they are:

  1. Move off bullseye. python:3.10-bookworm, or a newer Python on bookworm. Bookworm is current stable and its security pool is actively maintained. Needs a check that depthai and opencv-python are happy there.
  2. Point apt at archive.debian.org if bullseye really has been archived, plus Acquire::Check-Valid-Until "false". Keeps the current image but explicitly freezes it on an EOL distro — a security posture worth deciding deliberately.
  3. Use snapshot.debian.org for a genuinely reproducible apt timestamp. Most reproducible, slowest.
  4. Trim the package list. Several of these are heavy GUI/GTK/mesa dependencies (libgtk2.0-dev, libgtk-3-dev, mesa-va-drivers, libegl1-mesa-dev) that a headless CherryPy webserver plausibly does not need. Shrinking the list both speeds the build and removes most of the 404-ing packages. Worth checking what opencv-python actually requires at runtime versus what was copied from an OpenCV-from-source guide.

Related: #113 (pinning requirements.txt and the base image) covers the same reproducibility theme — an unpinned floating base image is why this changed under us without a commit.

Note

Found while investigating the CI failure on #118; that PR is unaffected by the fix and should not carry it.

Activity

  1. gmacario commented on Sep 13, 2026

    @gmacario
    MemberAuthor

    Third data point, and it changes the picture slightly — worth recording.

    I re-ran the #118 job (run 34765516598, attempt 2). It failed identically at Dockerfile:3, exit 100, from a third distinct mirror backend:

    Observation Mirror IP Result
    #118 attempt 1 146.75.30.132 ❌ 404s
    main (workflow_dispatch) 140.248.146.132 ❌ 404s
    #118 attempt 2 (re-run) 146.75.94.132 ❌ 404s

    Three backends, same failure — so this is conclusively not a flake or a single stale CDN node.

    The failing set is growing. Packages that downloaded successfully in the first two runs are now 404ing too:

    libsystemd0_247.3-7+deb11u8          (was Get: OK, now Err)
    libsoup2.4-1_2.72.0-2+deb11u3        (was Get: OK, now Err)
    libzvbi0_0.2.35-18+deb11u1           (was Get: OK, now Err)
    libavcodec-dev_4.3.9-0+deb11u2       (was Get: OK, now Err)
    librabbitmq4_0.10.0-1+deb11u3        (was Get: OK, now Err)
    libavformat58_4.3.9-0+deb11u2        (was Get: OK, now Err)
    

    That progression — over roughly seven minutes — is consistent with an in-progress prune of the bullseye-security pool while the index still advertises the old filenames. It rules out "transient mirror-sync hiccup" (which would recover, not spread), and it means the problem will get worse rather than resolve itself.

    Practical consequence for the fix ranking above: option 4 (trimming the package list) is now much less attractive on its own, since the set of unavailable packages is still expanding and could reach whatever subset we keep. Option 1 (move off bullseye to bookworm) or option 3 (pin to a snapshot.debian.org timestamp) are the ones that actually stop this recurring.

    Still not verified from my side why the pool is being pruned — my environment cannot reach deb.debian.org to check.


    Generated by Claude Code

  2. self-assigned this
    on Sep 14, 2026
  3. added theissue type on Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions