Repository navigation
[BUG]: Publish Docker Image fails on main — bullseye-security packages 404 at Dockerfile:3 #119
Description
Description of the bug
The Publish Docker Image workflow fails for every build, on main as well as on pull requests. code/oak-d-lite-module/Dockerfile cannot get past line 3.
Dockerfile:3
1 | FROM python:3.10-bullseye
2 |
3 | >>> RUN apt-get update && apt-get install -y wget build-essential cmake pkg-config
libjpeg-dev libtiff5-dev libavcodec-dev ... gfortran git
E: Failed to fetch http://deb.debian.org/debian-security/pool/updates/main/t/tiff/libtiff5-dev_4.2.0-1%2bdeb11u8_amd64.deb 404 Not Found
E: Failed to fetch http://deb.debian.org/debian-security/pool/updates/main/f/ffmpeg/libavcodec58_4.3.9-0%2bdeb11u2_amd64.deb 404 Not Found
E: Failed to fetch http://deb.debian.org/debian-security/pool/updates/main/m/mesa/libgl1-mesa-dri_20.3.5-1%2bdeb11u1_amd64.deb 404 Not Found
... (~13 packages in total)
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
exit code: 100
Steps To Reproduce
Any build triggers it. Confirmed on the base branch itself:
| Run | Ref | Content | Result |
|---|---|---|---|
| 34765657297 | main @ 1a0e71d |
no PR changes (workflow_dispatch) |
❌ same 404s |
| 34765516598 | PR #118 | docs/Python-only diff | ❌ same 404s |
What the evidence shows
- Not caused by any PR. The failing step is
Dockerfile:3; the firstCOPYisDockerfile:19. No repository file has entered the image when it dies, so no diff can influence it. Confirmed by themainrun above. - Not a single bad mirror node. The two runs hit different CDN backends (
146.75.30.132and140.248.146.132) and failed identically. - Not a transient network blip. The earlier symptom of that kind was
Connection reset by peer; this is a clean404 Not Found. - Partial, not wholesale.
apt-get updatesucceeds,bullseye/mainpackages all download, and somebullseye-securitypackages download fine (git,libavutil-dev,libswresample-dev). Only a subset 404. So thebullseye-securityindex being served references.debfiles that are no longer in the pool.
Why now
The last green build was run 166 on 2026-08-07. Nothing has rebuilt the image between then and 2026-09-13, so the breakage window is those five weeks, and the first build afterwards surfaced it.
Debian 11 "bullseye" is at/near end of LTS, and a pool prune or archive transition during that window fits the symptom. I was not able to verify this against deb.debian.org / archive.debian.org directly (network egress from my environment is proxied and returns 403 for those hosts), so treat the cause as a strong hypothesis and the behaviour above as established fact.
Possible fixes
Roughly in order of how durable they are:
- Move off bullseye.
python:3.10-bookworm, or a newer Python on bookworm. Bookworm is current stable and its security pool is actively maintained. Needs a check thatdepthaiandopencv-pythonare happy there. - Point apt at
archive.debian.orgif bullseye really has been archived, plusAcquire::Check-Valid-Until "false". Keeps the current image but explicitly freezes it on an EOL distro — a security posture worth deciding deliberately. - Use
snapshot.debian.orgfor a genuinely reproducible apt timestamp. Most reproducible, slowest. - Trim the package list. Several of these are heavy GUI/GTK/mesa dependencies (
libgtk2.0-dev,libgtk-3-dev,mesa-va-drivers,libegl1-mesa-dev) that a headless CherryPy webserver plausibly does not need. Shrinking the list both speeds the build and removes most of the 404-ing packages. Worth checking whatopencv-pythonactually requires at runtime versus what was copied from an OpenCV-from-source guide.
Related: #113 (pinning requirements.txt and the base image) covers the same reproducibility theme — an unpinned floating base image is why this changed under us without a commit.
Note
Found while investigating the CI failure on #118; that PR is unaffected by the fix and should not carry it.
Activity
gmacario commentedon Sep 13, 2026
Third data point, and it changes the picture slightly — worth recording.
I re-ran the #118 job (run 34765516598, attempt 2). It failed identically at
Dockerfile:3, exit 100, from a third distinct mirror backend:146.75.30.132main(workflow_dispatch)140.248.146.132146.75.94.132Three backends, same failure — so this is conclusively not a flake or a single stale CDN node.
The failing set is growing. Packages that downloaded successfully in the first two runs are now 404ing too:
That progression — over roughly seven minutes — is consistent with an in-progress prune of the
bullseye-securitypool while the index still advertises the old filenames. It rules out "transient mirror-sync hiccup" (which would recover, not spread), and it means the problem will get worse rather than resolve itself.Practical consequence for the fix ranking above: option 4 (trimming the package list) is now much less attractive on its own, since the set of unavailable packages is still expanding and could reach whatever subset we keep. Option 1 (move off bullseye to bookworm) or option 3 (pin to a
snapshot.debian.orgtimestamp) are the ones that actually stop this recurring.Still not verified from my side why the pool is being pruned — my environment cannot reach
deb.debian.orgto check.Generated by Claude Code
Patch depthai's docker_dependencies.sh for bookworm package renames (#…