190 cleanup extra docker images created by mlflow

[mikaelsimard5]

Explanation of this bug

Closes #190.

mlops run left Docker images behind on every run, which shows up as a growing list of <none>:<none> images (bug #190).

General context on what happens: when you do mlops run there are always two images that are created/accessed; the per-project image, which is tagged as <name>:latest, and contains whatever the project's Dockerfile builds (mostly the OS, system and pip packages). Then, mlflow creates a second image (run-specific image rather than project), tagged as <name>:<first 7 chars of git commit>, which contains the code.

After investigating here's what creates a bunch of extra images tagged as <none>:<none>:

(1) the per-run mlflow image is never removed. If you do a first run (run A) then you'll have two images, <name>:latest and <name>:<first 7 chars of git commit>. If you do a second run on the same commit (run B), you'll end up with three images: the same <name>:latest as before, and then what happens is that the per-run mlflow image from run A has its tag moved to the per-run mlflow image of run B. Result is the per-run image of run B now has the <name>:<first 7 chars of git commit> tag, and the per-run image of run A has "lost" its tag and shows <none>:<none>.

(2) The old project image persists after rebuild_docker (-r). Rebuilding <name>:latest moves the tag to the new image and leaves the previous one as <none>:<none>, so similar mechanism as above but on the per-project image. This happens on Docker's classic image store, e.g. Linux servers.

Fixes

The changes below fix the issues so that we're always left with a single image (the project one, <name>:latest) as there is no need to preserve the other one.

(1) per-run image cleanup: mlflow.run is now wrapped in try/finally, and remove_run_image() removes <name>:<sha7> after every run, including failed or interrupted ones.
(2) rebuild cleanup: after a rebuild, remove_replaced_image() removes the previous <name>:latest image, but only if it has no name left.

Possible issue

The <name>:<sha7> tag format is internal to mlflow. I checked it against mlflow 2.0.0, 3.0.0, every release from 3.11.1 to 3.16.1, and current master. It's identical in all of them. If a future mlflow changes it, the cleanup logs a warning, and test_run would fail in CI so we would notice.

Testing

• New tests: test_remove_run_image and test_remove_replaced_image. test_run now also asserts that only <name>:latest remains after a run.

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
mlops
Experiment.py	186	42	77%	22, 51–56, 79, 99, 101, 106, 160–171, 202, 204–205, 255–259, 263–264, 273–274, 277–282, 317–318, 327–328, 348–349, 356–363
ProjectFile.py	23	1	96%	41
cli.py	55	17	69%	19–22, 36, 59–66, 86–88, 96–99, 107–108, 113
mlops/release
Release.py	33	7	79%	24, 28, 33–38
mlops/release/destinations
ReleaseDestination.py	9	1	89%	18
SharepointDestination.py	6	2	67%	12–13
ZenodoDestination.py	33	19	42%	21–25, 29–35, 43–60
mlops/release/sources
MLFlowSource.py	13	6	54%	13–14, 21–27
ReleaseSource.py	14	2	86%	14, 23
mlops/utils
Config.py	15	15	0%	1–22
TOTAL	424	112	74%

Tests	Skipped	Failures	Errors	Time
18	0 💤	0 ❌	0 🔥	30m 24s ⏱️

[mikewoodward94]

Excellent investigative work and fix, LGTM!

(so happy this is finally fixed)