Skip to content

fix(mediafiles): list each folder once when removing old media files - #3454

Open
free-cup-holder-exe wants to merge 2 commits into
motioneye-project:devfrom
free-cup-holder-exe:fix/cleanup-folder-check
Open

free-cup-holder-exe wants to merge 2 commits into
motioneye-project:devfrom
free-cup-holder-exe:fix/cleanup-folder-check

Conversation

@free-cup-holder-exe

Copy link
Copy Markdown
Contributor

_remove_older_files() lists a file's folder after every file it removes, to remove folders that became empty or hold only preview thumbnails. That makes the cleanup quadratic in the folder size. At a 5 s snapshot interval a day folder holds 17,280 stills, and the 12-hourly cleanup usually removes about half of one. 0.44.0 had the same loop, but there the cleanup ran in its own process. Since #3451 it runs in the task pool with its single worker, so movie previews and uploads queued meanwhile wait until it is done. On one of our Pi 4 cameras running dev, the first cleanup after a restart delayed a movie preview by about 50 s.

This PR collects the folders of the removed files and checks each one once after the loop, subfolders before their parents, with the same handling of empty and thumbnail-only folders. That keeps the cleanup short enough for the shared pool.

Measured on a Raspberry Pi 4 (SD card, ext4, Python 3.13), dev b5b3ee5 against this branch. Each run used a fresh copy of the same tree, laid out like our cameras' media folders: 4 day folders of 5 s snapshots, motion pictures and movies with thumbnails, 62,530 files in total, 3 days preserved, with the oldest day folder half past the cutoff (8,800 of its 17,690 pictures are old). Each version ran twice:

picture cleanup folder listings in it movie cleanup
dev 147.5 s, 150.0 s 8,800 0.7 s
this PR 1.4 s, 1.4 s 1 0.3 s

Both left the same 53,683 files (all newer than the cutoff) and 5 folders.

End to end, with the same tree under a real server and a movie preview queued as soon as the cleanup started removing files: on dev the preview ran 149.6 s after it was queued. With this PR it ran after 5.3 s, the same as with an idle pool (tasks.add() delays a task by 5 s). Both servers removed the same 8,800 pictures, with no errors or warnings in either log.

With old stills in one folder only: 4,000 took 11 s on dev, 8,000 took 40 s, 16,000 took 176 s.

One behaviour difference: because subfolders are checked first, a folder that is left with only thumbnails once its emptied subfolder is gone is now always removed. Before, that depended on the order in which the files were listed. It can only happen with filename patterns that put media both in a folder and in its subfolders.

Limitations:

  • The cleanup still runs in the shared pool. It is linear now, but walking a large media folder still takes a moment (1.4 s for the tree above), and previews and uploads wait for it. Whether the cleanup should run outside the shared pool is your call, so I've left it out of this PR.
  • If the file loop is cut short (an exception, a kill or a power loss), folders it already emptied are not checked in that run. Later runs don't see them either, since they hold no media any more, so they stay as empty folders. On dev that window was only the moment between a folder's last removal and its check. I can wrap the loop in try/finally if you prefer.
  • The test files were empty, so removing them was cheaper than removing real stills. That cost is the same for both versions.

Tests: TestRemoveOlderFiles adds 7 tests.

  • test_lists_each_folder_once fails on dev (200 folder listings instead of 1).
  • The others pin the existing handling: old/new files, emptied folders, the target directory and .keep, left-over thumbnails, folders with other files, and the cloud cleanup call.
  • test_checks_subfolders_before_their_parent fails without the sort (10 of 10 hash seeds). It does not reliably fail on dev, where the outcome depends on the listing order.

The full suite passes on the Pi (237 tests). I also ran dev and this branch on 500 random media trees (nested folders, thumbnails, mixed ages and extensions), and both left the same end state on all of them.

_remove_older_files() listed a file's folder after every file it removed, to
remove folders that became empty or hold only preview thumbnails. A day folder
holds 17,280 stills at a 5 s snapshot interval, and the 12-hourly cleanup
usually removes about half of one, so a cleanup read over 10^8 folder entries:
8,800 old stills in a 17,690-entry folder took 150 s on a Raspberry Pi 4. Since
motioneye-project#3451 the cleanup runs in the single-worker task pool, so movie previews and
uploads queued meanwhile wait until it is done.

Collect the folders of the removed files and check each one once after the
loop, subfolders before their parents, with the same handling of empty and
thumbnail-only folders. The same cleanup now takes 1.4 s. Checking subfolders
first also removes a folder that is left with only thumbnails once its emptied
subfolder is gone; before, that depended on the order in which the files were
listed.
Marijn0

This comment was marked as outdated.

@MichaIng

MichaIng commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Uhh awesome. I think the try-except around individual removals mames sense, to not abort it entirely, and prevent the emtpy dir regression. We could also do a separate empty dir loop instead, but that would require an extension of the file listing method.

Cleanups in the shared single-thread pool is fine an intended: It prevents concurrent runs, and also cleanup, preview generation, and uploads, can conflict with each other. All I/O operations, so the limit is usually the drive, especially on SD cards and readers with low throughput, like on the older RPi and other small SBCs. And cleanups run only once a day by default. So whether thay are delayed by some seconds from a preview generation or upload does not matter much.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

3 participants