Repository navigation
fix(mediafiles): list each folder once when removing old media files - #3454
Open
free-cup-holder-exe wants to merge 2 commits into
Open
free-cup-holder-exe wants to merge 2 commits into
free-cup-holder-exe wants to merge 2 commits into
Conversation
_remove_older_files() listed a file's folder after every file it removed, to remove folders that became empty or hold only preview thumbnails. A day folder holds 17,280 stills at a 5 s snapshot interval, and the 12-hourly cleanup usually removes about half of one, so a cleanup read over 10^8 folder entries: 8,800 old stills in a 17,690-entry folder took 150 s on a Raspberry Pi 4. Since motioneye-project#3451 the cleanup runs in the single-worker task pool, so movie previews and uploads queued meanwhile wait until it is done. Collect the folders of the removed files and check each one once after the loop, subfolders before their parents, with the same handling of empty and thumbnail-only folders. The same cleanup now takes 1.4 s. Checking subfolders first also removes a folder that is left with only thumbnails once its emptied subfolder is gone; before, that depended on the order in which the files were listed.
Marijn0
approved these changes
Oct 14, 2026
Member
|
Uhh awesome. I think the try-except around individual removals mames sense, to not abort it entirely, and prevent the emtpy dir regression. We could also do a separate empty dir loop instead, but that would require an extension of the file listing method. Cleanups in the shared single-thread pool is fine an intended: It prevents concurrent runs, and also cleanup, preview generation, and uploads, can conflict with each other. All I/O operations, so the limit is usually the drive, especially on SD cards and readers with low throughput, like on the older RPi and other small SBCs. And cleanups run only once a day by default. So whether thay are delayed by some seconds from a preview generation or upload does not matter much. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
_remove_older_files()lists a file's folder after every file it removes, to remove folders that became empty or hold only preview thumbnails. That makes the cleanup quadratic in the folder size. At a 5 s snapshot interval a day folder holds 17,280 stills, and the 12-hourly cleanup usually removes about half of one. 0.44.0 had the same loop, but there the cleanup ran in its own process. Since #3451 it runs in the task pool with its single worker, so movie previews and uploads queued meanwhile wait until it is done. On one of our Pi 4 cameras running dev, the first cleanup after a restart delayed a movie preview by about 50 s.This PR collects the folders of the removed files and checks each one once after the loop, subfolders before their parents, with the same handling of empty and thumbnail-only folders. That keeps the cleanup short enough for the shared pool.
Measured on a Raspberry Pi 4 (SD card, ext4, Python 3.13), dev b5b3ee5 against this branch. Each run used a fresh copy of the same tree, laid out like our cameras' media folders: 4 day folders of 5 s snapshots, motion pictures and movies with thumbnails, 62,530 files in total, 3 days preserved, with the oldest day folder half past the cutoff (8,800 of its 17,690 pictures are old). Each version ran twice:
Both left the same 53,683 files (all newer than the cutoff) and 5 folders.
End to end, with the same tree under a real server and a movie preview queued as soon as the cleanup started removing files: on dev the preview ran 149.6 s after it was queued. With this PR it ran after 5.3 s, the same as with an idle pool (
tasks.add()delays a task by 5 s). Both servers removed the same 8,800 pictures, with no errors or warnings in either log.With old stills in one folder only: 4,000 took 11 s on dev, 8,000 took 40 s, 16,000 took 176 s.
One behaviour difference: because subfolders are checked first, a folder that is left with only thumbnails once its emptied subfolder is gone is now always removed. Before, that depended on the order in which the files were listed. It can only happen with filename patterns that put media both in a folder and in its subfolders.
Limitations:
try/finallyif you prefer.Tests:
TestRemoveOlderFilesadds 7 tests.test_lists_each_folder_oncefails on dev (200 folder listings instead of 1)..keep, left-over thumbnails, folders with other files, and the cloud cleanup call.test_checks_subfolders_before_their_parentfails without the sort (10 of 10 hash seeds). It does not reliably fail on dev, where the outcome depends on the listing order.The full suite passes on the Pi (237 tests). I also ran dev and this branch on 500 random media trees (nested folders, thumbnails, mixed ages and extensions), and both left the same end state on all of them.