fix(core): 🐛 retain folders after capped deletion query

[newschapmj1]

• Retain folder records when deletion queries return 10,000 entries
• Propagate incomplete deletion status to ancestor folders
• Warn that deletion will continue on a later crawl
• Add regression tests and documentation

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[newschapmj1]

Fix for remove_deleted only reconciles the first 10,000 files in a directory + Orphan files
#2530

(It is a simple/limited 'Fix'- must rerun crawl to delete file >10K total)

[dadoonet]

Ha! That's smart. I never thought about the fact that a subsequent run would actually continue cleaning files and folders after 10000 items!
So if I read the code correctly, the PR consists of warning the user that something is not yet done. As this will be cleaned later on, should this be an info vs a warn?
And may be print in the logs the link to https://fscrawler.readthedocs.io/en/latest/admin/status.html#forcing-a-new-scan, so they would know how "not to wait" too long.

Which makes me think that we can probably test this with an integration test:

1. Create 1 marker file, 10 000 + x random files between 1 and 500 in the test tmp dir
2. Launch FSCrawler with update_rate = 1s on this folder
3. wait until we have indexed 10xxx docs
4. Pause the crawler
5. Remove all the docs but the marker file
6. wait until we just have one document in the index.

I'm pasting here the idea but this could be done in another PR.

[dadoonet]

LGTM with only a small change to apply.
Also, could you add this as a fix in docs/source/release/3.1.md?

See an example at https://github.com/dadoonet/fscrawler/blob/main/docs/source/release/3.0.md?plain=1#L99

And please add your name at the end of the same file as it was done in https://github.com/dadoonet/fscrawler/blob/main/docs/source/release/3.0.md?plain=1#L142-L143

Thanks!

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[newschapmj1]

Thanks for looking at this PR so promptly.

I will submit a second PR to add a MVN test for this.