fix(pipeline): honor discovery exclusions in pkgmap/envscan repo walks

[Sowiedu]

Fixes #792.

Problem

The auxiliary filesystem walks in pkgmap (cbm_pkgmap_scan_repo, called from merge_pkg_entries() at the parallel-extract join) and envscan only skip hardcoded directory names (cbm_should_skip_dir() / a private is_ignored_dir() list). They never consult the project's gitignore-derived exclusion list that discovery honors. On a repo with a large gitignored artifact tree this walk dominates the entire pipeline: with a 1.6M-file gitignored tile directory, index_repository takes 15–18 minutes (silent between two log lines, looks like a hang) instead of ~3 seconds.

Fix

• New helper cbm_pipeline_relpath_is_excluded() (in path_alias.c) that prefix-matches a repo-relative path against the pipeline ctx's excluded-dirs list.
• cbm_pkgmap_scan_repo() now takes the ctx's excluded_dirs/excluded_count and consults the helper before descending, in addition to the existing cbm_should_skip_dir() and Windows reparse-point checks.
• The envscan walker does the same via cbm_scan_project_env_urls_excluded(); the original cbm_scan_project_env_urls() remains as a wrapper (NULL exclusions) so existing callers are unaffected.

Hardcoded skip lists are kept as a fast path; the exclusion check only adds prefix comparisons on directory descent.

Verification

On the affected repo (~1,000 source files + 1,638,518-file gitignored map-import/output/, Windows 11/NTFS):

	before	after
full index	15–18 min	3.0 s
incremental no-op	~7 min	0.4 s

Synthetic check: repo with a gitignored bigdir/ of ~20k files — the walk no longer descends into it (verified via timing and open-handle inspection with Sysinternals handle64). Graph output is unchanged on repos without large ignored dirs (node/edge counts identical before/after on the same tree).

Diagnosis details, handle dumps, and the elimination path are in #792.

[phpmac]

v12 pls audit

[DeusData]

Thanks for the focused fix. Triage: high-priority performance/stability PR for #792.

Review will focus on whether pkgmap/envscan now share the same exclusion model as discovery, and whether the regression fixture proves a large ignored artifact tree is skipped rather than merely making this one repo faster. This should stay scoped to exclusion propagation and progress/degradation behavior.

[DeusData]

Thank you — of the several ignore-semantics PRs open right now, this is the one closest to the root cause: reusing discovery's materialized exclusion decisions instead of re-implementing matching per walker is exactly the right foundation, and the boundary-correct prefix match checks out. Three asks before merge:

1. A small regression test (repo rule: reproduce-first) — e.g. an excluded dir containing a package.json that pkgmap must not scan.
2. A body correction: cbm_scan_project_env_urls currently has no production callers (the envscan half keeps tests compiling but can't have contributed to the hang), and the PR also fixes the path-alias walk, which deserves to be named — it's likely essential to your 3.0s result.
3. If cheap for you: the incremental path (pipeline_incremental.c's ctx lacks excluded_dirs, so merge_pkg_entries still walks unexcluded on real incremental reindexes) — otherwise we'll file it as an immediate follow-up so the 15-minute walk can't survive there.
   With Windows support: CGO/tree-sitter build fails — WSL2 workaround included #1 this merges quickly — it's priority/high for good reason. Thanks again for the thorough diagnosis in pkgmap/envscan repo walks bypass gitignore exclusions - 15+ min apparent hang on repos with large ignored artifact dirs #792.

---

Update: to keep momentum on the bug backlog, we're going to carry the changes above over the line ourselves shortly — a distilled follow-up on current main implementing the notes in this thread, with you credited as Co-authored-by on the commit, and this PR closed referencing it. If you'd prefer to push the update yourself, just reply within the next couple of days and we'll gladly take yours instead. Thanks again for the contribution!

[DeusData]

Thank you — of all the ignore-semantics work in flight, yours was the one closest to the root cause: reusing discovery's materialized exclusion decisions instead of re-implementing matching per walker was exactly right, and your boundary-correct prefix predicate carried over verbatim. We took it over the line with the missing regression tests (control-first so they can't pass vacuously), the incremental-path plumbing (your fix's one gap — the ctx used by incremental reindexes didn't carry the exclusions, tracked as #804), and a small narrative correction (envscan's walker has no production callers). Merged as 2e7c64b (PR #811) with you credited as co-author — closes #792 and #804. Thanks again for the excellent diagnosis in #792!