Skip to content

metro-file-map: Lazily stat files in the node crawler - #2104

Draft
robhogan wants to merge 1 commit into
mainfrom
pr2104
Draft

robhogan wants to merge 1 commit into
mainfrom
pr2104

Conversation

@robhogan

@robhogan robhogan commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Without Watchman, the node crawler lstats every file on every start, so that getDifference can tell what changed while Metro wasn't running:

activeCalls++;
fs.lstat(file, (err, stat) => {
activeCalls--;
if (!err && stat) {
result.set(childNormal, [
stat.mtime.getTime(),
stat.size,
0,
null,
isSymbolicLink ? 1 : 0,
]);
}
resolveIfDone();

Most of those files are never read, and an mtime only matters for a file with something derived from it (a SHA-1 or plugin data) - that's all a change can invalidate.

This takes the idea in @kitten's #1686 and makes it hold across restarts. A regular file is crawled without a stat, and its mtime is recorded when data is first derived from it, from a stat taken before it's read. A file with no mtime then has nothing that could be stale, so a start only stats the files that had an mtime before.

  • A recrawl while watching still stats every file, because consumers like PackageCache hold contents read outside the file map.
  • Symlinks are still read eagerly - every lookup through a link needs its target.
  • linkStats() gives a null modifiedTime and size for a file that hasn't been stat'd.
  • CACHE_BREAKER is bumped, or an existing cache would keep every file stat'd.

Benchmark

Without Watchman, this takes 40-45% off a cold or warm start on Linux and Windows (Linux warm: 649ms -> 371ms), 9-15% off on macOS, and 4-7% off RSS. Watch mode gains less on Linux and Windows, where setting up the watcher dominates - the rest of this stack deals with that. The cost is a stat when a file is first read: 10-20% more on a first SHA-1 pass over the bundle's files (8-47ms here), once per file per cache.

main This PR vs main
macOS: Cold start 156ms 132ms -15.4% (-23.3% to -8.7%)
Linux: Cold start 644ms 360ms -44.0% (-45.7% to -43.3%)
Windows: Cold start 369ms 208ms -43.7% (-46.6% to -42.7%)
macOS: Warm start 142ms 129ms -9.4% (-10.7% to -8.3%)
Linux: Warm start 649ms 371ms -42.8% (-43.3% to -42.5%)
Windows: Warm start 357ms 225ms -37.2% (-37.7% to -36.7%)
macOS: Watch-mode start, warm cache 143ms 130ms -9.6% (-10.6% to -8.7%)
Linux: Watch-mode start, warm cache 1,896ms 1,617ms -14.7% (-15.9% to -14.0%)
Windows: Watch-mode start, warm cache 2,554ms 2,429ms -4.9% (-5.4% to -4.3%)
macOS: First SHA-1 of the bundle's 3,125 files 75ms 83ms +10.4% (+8.6% to +11.8%)
Linux: First SHA-1 of the bundle's 3,125 files 110ms 131ms +19.5% (+17.2% to +20.5%)
Windows: First SHA-1 of the bundle's 3,125 files 270ms 317ms +17.2% (+13.7% to +19.5%)
macOS: RSS after a warm start 142.6MB 132.1MB -7.4% (-7.9% to -6.0%)
Linux: RSS after a warm start 175.1MB 164.1MB -6.3% (-6.9% to -6.0%)
Windows: RSS after a warm start 149.6MB 144.0MB -3.8% (-4.1% to -3.3%)

Changelog

 - **[Performance]**: metro-file-map: Start faster without Watchman, by not stat-ing files whose contents Metro hasn't read

Test plan

New unit tests, and a stats-between-starts-test.js that edits files between two runs of FileMap over a real directory and cache. SHA-1s, eager and lazy plugin data and a retargeted symlink are recomputed, and an unread file is stat'd only once it's read. These include the cases #1686 gets wrong.

Benchmark app (AI-driven harness, used throughout this stack): a React Native 0.87.1 app with 23 common dependencies, 34k files in the file map and 3.1k modules in its iOS dev bundle, started through Metro's DependencyGraph with resolver.useWatchman: false. Each round runs, per variant: 2 cold starts, a watch-mode session that computes the SHA-1s of the bundle's files (as transforming them does) and saves them to the cache, 3 warm starts and 3 watch-mode starts from that cache, and 2 watch-mode cold starts. 10 interleaved rounds per OS, alternating the variant order, then medians and bootstrap 95% CIs. Heap is measured after a forced GC. Machines: Mac mini M4 Pro (macOS, APFS, Node 26.11.1), Mac mini i5-4278U (Linux Mint 22.3, ext4, Node 24.21.0) and a Ryzen 7 7730U laptop (Windows 11, NTFS, Defender on, Node 24.21.0).

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 25, 2026
@robhogan
robhogan added this pull request to stack #2109 October 25, 2026 15:22
robhogan added a commit that referenced this pull request Oct 25, 2026
Without Watchman, `Watcher` starts its backends before the crawl and buffers their events, so that a change made during the crawl isn't missed:

https://github.com/react/metro/blob/56efab302200568c919ac1b3c28dacf8d84fc692/packages/metro-file-map/src/Watcher.js#L118-L127

For `FallbackWatcher` that means walking every root to watch each directory, and then the node crawler walks the same tree again. This lets the crawl do both. Just before the crawler lists a directory it passes it to the watcher, which watches it - the order the watcher's own walk uses, so a change still can't slip between the two. Directories the crawl skips as ignored are passed on too, and the watcher walks those itself unless it ignores them as well (eg `node_modules` without `retainAllFiles`).

The hand-off is an internal second parameter of the built-in node crawler, so there's no API change. A `crawlerFactory` crawler, a scoped recrawl and a root added with `addRoot` keep the watcher's own walk. The longer-term shape for this is a backend that owns both crawling and watching, which this would move into.

## Benchmark

This takes another 25% off watch-mode startup on Linux and Windows. With the rest of this stack, watch-mode startup without Watchman goes from 1,894ms to 508ms (-73%) on Linux, and from 2,580ms to 539ms (-79%) on Windows. On macOS, which uses `NativeWatcher`, PRs 2-5 measure the same as the first PR (124ms).

|  | `main` | #2107 | This PR | vs `main` | vs #2107 |
|---|---|---|---|---|---|
| Linux: Watch-mode start, warm cache | 1,894ms | 685ms | 508ms | -73.2% (-73.4% to -72.9%) | -25.9% (-26.4% to -25.4%) |
| Windows: Watch-mode start, warm cache | 2,580ms | 721ms | 539ms | -79.1% (-79.3% to -78.3%) | -25.2% (-27.3% to -22.3%) |
| Linux: Watch-mode start, cold cache | 1,876ms | 652ms | 472ms | -74.8% (-75.2% to -74.5%) | -27.6% (-28.5% to -26.7%) |
| Windows: Watch-mode start, cold cache | 2,567ms | 695ms | 516ms | -79.9% (-80.2% to -79.8%) | -25.7% (-28.2% to -24.9%) |
| Linux: Heap after a watch-mode start | 31.8MB | 28.3MB | 28.3MB | -11.1% (-11.1% to -11.0%) | -0.0% (-0.0% to -0.0%) |
| Windows: Heap after a watch-mode start | 31.9MB | 28.4MB | 28.4MB | -11.1% (-11.1% to -11.1%) | -0.0% (-0.0% to -0.0%) |
| Linux: RSS after a watch-mode start | 197.4MB | 176.3MB | 170.9MB | -13.4% (-15.0% to -12.0%) | -3.1% (-4.6% to -1.6%) |
| Windows: RSS after a watch-mode start | 205.7MB | 182.6MB | 184.6MB | -10.3% (-10.7% to -9.4%) | +1.1% (+0.9% to +1.4%) |

## Changelog

```
 - **[Performance]**: metro-file-map: Set up file watching during the crawl instead of after it, on Linux and Windows without Watchman
```

## Test plan

New `Watcher-test.js` cases: the crawl's directories reach a `FallbackWatcher`, and a recrawl or a `crawlerFactory` crawler leaves it to walk. New `watch-integration-test.js` case for a directory the crawl skips but the watcher doesn't. Benchmark as in #2104.
robhogan added a commit that referenced this pull request Oct 25, 2026
`NativeWatcher.startWatching` resolves as soon as `fs.watch` returns, but on macOS that can be before any changes are reported. libuv only signals another thread to (re)create the FSEvents stream, and the stream reports only changes made after it starts:

https://github.com/libuv/libuv/blob/2b4b918d3381100854250c89d5159d4206daafb7/src/unix/fsevents.c#L362-L372

So a change made straight after startup can be silently missed, and since #2054 started non-Watchman backends before the crawl, so can one made during the crawl. On a slow host the gap is long enough to hit regularly - it's the remaining flake in the Native watcher integration tests, where the suite's first event never arrives. 14 of 297 macOS test jobs since #1974 hit it, across all three Node versions.

This has `Watcher` confirm each root's watch before relying on it. Once every backend has started, it writes a health check file into each root and waits for it to be reported, as `checkHealth()` already does for the project root. If a watch can't be confirmed within 10s, Metro warns and carries on. Roots added later with `addRoot` are confirmed the same way.

A backend opts in by returning `false` from a new static `isLiveOnStart()`, which only `NativeWatcher` does - Watchman subscribes from the crawl's clock, and `FallbackWatcher`'s watches are live as soon as `fs.watch` returns. It's done in `Watcher`, after all the backends have started, because `NativeWatcher`s for multiple roots share one FSEvents stream, which restarts as each root is added.

This adds ~10ms to a watch-mode start on macOS without Watchman (145ms -> 155ms on our benchmark app).

Changelog:
```
 - **[Fix]**: Fix changes made immediately after startup sometimes being missed on macOS
```

Test plan:
New `Watcher-test.js` cases cover confirming each root before the crawl, warning and carrying on when a watch can't be confirmed, confirming a root added later, and leaving backends that are live on start alone. The backend integration tests now wait for a cookie of their own, as `Watcher` does.

To reproduce the race, I patched `NativeWatcher` to drop all events for its first 300ms. A `FileMap` in watch mode with two roots, writing a file to each as soon as `build()` resolves, sees neither change without this PR (3/3 runs) and both with it (3/3).

The `metro-file-map` suite passes on macOS, Linux and Windows. Benchmark (AI-driven): the harness from #2104, 8 interleaved rounds on an M4 Pro Mac mini, n=32 warm watch-mode starts per variant (+6.7%, 95% CI +5.2% to +8.0%).
robhogan added a commit that referenced this pull request Oct 25, 2026
Without Watchman, `Watcher` starts its backends before the crawl, so that nothing changed during the crawl is missed:

https://github.com/react/metro/blob/56efab302200568c919ac1b3c28dacf8d84fc692/packages/metro-file-map/src/Watcher.js#L118-L127

For `FallbackWatcher` that means walking the tree once to watch it and again to crawl it. This lets the crawl do both: the crawler hands each directory to the watcher just before listing it, the order the watcher's own walk uses. Directories the crawl skips as ignored are handed over too, for the watcher to walk unless it ignores them as well (eg `node_modules` without `retainAllFiles`).

This is internal to the built-in node crawler. A `crawlerFactory` crawler, a recrawl and a root added with `addRoot` keep the watcher's own walk.

## Benchmark

This takes another 28-30% off watch-mode startup on Linux and Windows. With the rest of this stack, watch-mode startup without Watchman goes from 1,896ms to 510ms (-73%) on Linux, and from 2,554ms to 522ms (-80%) on Windows. On macOS, which uses `NativeWatcher`, PRs 2-5 measure within noise of #2104.

|  | `main` | #2107 | This PR | vs `main` | vs #2107 |
|---|---|---|---|---|---|
| Linux: Watch-mode start, warm cache | 1,896ms | 707ms | 510ms | -73.1% (-73.4% to -72.8%) | -27.9% (-28.6% to -27.2%) |
| Windows: Watch-mode start, warm cache | 2,554ms | 721ms | 522ms | -79.6% (-79.7% to -79.4%) | -27.7% (-28.0% to -27.2%) |
| Linux: Watch-mode start, cold cache | 1,882ms | 699ms | 491ms | -73.9% (-74.5% to -73.5%) | -29.7% (-31.1% to -28.5%) |
| Windows: Watch-mode start, cold cache | 2,547ms | 709ms | 507ms | -80.1% (-80.2% to -80.0%) | -28.5% (-28.9% to -27.9%) |
| Linux: Heap after a watch-mode start | 31.8MB | 28.3MB | 28.3MB | -11.1% (-11.1% to -11.1%) | -0.1% (-0.1% to -0.1%) |
| Windows: Heap after a watch-mode start | 31.9MB | 28.4MB | 28.4MB | -11.1% (-11.1% to -11.1%) | -0.1% (-0.1% to -0.1%) |
| Linux: RSS after a watch-mode start | 197.6MB | 175.2MB | 174.3MB | -11.8% (-12.3% to -11.3%) | -0.5% (-0.9% to +0.0%) |
| Windows: RSS after a watch-mode start | 206.1MB | 182.9MB | 183.9MB | -10.7% (-11.3% to -10.3%) | +0.6% (+0.4% to +0.8%) |

## Changelog

```
 - **[Performance]**: metro-file-map: Set up file watching during the crawl instead of after it, on Linux and Windows without Watchman
```

## Test plan

New unit tests, and a `watch-integration-test.js` case for a directory the crawl skips but the watcher doesn't. Benchmark as in #2104.
Without Watchman, the node crawler `lstat`s every file on every start, so that `getDifference` can tell what changed while Metro wasn't running:

https://github.com/react/metro/blob/56efab302200568c919ac1b3c28dacf8d84fc692/packages/metro-file-map/src/crawlers/node/index.js#L83-L96

Most of those files are never read, and an mtime only matters for a file with something derived from it (a SHA-1 or plugin data) - that's all a change can invalidate.

This takes the idea in @kitten's #1686 and makes it hold across restarts. A regular file is crawled without a stat, and its mtime is recorded when data is first derived from it, from a stat taken before it's read. A file with no mtime then has nothing that could be stale, so a start only stats the files that had an mtime before.

 - A recrawl while watching still stats every file, because consumers like `PackageCache` hold contents read outside the file map.
 - Symlinks are still read eagerly - every lookup through a link needs its target.
 - `linkStats()` gives a null `modifiedTime` and `size` for a file that hasn't been stat'd.
 - `CACHE_BREAKER` is bumped, or an existing cache would keep every file stat'd.

## Benchmark

Without Watchman, this takes 40-45% off a cold or warm start on Linux and Windows (Linux warm: 649ms -> 371ms), 9-15% off on macOS, and 4-7% off RSS. Watch mode gains less on Linux and Windows, where setting up the watcher dominates - the rest of this stack deals with that. The cost is a stat when a file is first read: 10-20% more on a first SHA-1 pass over the bundle's files (8-47ms here), once per file per cache.

|  | `main` | This PR | vs `main` |
|---|---|---|---|
| macOS: Cold start | 156ms | 132ms | -15.4% (-23.3% to -8.7%) |
| Linux: Cold start | 644ms | 360ms | -44.0% (-45.7% to -43.3%) |
| Windows: Cold start | 369ms | 208ms | -43.7% (-46.6% to -42.7%) |
| macOS: Warm start | 142ms | 129ms | -9.4% (-10.7% to -8.3%) |
| Linux: Warm start | 649ms | 371ms | -42.8% (-43.3% to -42.5%) |
| Windows: Warm start | 357ms | 225ms | -37.2% (-37.7% to -36.7%) |
| macOS: Watch-mode start, warm cache | 143ms | 130ms | -9.6% (-10.6% to -8.7%) |
| Linux: Watch-mode start, warm cache | 1,896ms | 1,617ms | -14.7% (-15.9% to -14.0%) |
| Windows: Watch-mode start, warm cache | 2,554ms | 2,429ms | -4.9% (-5.4% to -4.3%) |
| macOS: First SHA-1 of the bundle's 3,125 files | 75ms | 83ms | +10.4% (+8.6% to +11.8%) |
| Linux: First SHA-1 of the bundle's 3,125 files | 110ms | 131ms | +19.5% (+17.2% to +20.5%) |
| Windows: First SHA-1 of the bundle's 3,125 files | 270ms | 317ms | +17.2% (+13.7% to +19.5%) |
| macOS: RSS after a warm start | 142.6MB | 132.1MB | -7.4% (-7.9% to -6.0%) |
| Linux: RSS after a warm start | 175.1MB | 164.1MB | -6.3% (-6.9% to -6.0%) |
| Windows: RSS after a warm start | 149.6MB | 144.0MB | -3.8% (-4.1% to -3.3%) |

## Changelog

```
 - **[Performance]**: metro-file-map: Start faster without Watchman, by not stat-ing files whose contents Metro hasn't read
```

## Test plan

New unit tests, and a `stats-between-starts-test.js` that edits files between two runs of FileMap over a real directory and cache. SHA-1s, eager and lazy plugin data and a retargeted symlink are recomputed, and an unread file is stat'd only once it's read. These include the cases #1686 gets wrong.

Benchmark app (AI-driven harness, used throughout this stack): a React Native 0.87.1 app with 23 common dependencies, 34k files in the file map and 3.1k modules in its iOS dev bundle, started through Metro's `DependencyGraph` with `resolver.useWatchman: false`. Each round runs, per variant: 2 cold starts, a watch-mode session that computes the SHA-1s of the bundle's files (as transforming them does) and saves them to the cache, 3 warm starts and 3 watch-mode starts from that cache, and 2 watch-mode cold starts. 10 interleaved rounds per OS, alternating the variant order, then medians and bootstrap 95% CIs. Heap is measured after a forced GC. Machines: Mac mini M4 Pro (macOS, APFS, Node 26.11.1), Mac mini i5-4278U (Linux Mint 22.3, ext4, Node 24.21.0) and a Ryzen 7 7730U laptop (Windows 11, NTFS, Defender on, Node 24.21.0).
robhogan added a commit that referenced this pull request Oct 26, 2026
Without Watchman, `Watcher` starts its backends before the crawl, so that nothing changed during the crawl is missed:

https://github.com/react/metro/blob/56efab302200568c919ac1b3c28dacf8d84fc692/packages/metro-file-map/src/Watcher.js#L118-L127

For `FallbackWatcher` that means walking the tree once to watch it and again to crawl it. This lets the crawl do both: the crawler hands each directory to the watcher just before listing it, the order the watcher's own walk uses. Directories the crawl skips as ignored are handed over too, for the watcher to walk unless it ignores them as well (eg `node_modules` without `retainAllFiles`).

This is internal to the built-in node crawler. A `crawlerFactory` crawler, a recrawl and a root added with `addRoot` keep the watcher's own walk.

## Benchmark

This takes another 28-30% off watch-mode startup on Linux and Windows. With the rest of this stack, watch-mode startup without Watchman goes from 1,896ms to 510ms (-73%) on Linux, and from 2,554ms to 522ms (-80%) on Windows. On macOS, which uses `NativeWatcher`, PRs 2-5 measure within noise of #2104.

|  | `main` | #2107 | This PR | vs `main` | vs #2107 |
|---|---|---|---|---|---|
| Linux: Watch-mode start, warm cache | 1,896ms | 707ms | 510ms | -73.1% (-73.4% to -72.8%) | -27.9% (-28.6% to -27.2%) |
| Windows: Watch-mode start, warm cache | 2,554ms | 721ms | 522ms | -79.6% (-79.7% to -79.4%) | -27.7% (-28.0% to -27.2%) |
| Linux: Watch-mode start, cold cache | 1,882ms | 699ms | 491ms | -73.9% (-74.5% to -73.5%) | -29.7% (-31.1% to -28.5%) |
| Windows: Watch-mode start, cold cache | 2,547ms | 709ms | 507ms | -80.1% (-80.2% to -80.0%) | -28.5% (-28.9% to -27.9%) |
| Linux: Heap after a watch-mode start | 31.8MB | 28.3MB | 28.3MB | -11.1% (-11.1% to -11.1%) | -0.1% (-0.1% to -0.1%) |
| Windows: Heap after a watch-mode start | 31.9MB | 28.4MB | 28.4MB | -11.1% (-11.1% to -11.1%) | -0.1% (-0.1% to -0.1%) |
| Linux: RSS after a watch-mode start | 197.6MB | 175.2MB | 174.3MB | -11.8% (-12.3% to -11.3%) | -0.5% (-0.9% to +0.0%) |
| Windows: RSS after a watch-mode start | 206.1MB | 182.9MB | 183.9MB | -10.7% (-11.3% to -10.3%) | +0.6% (+0.4% to +0.8%) |

## Changelog

```
 - **[Performance]**: metro-file-map: Set up file watching during the crawl instead of after it, on Linux and Windows without Watchman
```

## Test plan

New unit tests, and a `watch-integration-test.js` case for a directory the crawl skips but the watcher doesn't. Benchmark as in #2104.
robhogan added a commit that referenced this pull request Oct 26, 2026
Without Watchman, `Watcher` starts its backends before the crawl, so that nothing changed during the crawl is missed:

https://github.com/react/metro/blob/56efab302200568c919ac1b3c28dacf8d84fc692/packages/metro-file-map/src/Watcher.js#L118-L127

For `FallbackWatcher` that means walking the tree once to watch it and again to crawl it. This lets the crawl do both: the crawler hands each directory to the watcher just before listing it, the order the watcher's own walk uses. Directories the crawl skips as ignored are handed over too, for the watcher to walk unless it ignores them as well (eg `node_modules` without `retainAllFiles`).

This is internal to the built-in node crawler. A `crawlerFactory` crawler, a recrawl and a root added with `addRoot` keep the watcher's own walk.

## Benchmark

This takes another 28-30% off watch-mode startup on Linux and Windows. With the rest of this stack, watch-mode startup without Watchman goes from 1,896ms to 510ms (-73%) on Linux, and from 2,554ms to 522ms (-80%) on Windows. On macOS, which uses `NativeWatcher`, PRs 2-5 measure within noise of #2104.

|  | `main` | #2107 | This PR | vs `main` | vs #2107 |
|---|---|---|---|---|---|
| Linux: Watch-mode start, warm cache | 1,896ms | 707ms | 510ms | -73.1% (-73.4% to -72.8%) | -27.9% (-28.6% to -27.2%) |
| Windows: Watch-mode start, warm cache | 2,554ms | 721ms | 522ms | -79.6% (-79.7% to -79.4%) | -27.7% (-28.0% to -27.2%) |
| Linux: Watch-mode start, cold cache | 1,882ms | 699ms | 491ms | -73.9% (-74.5% to -73.5%) | -29.7% (-31.1% to -28.5%) |
| Windows: Watch-mode start, cold cache | 2,547ms | 709ms | 507ms | -80.1% (-80.2% to -80.0%) | -28.5% (-28.9% to -27.9%) |
| Linux: Heap after a watch-mode start | 31.8MB | 28.3MB | 28.3MB | -11.1% (-11.1% to -11.1%) | -0.1% (-0.1% to -0.1%) |
| Windows: Heap after a watch-mode start | 31.9MB | 28.4MB | 28.4MB | -11.1% (-11.1% to -11.1%) | -0.1% (-0.1% to -0.1%) |
| Linux: RSS after a watch-mode start | 197.6MB | 175.2MB | 174.3MB | -11.8% (-12.3% to -11.3%) | -0.5% (-0.9% to +0.0%) |
| Windows: RSS after a watch-mode start | 206.1MB | 182.9MB | 183.9MB | -10.7% (-11.3% to -10.3%) | +0.6% (+0.4% to +0.8%) |

## Changelog

```
 - **[Performance]**: metro-file-map: Set up file watching during the crawl instead of after it, on Linux and Windows without Watchman
```

## Test plan

New unit tests, and a `watch-integration-test.js` case for a directory the crawl skips but the watcher doesn't. Benchmark as in #2104.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant