perf: reuse a chain's sector list across opens instead of rewalking the FAT - #89
Open
francisdb wants to merge 2 commits into
Open
perf: reuse a chain's sector list across opens instead of rewalking the FAT#89francisdb wants to merge 2 commits into
francisdb wants to merge 2 commits into
Conversation
…he FAT A stream opens its chain anew for every buffer refill, and Chain::new walks the whole FAT chain each time, so reading or writing a long stream costs O(n^2) in its sector count. The allocator now keeps the sector list of the last dropped chain and hands it to the next open of the same start sector; any FAT change discards it.
The existing read benches read each stream with a single read_to_end. Add a full io::copy of a 256 MiB stream and a thousand seek-and-read pairs over it, the two access patterns that refill the stream buffer most often.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the remaining cause of #57.
A
Streamopens its chain anew for every buffer refill, andChain::newwalks the whole FAT chain each time, so reading a long stream costs O(n^2) in its sector count. #79 hid most of it with the 1 MiB buffer, but each refill of a 1 GiB stream still walked 262144 FAT entries.The allocator now keeps the sector list of the last dropped
Chainand hands it to the next open of the same start sector; anyset_fatdiscards it.Chainkeeps its list in step on shrink and free so the list handed back always matches the FAT. Same idea as #81, applied to any chain.New benches, one 256 MiB stream in memory:
The #57 scenario (1 GiB stream, copy the first 50 MiB): 34 ms on 0.14.0, 14 ms here.