perf: read and write large requests past the stream buffer - #87
Open
francisdb wants to merge 1 commit into
Open
Conversation
Reads and writes of 64KiB or more go straight to the underlying file, and read_to_end reserves the remaining length once (bounded by what the chain can hold) instead of growing through the buffer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every read went through the stream's buffer, which each new
Streamgrew from 1 KiB up to 1 MiB (zero-filled at each step) and then copied out of, soread_to_endon a large stream cost a zero fill and two copies per byte plus the reallocations of the caller's vector.Reads and writes of 64 KiB or more now go straight to the underlying file.
read_to_endreserves the remaining length once and reads into it directly, in 16 MiB pieces. The reservation is bounded by what the stream's chain can actually hold, so a malformed file claiming an enormous length cannot make it allocate for the claim (the fuzz target caught exactly that in an earlier version).Tests: mixed large and small reads and writes including a large write into the middle of a stream; buffered writes visible to a following direct read;
read_to_endof small and mini streams, of a stream longer than one piece, and of a stream whose entry claims about 4 GB (must fail without reserving for it).