fetch: rework negotiation tip options#2085
Open
derrickstolee wants to merge 7 commits intogitgitgadget:masterfrom
Open
fetch: rework negotiation tip options#2085derrickstolee wants to merge 7 commits intogitgitgadget:masterfrom
derrickstolee wants to merge 7 commits intogitgitgadget:masterfrom
Conversation
derrickstolee
changed the title
Author
|
/submit |
|
Submitted as pull.2085.git.1775658970.gitgitgadget@gmail.com To fetch this version into To fetch this version to local tag |
|
Junio C Hamano wrote on the Git mailing list (how to reply to this email): "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> Based on my understanding, the '--negotiation-tip' option is close but not
> quite what I want. I could have the client only advertise 'release' and
> 'main' and never advertise any user branches. But then we'd download all
> content from each user branch every time it updates. Perhaps this would
> happen even with opportunistic inclusion of more haves, but I'd like to
> explore this area more.
>
> There's also an issue that the '--negotiation-tip' feature doesn't seem to
> have a config key that enables it without CLI arguments. This is something
> that we could consider independently.
> ...
> Big picture questions to think about:
>
> * Is this a valuable addition to the fetch negotiation?
> * Is the interaction between --must-have and --negotiation-tip correct?
> * Is the "must have" name sensical to users? I expect that this only
> matters to experts, but I'm open to better names that could be more
> self-documenting.
> * Should we add a similar config key for --negotiation-tip?
Just like you, I hate the name "must have", but stepping back a bit,
would it work if we add a single boolean option that says "use the
negotiation tips as the primary source of 'have's you'd send, but
unlike the way how the original negotiation-tip feature worked
without this bit enabled, which did not send anything other than the
ones reachable by negotiation tips, do advertise opportunistically
other tips", essentially turning the existing negotiation-tips
feature into your must-have feature? You could even call the option
"--negotiate-better(=(yes|no))" or something, perhaps?
|
|
Derrick Stolee wrote on the Git mailing list (how to reply to this email): On 4/8/2026 2:59 PM, Junio C Hamano wrote:
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
>> Based on my understanding, the '--negotiation-tip' option is close but not
>> quite what I want. I could have the client only advertise 'release' and
>> 'main' and never advertise any user branches. But then we'd download all
>> content from each user branch every time it updates. Perhaps this would
>> happen even with opportunistic inclusion of more haves, but I'd like to
>> explore this area more.
>>
>> There's also an issue that the '--negotiation-tip' feature doesn't seem to
>> have a config key that enables it without CLI arguments. This is something
>> that we could consider independently.
>> ...
>> Big picture questions to think about:
>>
>> * Is this a valuable addition to the fetch negotiation?
>> * Is the interaction between --must-have and --negotiation-tip correct?
>> * Is the "must have" name sensical to users? I expect that this only
>> matters to experts, but I'm open to better names that could be more
>> self-documenting.
>> * Should we add a similar config key for --negotiation-tip?
>
> Just like you, I hate the name "must have", but stepping back a bit,
> would it work if we add a single boolean option that says "use the
> negotiation tips as the primary source of 'have's you'd send, but
> unlike the way how the original negotiation-tip feature worked
> without this bit enabled, which did not send anything other than the
> ones reachable by negotiation tips, do advertise opportunistically
> other tips", essentially turning the existing negotiation-tips
> feature into your must-have feature? You could even call the option
> "--negotiate-better(=(yes|no))" or something, perhaps?
I like this line of thought. You essentially want to use the existing
scaffolding of the --negotiate-tip option but change it from being a
_maximum set_ to being a _minimum set_.
## Considering --negotiation-tip-mode=<mode>
With that in mind, we could have an option like --negotiation-tip-mode
that takes one of a few options. Here are some word choices that I
immediately thought about:
* maximum|minimum: Are these sets a maximum set to choose from or a
minimum set to include?
* restrict|include: Are we restricting the haves to this set, or are
we including these tips by default?
* v1|v2: Use numerical versions to indicate the mode without commentary
so it could be extended in the future to v3 or more.
None of these jump out as a clear winner in my head. I'm interested in
more exploration of this space before rerolling.
## To mix modes, or not to mix modes?
One downside of this approach is that it disables the ability to use
both modes, at least in its most obvious implementation. What if someone
wants to force a minimum set of wants but also wants to focus the set
of additional wants to a specific ref space?
Theoretically, we could implement the option to toggle with multiple
options, using
--negotiation-tip-mode=minimum --negotiation-tip=refs/remotes/origin/main \
--negotiation-tip-mode=maximum --negotiation-tip=refs/remotes/origin/*
and as we process the --negotiation-tip options we'd put the input data
into different lists. Would this complexity be worth it compared to making
a new set of options?
This also becomes more complicated how to describe the interaction of
these options and any config options that enable them by default. When
exactly does the config get ignored in favor of CLI options?
## Considering --negotiation-(required|restricted)
We could alternatively create two new types of options that are clearly
related:
* --negotiation-restricted works exactly like --negotiation-tips and
would be a synonym (with the old one being "deprecated" in favor of
the newer one).
* --negotiation-required works like the --must-have in this series.
---
Thanks for considering these options with me. There is a lot of room
for creativity here. This series isn't even my first attempt at this
functionality because there are so many possible ways to accomplish
this goal.
Thanks,
-Stolee
|
The 'fetch follows tags by default' test sorts using 'sort -k 4', but for-each-ref output only has 3 columns. This relies on sort treating records with fewer fields as having an empty fourth field, which may produce unstable results depending on locale. Use 'sort -k 3' to match the actual number of columns in the output. Signed-off-by: Derrick Stolee <stolee@gmail.com>
derrickstolee
force-pushed
the
must-have
branch
from
6c227f1 to
c733b14
Compare
derrickstolee
changed the title
The --negotiation-tip option to 'git fetch' and 'git pull' allows users to specify that they want to focus negotiation on a small set of references. This is a _restriction_ on the negotiation set, helping to focus the negotiation when the ref count is high. However, it doesn't allow for the ability to opportunistically select references beyond that list. This subtle detail that this is a 'maximum set' and not a 'minimum set' is not immediately clear from the option name. This makes it more complicated to add a new option that provides the complementary behavior of a minimum set. For now, create a new synonym option, --negotiation-restrict, that behaves identically to --negotiation-tip. Update the documentation to make it clear that this new name is the preferred option, but we keep the old name for compatibility. Update a few warning messages with the new option, but also make them translatable with the option name inserted by formatting. At least one of these messages will be reused later for a new option. Signed-off-by: Derrick Stolee <stolee@gmail.com>
The previous change added the --negotiation-restrict synonym for the --negotiation-tips option for 'git fetch'. In anticipation of adding a new option that behaves similarly but with distinct changes to its behavior, rename the internal representation of this data from 'negotiation_tips' to 'negotiation_restrict_tips'. The 'tips' part is kept because this is an oid_array in the transport layer. This requires the builtin to handle parsing refs into collections of oids so the transport layer can handle this cleaner form of the data. Signed-off-by: Derrick Stolee <stolee@gmail.com>
In a previous change, the --negotiation-restrict command-line option of 'git fetch' was added as a synonym of --negotiation-tips. Both of these options restrict the set of 'haves' the client can send as part of negotiation. This was previously not available via a configuration option. Add a new 'remote.<name>.negotiationRestrict' multi-valued config option that updates 'git fetch <name>' to use these restrictions by default. If the user provides even one --negotiation-restrict argument, then the config is ignored. Signed-off-by: Derrick Stolee <stolee@gmail.com>
Add a new --negotiation-require option to 'git fetch', which ensures that certain ref tips are always sent as 'have' lines during fetch negotiation, regardless of what the negotiation algorithm selects. This is useful when the repository has a large number of references, so the normal negotiation algorithm truncates the list. This is especially important in repositories with long parallel commit histories. For example, a repo could have a 'dev' branch for development and a 'release' branch for released versions. If the 'dev' branch isn't selected for negotiation, then it's not a big deal because there are many in-progress development branches with a shared history. However, if 'release' is not selected for negotiation, then the server may think that this is the first time the client has asked for that reference, causing a full download of its parallel commit history (and any extra data that may be unique to that branch). This is based on a real example where certain fetches would grow to 60+ GB when a release branch updated. This option is a complement to --negotiation-restrict, which reduces the negotiation ref set to a specific list. In the earlier example, using --negotiation-restrict to focus the negotiation to 'dev' and 'release' would avoid those problematic downloads, but would still not allow advertising potentially-relevant user brances. In this way, the 'require' version solves the problem I mention while allowing negotiation to pick other references opportunistically. The two options can also be combined to allow the best of both worlds. The argument may be an exact ref name or a glob pattern. Non-existent refs are silently ignored. Also add --negotiation-require to 'git pull' passthrough options. Signed-off-by: Derrick Stolee <stolee@gmail.com>
…quire Add a new 'remote.<name>.negotiationRequire' multi-valued config option that provides default values for --negotiation-require when no --negotiation-require arguments are specified over the command line. This is a mirror of how 'remote.<name>.negotiationRestrict' specifies defaults for the --negotiation-restrict arguments. Each value is either an exact ref name or a glob pattern whose tips should always be sent as 'have' lines during negotiation. The config values are resolved through the same resolve_negotiation_require() codepath as the CLI options. This option is additive with the normal negotiation process: the negotiation algorithm still runs and advertises its own selected commits, but the refs matching the config are sent unconditionally on top of those heuristically selected commits. Signed-off-by: Derrick Stolee <stolee@gmail.com>
When push.negotiate is enabled, 'git push' spawns a child 'git fetch --negotiate-only' process to find common commits. Pass --negotiation-require and --negotiation-restrict options from the 'remote.<name>.negotiationRequire' and 'remote.<name>.negotiationRestrict' config keys to this child process. When negotiationRestrict is configured, it replaces the default behavior of using all remote refs as negotiation tips. This allows the user to control which local refs are used for push negotiation. When negotiationRequire is configured, the specified ref patterns are passed as --negotiation-require to ensure their tips are always sent as 'have' lines during push negotiation. This change also updates the use of --negotiation-tip into --negotiation-restrict now that the new synonym exists. Signed-off-by: Derrick Stolee <stolee@gmail.com>
derrickstolee
force-pushed
the
must-have
branch
from
c733b14 to
7cccf59
Compare
Author
|
/submit |
|
Submitted as pull.2085.v2.git.1776266066.gitgitgadget@gmail.com To fetch this version into To fetch this version to local tag |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fetch negotiation aims to find enough information from haves and wants such that the server can be reasonably confident that it will send all necessary objects and not too many "extra" objects that the client already has. However, this can break down if there are too many references, since Git truncates the list of haves based on a few factors (a 256 count limit or the server sending an ACK at the right time).
We already have the
--negotiation-tipfeature to focus the set of references that are used in negotiation, but I feel like this is designed backwards. I'd rather that we have a way to say "this is an important set of refs, but feel free to add more refs if needed" than "only use these refs for negotiation".Here's an example that demonstrates the problem. In an internal monorepo, developers work off of the 'main' branch so there are thousands of user branches that each add a few commits different from the 'main' branch. However, there is also a long-lived 'release' branch. This branch has a first-parent history that is parallel to 'main' and each of those commits is a merge whose second parent is a commit from 'main' that had a successful CI run. There are additional changes in the 'release' branch merge commits that add some changelog data, so there is a nontrivial set of novel blob content in that branch and not just a different set of commits.
The problem we had was that our georeplication system was regularly fetching from the origin and trying to get all data from all reachable branches. When the 'release' branch updated, the client would run out of haves before advertising its copy of the 'release' branch, but it would still list the new 'release' tip as a want. The server would then think that the client had never fetched that branch before and would send all of the changelog data from the whole history of the repo. (This led to a lot of downstream problems; we mitigated by setting a refspec that stopped fetching the 'release' branch, but this is not ideal.)
What I'd like is a mechanism to say "always advertise the client's version of 'main' and 'release' but also opportunistically include some user branches".
Based on my understanding, the '--negotiation-tip' option is close but not quite what I want. I could have the client only advertise 'release' and 'main' and never advertise any user branches. But then we'd download all content from each user branch every time it updates. Perhaps this would happen even with opportunistic inclusion of more haves, but I'd like to explore this area more.
There's also an issue that the '--negotiation-tip' feature doesn't seem to have a config key that enables it without CLI arguments. This is something that we could consider independently.
This patch series adds a new '--negotiation-require' option that does what I want: it makes sure that these references are used for 'have's during negotiation. In order to help clarify the difference between this and '--negotiation-tip', I first create a synonym called '--negotiation-restrict'.
Both of these options get 'remote.*.negotiation(Require|Restrict)' config options that enable their behavior by default.
During development, I had briefly considered only using config values, but that required some strange changes to care about the remote name in the transport layer. This was most different in the 'git push' integration. When I discovered the '--negotiation-tip' feature during the process, that gave me a clear pattern to follow with the addition of a config on top.
Updates in v2
This version is a near-complete rewrite based on feedback around the names of the previous option and config. The --negotiation-restrict option is new and the ability to set it via config is also new.
I did try to be more careful around translatable error messages, too.
Thanks,
-Stolee
cc: gitster@pobox.com
cc: ps@pks.im