Get rid of LLM disclosure checkboxes - #160785
Conversation
These had a bunch of issues: - They're extra work every time someone opens a PR - They didn't show up at all if people used `gh pr create` - They had "bad vibes" -- reviewers often don't want to think about LLMs, and adding a checkbox makes them very "in-your-face" for the reviewer. - Every PR now shows "1 of 2 tasks", which is useless noise. - The hovered PR description is now useless. Replace them with an HTML comment that says "remember to disclose if you used an LLM". This seems ok and low-noise for now. If we find that people are ignoring the comment, we could edit the triagebot welcome message to include a reminder and a link to the policy.
rustbot
added
the
S-waiting-on-review
|
rustbot has assigned @Mark-Simulacrum. Use Why was this reviewer chosen?The reviewer was selected based on:
|
| If you do not check a box, a reviewer may ask you whether an LLM was involved. | ||
| If you used an LLM to generate code, please disclose that according to our [guidelines][disclosure guidelines]. | ||
| LLM contributions are not banned, but are held to a higher standard of review and correctness. | ||
| If you do not want your disclosure to be part of the permanent git history, add `<!-- homu-ignore:start` before it. |
There was a problem hiding this comment.
Is the tooling smart enough to not parse the homu-ignore when it's within a comment?
|
A potential problem is: If a PR doesn't mention LLMs, we're left guessing whether it's a human who has seen the instructions, or if it's an LLM that hasn't seen the instructions. |
|
yes. that’s why the penalty is “a reviewer might ask you whether your PR is LLM-generated”, not “close the PR on sight”. |
| If you do not check a box, a reviewer may ask you whether an LLM was involved. | ||
| If you used an LLM to generate code, please disclose that according to our [guidelines][disclosure guidelines]. | ||
| LLM contributions are not banned, but are held to a higher standard of review and correctness. | ||
| If you do not want your disclosure to be part of the permanent git history, add `<!-- homu-ignore:start` before it. |
There was a problem hiding this comment.
Could we just drop this line instead, please? The disclosure is useful information.
There was a problem hiding this comment.
people should have control over what they put in a permanent record.
| Please read our [LLM policy] before opening a PR, | ||
| and check one of the boxes above to indicate whether you've used an LLM. | ||
| If you do not check a box, a reviewer may ask you whether an LLM was involved. | ||
| If you used an LLM to generate code, please disclose that according to our [guidelines][disclosure guidelines]. |
There was a problem hiding this comment.
s/code/any part of this PR, including the PR description/ ?
Point being that the prohibition is on code, comments, docs, and the PR description.
There was a problem hiding this comment.
Also, the disclosure guidelines don't mention or link to the LLM policy. They briefly mention requiring a mentor, but the general tone makes it sound like LLM-generated contributions are by default accepted.
There was a problem hiding this comment.
Oh, I see now the policy is linked to from the parent page (https://rustc-dev-guide.rust-lang.org/llm-guidance.html). But if someone doesn't navigate back to that parent page, the overall tenor of the discussion seems misleading.
There was a problem hiding this comment.
let’s please not turn this into a review of the dev guide by proxy. open an issue or PR there if you want to suggest changes.
s/code/any part of this PR, including the PR description/ ?
👍 i’ll write something up
7 participants
These had a bunch of issues:
gh pr createReplace them with an HTML comment that says "remember to disclose if you used an LLM". This seems ok and low-noise for now.
If we find that people are ignoring the comment,
we could edit the triagebot welcome message to include a reminder and a link to the policy (cc @Kobzol, i believe you'd planned to do this already).