Skip to content

Allow positive rollout refills and fix multi-turn sample merging - #4

Draft
neubig wants to merge 17 commits into
miles-tinkerfrom
science-rl-openhands
Draft

neubig wants to merge 17 commits into
miles-tinkerfrom
science-rl-openhands

Conversation

@neubig

@neubig neubig commented Oct 10, 2026 •

Copy link
Copy Markdown

Allow any positive rollout refill size independently of the target batch size, while retaining nonpositive-size rejection. Remove the obsolete adapter field from multi-turn sample construction to match the current Sample schema. The change remains general purpose and limited to three files.

Validation: existing refill regression cases cover sizes 1, 8, 32, 64 and rejection of 0/-1. All six refill cases and ten multi-turn merge tests pass inside the ABCI CUDA image. Existing full CI failures include unrelated fork/base issues (missing AdapterRef imports, SGLang seed dataclass assumptions and adapter reconciliation); this PR does not expand into those subsystems. Keep this PR open.

Tracking

Tracked in Linear as GNE-1217 — Commit Qwen3.8-27B science training recipe (canonical tracker for this PR; no separate GitHub issue).

Added by an AI agent (OpenHands) on behalf of Graham Neubig to organize previously unlinked PRs.

@neubig

neubig commented Oct 11, 2026

Copy link
Copy Markdown
Author

The live truncation audit found a limitation in the current v1 agentic integration, tracked in issue #5. In ABCI rollout 2, 104/256 samples are truncated: 43 hit the 32,768-token training cap and 61 end earlier with exactly 4,096 tokens in the final retained model generation. Every truncated sample has later session tokens excluded from training; 13 completed episodes ultimately pass the verifier. The verifier scores the full execution, while the optimizer sees only its retained prefix, so the current update omits later policy actions and their learning signal.

These are completed, graded episodes rather than discarded infrastructure errors. Exact-prefix/log-probability checks establish that retained tokens were sampled; they do not establish full trajectory coverage. Increasing only the per-call output allowance will not resolve post-execution clipping. Separate model-call truncation, training-context clipping, and SDK action-limit metrics are needed.

A reproducible audit and aggregate evidence are published in science-agent PR #17, commit 9587b6b. No private task content or traces are posted here. Both native jobs remain live; epoch-two acceptance is still pending, and this PR remains unmerged.

@neubig neubig changed the title Train Qwen3.5-4B scientific agents with native GRPO, OpenHands, and Sail Allow small rollout refills and fix multi-turn sample merging Oct 11, 2026
@neubig neubig changed the title Allow small rollout refills and fix multi-turn sample merging Allow positive rollout refills and fix multi-turn sample merging Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants