Hi, thanks for releasing the OEL code. We reproduced the Frozen Lake in-context experiments from the pinned commit (4f2a9de) and have two questions plus one observation we'd like to check with you.
-
How is the "In-Context" column of Table 2 aggregated over the K=10 experiences?
ray_trainer.py supports in-context evaluation via trainer.eval_prepend_experience / trainer.experience_path (one experience file per invocation), but scripts/textgame_eval.sh doesn't expose these keys, and we couldn't find the outer loop that produces the reported numbers. Is it: run the held-out 128-seed evaluation once per experience file, then average the 10 pass rates? And does "averaged over 10 random seeds" refer to the K=10 extraction seeds, or to independent evaluation seeds? (This matters more than it looks: in our runs, per-draw quality for the 1.7B self experience varies hugely — per-draw pass rates ranged 0.26–0.91, sd ≈ 0.20 — while 4B/8B experiences are stable, sd 0.03–0.06.)
-
What is the step-level truncation rate at max_tokens=1024?
With Qwen3-1.7B in thinking mode, we measure that most per-step responses naturally exceed 1024 tokens when generation is not budget-limited (mean ≈ 1,500 tokens; 50–74% of steps exceed 1024, depending on the experience arm). Could you share the truncation / unparseable-action rates you observed under the paper's configuration?
-
Observation: at a non-binding budget, the self-vs-cross ordering reverses.
Matching your protocol (official 128 held-out seeds, K=10 with seeds 50..500, val sampling 0.7/0.8/20, v3/v4 extraction prompts verbatim, sequential accumulation) but raising the per-step budget to 4096 so that >96% of actions parse, we get In-Context pass rates of: no-experience 0.48, self (1.7B) 0.60, Qwen3-4B 0.88, Qwen3-8B 0.83, larger models ≈ 0.85 — i.e., cross-model experience is decisively better than self experience, the opposite ordering of Table 2 (23.8 vs 18.0). One mechanism consistent with both results: self-extracted experience makes the model's thinking markedly shorter (mean 1,217 vs ≈1,550 tokens for cross arms), so at a 1024 budget the self arm has roughly twice the budget-survival rate of cross arms. If that's what drives the published ordering, the "off-policy knowledge is worse than self knowledge" conclusion may be specific to the 1024 budget rather than a property of the knowledge itself.
Hi, thanks for releasing the OEL code. We reproduced the Frozen Lake in-context experiments from the pinned commit (4f2a9de) and have two questions plus one observation we'd like to check with you.
How is the "In-Context" column of Table 2 aggregated over the K=10 experiences?
ray_trainer.py supports in-context evaluation via trainer.eval_prepend_experience / trainer.experience_path (one experience file per invocation), but scripts/textgame_eval.sh doesn't expose these keys, and we couldn't find the outer loop that produces the reported numbers. Is it: run the held-out 128-seed evaluation once per experience file, then average the 10 pass rates? And does "averaged over 10 random seeds" refer to the K=10 extraction seeds, or to independent evaluation seeds? (This matters more than it looks: in our runs, per-draw quality for the 1.7B self experience varies hugely — per-draw pass rates ranged 0.26–0.91, sd ≈ 0.20 — while 4B/8B experiences are stable, sd 0.03–0.06.)
What is the step-level truncation rate at max_tokens=1024?
With Qwen3-1.7B in thinking mode, we measure that most per-step responses naturally exceed 1024 tokens when generation is not budget-limited (mean ≈ 1,500 tokens; 50–74% of steps exceed 1024, depending on the experience arm). Could you share the truncation / unparseable-action rates you observed under the paper's configuration?
Observation: at a non-binding budget, the self-vs-cross ordering reverses.
Matching your protocol (official 128 held-out seeds, K=10 with seeds 50..500, val sampling 0.7/0.8/20, v3/v4 extraction prompts verbatim, sequential accumulation) but raising the per-step budget to 4096 so that >96% of actions parse, we get In-Context pass rates of: no-experience 0.48, self (1.7B) 0.60, Qwen3-4B 0.88, Qwen3-8B 0.83, larger models ≈ 0.85 — i.e., cross-model experience is decisively better than self experience, the opposite ordering of Table 2 (23.8 vs 18.0). One mechanism consistent with both results: self-extracted experience makes the model's thinking markedly shorter (mean 1,217 vs ≈1,550 tokens for cross arms), so at a 1024 budget the self arm has roughly twice the budget-survival rate of cross arms. If that's what drives the published ordering, the "off-policy knowledge is worse than self knowledge" conclusion may be specific to the 1024 budget rather than a property of the knowledge itself.