You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
run_jq/run_yq rely on BufWriter's Drop to flush on most early
returns; #1563/#1673 patched three sites in yq_runner.rs's 'm2_files
loop and one in jq_runner.rs's lazy-path loop where a later
file's/document's error could otherwise skip an explicit flush and
silently lose an earlier file's already-buffered output (Drop's own
flush swallows any error rather than propagating it).
A multi-angle review of that PR (10 independent finder passes) confirmed
the fix is correct but incomplete: the same failure shape exists at
several more sites in both files, all inside the very functions #1673
touched. Fixing them required three follow-up commits within the PR
before converging even on the narrow set the review happened to check —
strong evidence that patching individual ?-return sites by hand isn't
converging, and a fresh audit would very likely turn up still more.
Known remaining unguarded sites (each: a loop writes real output to
the shared writer/out across multiple iterations, and a later
iteration's write/build call can fail via bare ? without an explicit
flush first):
yq_runner.rs: the stream_cursor! macro (~3299-3367, invoked at
3399/3447/3509/3572) and the direct stream_maybe_colored/writeln!
calls at ~3438/3545/3554/3563 — the actual per-document streaming
writes inside the 'm2_files loop fix(jq): flush stdout explicitly before --validate's lazy-path early return #1673 already touched for its read_file/yaml_validate_guard/YamlIndex::build sites.
yq_runner.rs: the general/DOM "collect" output path (~4383-4431) —
the default fallback for any multi-file run not eligible for M2
streaming (e.g. --pretty/-P).
yq_runner.rs: the --raw-input per-line loop (~3843-3855) and the --eval-all per-result loop (~3652-3671).
jq_runner.rs: the DSV streaming loop's write_output(...)? (~1076;
its own halt path three lines below already flushes explicitly,
showing the ordinary-error case was just missed).
jq_runner.rs: the materializing branch's three loops — null_input
arm (~1397), the input/inputs queue-pop arm (~1426), and the plain
per-input arm (~1444) — each calling write_output(...)? across
multiple documents. This directly contradicts fix(jq): flush stdout explicitly before --validate's lazy-path early return #1673's own added
comment claiming this branch's early returns "provably run before
anything is ever written to out" — true only for the single get_inputs failure before the loop starts, not for iteration N>=2 of
any of these three loops.
Proposed direction (from the review's "right altitude" finding,
worth validating before implementing): move the safety net to writer construction (jq_runner.rs and yq_runner.rs's BufWriter::new(stdout.lock())) instead of continuing to annotate each
early return by hand. A Drop-based guard can't propagate a Result,
but it can at least make a flush failure loud (e.g. log to stderr)
instead of silent -- closing the whole bug class in one place rather
than requiring another manual audit pass every time a new early return
is added to either ~5,000-line function. A full Result-propagating fix
would need a finish(self) -> Result<()> consumed at every exit
instead, which is a larger refactor of both functions' control flow.
Not urgent (the concrete --validate scenario #1563 named, plus the
directly adjacent sites in the same loop, are already fixed) but worth
tracking so it doesn't silently regress the "we already fixed this"
assumption the next time someone touches either runner's output loops.
run_jq/run_yqrely onBufWriter'sDropto flush on most earlyreturns; #1563/#1673 patched three sites in
yq_runner.rs's'm2_filesloop and one in
jq_runner.rs's lazy-path loop where a laterfile's/document's error could otherwise skip an explicit flush and
silently lose an earlier file's already-buffered output (
Drop's ownflush swallows any error rather than propagating it).
A multi-angle review of that PR (10 independent finder passes) confirmed
the fix is correct but incomplete: the same failure shape exists at
several more sites in both files, all inside the very functions #1673
touched. Fixing them required three follow-up commits within the PR
before converging even on the narrow set the review happened to check —
strong evidence that patching individual
?-return sites by hand isn'tconverging, and a fresh audit would very likely turn up still more.
Known remaining unguarded sites (each: a loop writes real output to
the shared
writer/outacross multiple iterations, and a lateriteration's write/build call can fail via bare
?without an explicitflush first):
yq_runner.rs: thestream_cursor!macro (~3299-3367, invoked at3399/3447/3509/3572) and the direct
stream_maybe_colored/writeln!calls at ~3438/3545/3554/3563 — the actual per-document streaming
writes inside the
'm2_filesloop fix(jq): flush stdout explicitly before --validate's lazy-path early return #1673 already touched for itsread_file/yaml_validate_guard/YamlIndex::buildsites.yq_runner.rs: the general/DOM "collect" output path (~4383-4431) —the default fallback for any multi-file run not eligible for M2
streaming (e.g.
--pretty/-P).yq_runner.rs: the--raw-inputper-line loop (~3843-3855) and the--eval-allper-result loop (~3652-3671).jq_runner.rs: the DSV streaming loop'swrite_output(...)?(~1076;its own halt path three lines below already flushes explicitly,
showing the ordinary-error case was just missed).
jq_runner.rs: the materializing branch's three loops —null_inputarm (~1397), the
input/inputsqueue-pop arm (~1426), and the plainper-input arm (~1444) — each calling
write_output(...)?acrossmultiple documents. This directly contradicts fix(jq): flush stdout explicitly before --validate's lazy-path early return #1673's own added
comment claiming this branch's early returns "provably run before
anything is ever written to
out" — true only for the singleget_inputsfailure before the loop starts, not for iteration N>=2 ofany of these three loops.
jq_runner.rs: the identity fast path'sout.write_all(...)?calls(~1207), inside the very loop fix(jq): flush stdout explicitly before --validate's lazy-path early return #1673 already patched for its
--validateand value-materialization error paths.Proposed direction (from the review's "right altitude" finding,
worth validating before implementing): move the safety net to writer
construction (
jq_runner.rsandyq_runner.rs'sBufWriter::new(stdout.lock())) instead of continuing to annotate eachearly return by hand. A
Drop-based guard can't propagate aResult,but it can at least make a flush failure loud (e.g. log to stderr)
instead of silent -- closing the whole bug class in one place rather
than requiring another manual audit pass every time a new early return
is added to either ~5,000-line function. A full
Result-propagating fixwould need a
finish(self) -> Result<()>consumed at every exitinstead, which is a larger refactor of both functions' control flow.
Not urgent (the concrete
--validatescenario #1563 named, plus thedirectly adjacent sites in the same loop, are already fixed) but worth
tracking so it doesn't silently regress the "we already fixed this"
assumption the next time someone touches either runner's output loops.