Skip to content

jq,yq: run_jq/run_yq rely on Drop's silent flush at many more early-return sites beyond #1563 #1680

Description

@newhoggy

run_jq/run_yq rely on BufWriter's Drop to flush on most early
returns; #1563/#1673 patched three sites in yq_runner.rs's 'm2_files
loop and one in jq_runner.rs's lazy-path loop where a later
file's/document's error could otherwise skip an explicit flush and
silently lose an earlier file's already-buffered output (Drop's own
flush swallows any error rather than propagating it).

A multi-angle review of that PR (10 independent finder passes) confirmed
the fix is correct but incomplete: the same failure shape exists at
several more sites in both files, all inside the very functions #1673
touched. Fixing them required three follow-up commits within the PR
before converging even on the narrow set the review happened to check —
strong evidence that patching individual ?-return sites by hand isn't
converging, and a fresh audit would very likely turn up still more.

Known remaining unguarded sites (each: a loop writes real output to
the shared writer/out across multiple iterations, and a later
iteration's write/build call can fail via bare ? without an explicit
flush first):

  • yq_runner.rs: the stream_cursor! macro (~3299-3367, invoked at
    3399/3447/3509/3572) and the direct stream_maybe_colored/writeln!
    calls at ~3438/3545/3554/3563 — the actual per-document streaming
    writes inside the 'm2_files loop fix(jq): flush stdout explicitly before --validate's lazy-path early return #1673 already touched for its
    read_file/yaml_validate_guard/YamlIndex::build sites.
  • yq_runner.rs: the general/DOM "collect" output path (~4383-4431) —
    the default fallback for any multi-file run not eligible for M2
    streaming (e.g. --pretty/-P).
  • yq_runner.rs: the --raw-input per-line loop (~3843-3855) and the
    --eval-all per-result loop (~3652-3671).
  • jq_runner.rs: the DSV streaming loop's write_output(...)? (~1076;
    its own halt path three lines below already flushes explicitly,
    showing the ordinary-error case was just missed).
  • jq_runner.rs: the materializing branch's three loops — null_input
    arm (~1397), the input/inputs queue-pop arm (~1426), and the plain
    per-input arm (~1444) — each calling write_output(...)? across
    multiple documents. This directly contradicts fix(jq): flush stdout explicitly before --validate's lazy-path early return #1673's own added
    comment claiming this branch's early returns "provably run before
    anything is ever written to out" — true only for the single
    get_inputs failure before the loop starts, not for iteration N>=2 of
    any of these three loops.
  • jq_runner.rs: the identity fast path's out.write_all(...)? calls
    (~1207), inside the very loop fix(jq): flush stdout explicitly before --validate's lazy-path early return #1673 already patched for its
    --validate and value-materialization error paths.

Proposed direction (from the review's "right altitude" finding,
worth validating before implementing): move the safety net to writer
construction (jq_runner.rs and yq_runner.rs's
BufWriter::new(stdout.lock())) instead of continuing to annotate each
early return by hand. A Drop-based guard can't propagate a Result,
but it can at least make a flush failure loud (e.g. log to stderr)
instead of silent -- closing the whole bug class in one place rather
than requiring another manual audit pass every time a new early return
is added to either ~5,000-line function. A full Result-propagating fix
would need a finish(self) -> Result<()> consumed at every exit
instead, which is a larger refactor of both functions' control flow.

Not urgent (the concrete --validate scenario #1563 named, plus the
directly adjacent sites in the same loop, are already fixed) but worth
tracking so it doesn't silently regress the "we already fixed this"
assumption the next time someone touches either runner's output loops.

Metadata

Metadata

Assignees

No one assigned

    Labels

    SonnetSuitable for a Sonnet-class model to implement

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions