jq/yq: a #1194 structurally malformed key silently vanishes via keys_unsorted/materialize but raises via keys

[newhoggy]

Severity: Low-Medium — reachable via the shipped sjq/syq CLIs (unlike #1678), but only
for the narrow #1194 shape (a key the format's grammar never allowed at all, e.g. a bare
non-string JSON key {123: 1}), not the far more common #1247/#1642 undecodable-key shape,
which this branch already fixed everywhere.

Summary

Found while reviewing/fixing #1642, which closed the "same document, different answer"
inconsistency for a key whose bytes won't decode (#1247's fault). That fix left a
structurally-analogous, pre-existing gap open on the adjacent #1194 axis: a key the format's
grammar never allowed at all still raises on some routes and silently vanishes on others.

keys() (src/jq/document.rs), effective_keys, to_owned_at_depth,
to_owned_cursor_at_depth, to_owned_with_comments_at_depth, and the ToEntries builtin
(src/jq/eval_generic.rs) all do:

let Some(key) = key_display_string(&field.key) else {
    return Err(f.malformed_member_error());
};

key_display_string returns None exactly for a #1194 key (see its doc comment), so all of
these correctly raise.

But lazy_keys_array_to_owned and cursor_to_owned_at_depth (src/jq/lazy.rs), and
stream_lazy_keys_json/stream_lazy_keys_yaml (src/jq/stream.rs), instead do:

if let Some(key) = key_display_string(&field.key) {
    // push / write
}

with no else — a None (the #1194 case) just silently drops the field, no error. This
predates #1642 (the old code had the identical asymmetry via field.key_str()/if let StandardJson::String(k) = key), so #1642 didn't introduce it — but its "one document, many
answers" framing makes this easy to mistake as already closed, since it's exactly that bug
class, just on the #1194 axis instead of #1247's.

Repro

$ echo '{123:1,"a":1}' | sjq -c 'keys'           # raises (correct)
$ echo '{123:1,"a":1}' | sjq -c 'keys_unsorted'  # ["a"]   <- silently one entry short, exit 0
$ echo '{123:1,"a":1}' | sjq -c 'length'         # 2

keys and length/keys_unsorted disagree with each other on the same document, depending
on which builtin is asked — the same failure mode #1642's own issue description named.

Suggested direction

Not attempted; same two-way choice #1642 itself faced (raise everywhere vs. skip/preserve
everywhere), and picking "raise everywhere" here is more invasive than #1642's fix was:
stream_lazy_keys_json/stream_lazy_keys_yaml write through a core::fmt::Write-style
signature with no error channel today, the same constraint that made #1247's original
design doc (docs/plan/decode-failure-routing.md) split out and defer the value-side
streaming fix as its own stage rather than attempt it inline. lazy_keys_array_to_owned's
signature already returns Result, so that one site at least is a smaller lift.

Related

• jq: an undecodable object key gets four different answers from the same document (keys/to_entries/has raise; length/keys_unsorted/identity do not) #1642 — the review that found this; fixed the jq: one StandardJson::Error -> OwnedValue::Null arm remains in eval.rs's own to_owned_at_depth — prove unreachable or route it #1247/jq: an undecodable object key gets four different answers from the same document (keys/to_entries/has raise; length/keys_unsorted/identity do not) #1642 axis of the same class of bug,
  left this jq: a malformed object field (e.g. {invalid}) silently disappears instead of erroring #1194 axis open.
• jq: a malformed object field (e.g. {invalid}) silently disappears instead of erroring #1194 — the original structural-key issue this asymmetry belongs to.
• jq: one StandardJson::Error -> OwnedValue::Null arm remains in eval.rs's own to_owned_at_depth — prove unreachable or route it #1247 — established the EvalError-raising convention the fixed call sites already use.

[newhoggy]

Bill (grinding-backlog): investigated for pickup, found the repro as originally
written is now stale — leaving this open rather than implementing against an
outdated model.

3316a871c ("fix(jq, json): raise on a malformed object member wherever it
materializes", refs #1194, landed after this issue was filed) already broadened the
raise-on-malformed-key fix well beyond what this issue's body described as already
fixed. Live-tested against current main:

$ echo '{123:1,"a":1}' | sjq -c 'keys'
jq: error (at <stdin>:1): Invalid JSON text: expected string key, found '1'
$ echo '{123:1,"a":1}' | sjq -c 'keys_unsorted'
jq: error (at <stdin>:1): Invalid JSON text: expected string key, found '1'

Both now raise identically — the original "same document, different answer" repro no
longer reproduces. stream_lazy_keys_json/stream_lazy_keys_yaml (src/jq/stream.rs)
still literally contain the silent-skip code this issue names (confirmed by reading
current source), but I haven't yet traced why the plain keys_unsorted -c CLI path no
longer reaches it — worth a dedicated investigation rather than guessing.

New, different divergences turned up in its place while poking at this, not yet
triaged or filed:

• sjq -c 'keys_unsorted | .[0]' on the same document prints bare 123 (exit 0) —
  no error, and not a valid re-derivation of a string key either.
• sjq -c '0' on the same malformed document exits 0 printing 0 with no input
  validation at all, where real jq 1.7.1 parse-errors even a pure-constant filter
  (jq -c '0' also errors on this input, confirmed live) — looks like succinctly
  skips reading/validating input entirely when the filter never references .,
  which real jq does not do.
• sjq -c '1 + 1' on the same document does raise, unlike bare 0 — so whatever
  decides "does this filter need the input" isn't a simple "references . at all"
  check either.

3316a871c's own commit message names two specific remaining lazy/streaming gaps
(.[] via LazySource::Values, and [xyz123] | . via print_json's streaming
StandardJson::Error arm) as deliberately left open (#1247's design-doc Stage 6) —
neither matches stream_lazy_keys_json by name, so it's unclear whether that function
is a third still-open gap or already dead code for the CLI path. Whoever picks this
up next should start by tracing exactly what code path sjq -c 'keys_unsorted'
actually takes on current main before touching anything, rather than trusting this
issue's original repro.

Not claiming — leaving in Ready for a fresh pass with headroom to trace this
properly.

[newhoggy]

Claiming this one (agent: Cody, branch issue-1679-structurally-malformed-key-vanishes).

[newhoggy]

Fixed in #1811 (merged as `9f2e36e81`). Raised on the #1194-shaped structurally malformed key in all 5 call sites that were silently dropping it:

• `lazy_keys_array_to_owned` / `cursor_to_owned_at_depth` (`src/jq/lazy.rs`)
• `stream_lazy_keys_json` / `stream_lazy_keys_yaml` (`src/jq/stream.rs`)
• `write_json_at_depth`'s `LazyKeysArray` arm (`src/jq/lazy.rs`) — a 5th site found by code review, not in the original issue body: it was writing the malformed key's raw bytes verbatim, producing invalid JSON (`{123:1}` → `[123]`) rather than merely dropping the field.

`keys`/`keys_unsorted`/`length` now agree on every #1194-shaped document.