A query with non-ASCII text before an error makes prqlc panic, or put the error label in the wrong place:
$ printf 'from t\nderive {x = "éééééééééé"}\nselect {x, foo.bar.baz}\n' | prqlc compile
The application panicked (crashed).
Message: span Some(1:54-65) is out of bounds of the source (len = 57)
Location: prqlc/prqlc/src/error_message.rs:153
Reproduced on 93ab9fc. With fewer multi-byte characters it doesn't panic, but the label moves: after # café on the first line, the "Unknown name" label for a later column sits one column to the right of the name. prqlc debug annotate places frames with the same spans, so it can attach them to the wrong lines.
Cause. Tokens and error spans use different units.
- The lexer builds token spans straight from chumsky's
SimpleSpan (lexer/mod.rs:130-148). Over &str input, those are byte offsets.
- Everything downstream reads spans as char offsets: ariadne's
Source, and the lexer's own error path, convert_lexer_error, which converts byte offsets to char offsets on purpose.
- So every multi-byte character shifts later spans right by its extra bytes. Once a span runs past the source's char length, the
assert! in error_message.rs panics.
The chumsky 0.10 migration (#5223) appears to be where token spans became byte offsets.
Why this needs a decision. One of two conventions has to be picked, and spans are visible through the JS, Python and C bindings:
- Char offsets everywhere. Convert token spans in the lexer's
map_with closures, as convert_lexer_error already does. This needs a byte→char table built once per source; a per-token chars().count() would be quadratic.
- Byte offsets everywhere. Switch ariadne to byte indexing (
Config::with_index_type(IndexType::Byte)), drop the conversion in convert_lexer_error, and check the other consumers of spans, including the bindings and debug annotate.
Separately, the assert! in error_message.rs turns a bad span into a crash. Falling back to an error with no location would keep a span bug from taking the process down.
Found during the nightly code-quality survey.
A query with non-ASCII text before an error makes
prqlcpanic, or put the error label in the wrong place:Reproduced on 93ab9fc. With fewer multi-byte characters it doesn't panic, but the label moves: after
# caféon the first line, the "Unknown name" label for a later column sits one column to the right of the name.prqlc debug annotateplaces frames with the same spans, so it can attach them to the wrong lines.Cause. Tokens and error spans use different units.
SimpleSpan(lexer/mod.rs:130-148). Over&strinput, those are byte offsets.Source, and the lexer's own error path,convert_lexer_error, which converts byte offsets to char offsets on purpose.assert!inerror_message.rspanics.The chumsky 0.10 migration (#5223) appears to be where token spans became byte offsets.
Why this needs a decision. One of two conventions has to be picked, and spans are visible through the JS, Python and C bindings:
map_withclosures, asconvert_lexer_erroralready does. This needs a byte→char table built once per source; a per-tokenchars().count()would be quadratic.Config::with_index_type(IndexType::Byte)), drop the conversion inconvert_lexer_error, and check the other consumers of spans, including the bindings anddebug annotate.Separately, the
assert!inerror_message.rsturns a bad span into a crash. Falling back to an error with no location would keep a span bug from taking the process down.Found during the nightly code-quality survey.