Skip to content

Update chumsky and implement the remainder of KDLv2 - #33

Closed
tmke8 wants to merge 19 commits into
TheLostLambda:mainfrom
tmke8:chumsky-update-kdl2
Closed

tmke8 wants to merge 19 commits into
TheLostLambda:mainfrom
tmke8:chumsky-update-kdl2

Conversation

@tmke8

@tmke8 tmke8 commented Jan 23, 2026

Copy link
Copy Markdown
Collaborator

This is a very big PR that I don’t expect to me merged in one piece. I’m opening it here mostly to discuss if and how to merge it.

There are 5 parts to this PR:

Rebased commits from the kdl-v2 branch

The first 14 commits are the commits from kdl-v2, rebased onto main in order to get the recent CI fixes. As far as I remember, there were no conflicts.

Changes to make the tests pass

Commit 5b1137b is there just to make CI pass. It includes a regression to an error label, but that regression is fixed in the commit that updates chumsky.

Removal of the genericness around the span type

This is commit 67f5ed8. I expect this to be the most controversial change. The motivation for this was to make updating chumsky easier.

Before this change, the parser was generic over the span type. The code base contained two span types that could be used (and the library user could write their own): Span and LineSpan. Span was basically just a Range. LineSpan, on the other hand, was also storing the start and end position in terms of line and column number. The doc comment for src/span.rs claimed this:

//! Knus can generate two types of span during parsing
//!
//! 1. [`Span`] which only tracks byte offset from the start of the source code
//! 2. [`LineSpan`] which also track line numbers
//!
//! This distinction is important during parsing stage as [`Span`] is normally
//! faster. And [`LineSpan`] is still faster than find out line/column number
//! for each span separately, and is also more convenient if you need this
//! information.

I’m guessing this was written at a time where error reporting required specifying line and column numbers. But this isn’t true with miette. miette just wants the byte offsets and then does the rest on its own. In fact, after removing LineSpan from the code base, I didn’t have to change any test, because it wasn’t actually needed for anything.

So, I removed LineSpan and made the code not be generic anymore over the span type. The #[knus(span)] attribute still works though! And it’s much easier to use now, actually. The user just has to implement the trait DecodeSpan for their custom span type. And #[knus(span_type=Span)] isn’t required anymore:

use knus::span::Span;
use knus::traits::DecodeSpan;
use knus::decode::Context;

struct MySpan {
    start: usize,
    end: usize,
}

impl DecodeSpan for MySpan {
    fn decode_span(span: &Span, _ctx: &mut Context) -> Self {
        MySpan {
            start: span.0,
            end: span.1,
        }
    }
}

#[derive(knus::Decode)]
struct MyStruct {
    #[knus(span)]
    span: MySpan,
}

If anyone wants to know the line and column number, they can get it with this new method I added to Span:

    /// Converts the span's byte offsets to zero-based line/column pairs
    pub fn to_line_column(&self, text: &str) -> ((usize, usize), (usize, usize))

(This needs the original input as an argument, in order to know where the newlines are.)

This is a breaking change, but switching the parser to KDLv2 is a breaking change anyway, so I hope it’s fine. If you really don’t like this change, I can think about how to undo it, but it won’t be easy because there are so many changes on top of this that will become more complicated if the parser is still generic over the span.

Migrating from chumsky 0.9.* to 0.12

This is commit 2defcac.

This was not easy, because there were a lot of subtle breaking changes between chumsky 0.9.* and 0.10.0. There was a migration guide but that still left a lot of trial and error to get it working.

If you look at Cargo.toml, you’ll see that I’m currently installing chumsky from git. That’s because I sent two PR to chumsky that were necessary to get it working, which have been merged, but have not been released yet:

Implement the missing features from KDLv2

In commit c4fd14f I added support for multiline strings, which was still missing from the original kdl-v2 branch. The implementation for this ended up being quite verbose. This is the change I’d most like feedback on. Maybe I’m missing a way this could be simpler.


And that’s it! Are you interested in these changes @TheLostLambda ? How should we go about merging them?

@tmke8
tmke8 marked this pull request as draft January 23, 2026 13:33
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.89363% with 173 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.38%. Comparing base (581e719) to head (c4fd14f).

Files with missing lines Patch % Lines
src/grammar.rs 93.00% 71 Missing and 12 partials ⚠️
src/containers.rs 9.37% 29 Missing ⚠️
src/errors.rs 57.89% 19 Missing and 5 partials ⚠️
src/convert.rs 52.63% 9 Missing ⚠️
src/decode.rs 35.71% 9 Missing ⚠️
src/span.rs 91.17% 6 Missing ⚠️
src/convert_ast.rs 28.57% 5 Missing ⚠️
src/ast.rs 20.00% 2 Missing and 2 partials ⚠️
derive/src/definition.rs 81.81% 0 Missing and 2 partials ⚠️
derive/src/node.rs 90.00% 0 Missing and 1 partial ⚠️
... and 1 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #33      +/-   ##
==========================================
+ Coverage   74.16%   79.38%   +5.22%     
==========================================
  Files          15       15              
  Lines        3847     4468     +621     
  Branches     3847     4468     +621     
==========================================
+ Hits         2853     3547     +694     
+ Misses        857      770      -87     
- Partials      137      151      +14     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@TheLostLambda

Copy link
Copy Markdown
Owner

Hi @tmke8 !

Overall this looks like an outstanding set of changes! My time is pretty tight at the moment (writing up a thesis!), but I'm happy to start chipping away at this!

I can review and merge things however you think is easiest, but perhaps you could split things into a PR for 1: rebase and test fixing, 2: remove span generics, 3: chumsky update, 4: last of KDL multiline?

You can arrange those as stacked PRs if you'd like — Jujutsu is a great tool for that sort of thing if you've not checked it out already (it's completely replaced git for me).

Thank you for this monumental effort! Excited to get it all in!

@kleinesfilmroellchen kleinesfilmroellchen left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A cursory review of basic code style and Rust things to speed up the process here.

Comment thread src/grammar.rs Resolved
Comment thread src/grammar.rs Resolved
Comment thread derive/derive_decode.md Resolved
Comment thread src/span.rs Resolved
Comment thread src/span.rs Resolved
Comment thread src/span.rs Resolved
Comment thread src/span.rs Resolved
Comment thread src/errors.rs Resolved
Comment thread src/grammar.rs
Comment on lines -147 to +132
.then(take_until(newline().or(end())))
.then(
any()
.and_is(newline().not())
.and_is(end().not())
.repeated()
.then(newline().or(end())),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really silly, I assume chumsky upstream wasn’t quite in their right mind when they removed the take_until combinator…

Comment thread src/grammar.rs

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: didn’t review the multiline string additions here, too much for right now

@tmke8

tmke8 commented Feb 4, 2026

Copy link
Copy Markdown
Collaborator Author

You can arrange those as stacked PRs if you'd like

I tried to do that, but I think this is only possible if the feature branches are all in this repository.

So, I suppose we'll try to do this one-by-one. I opened the first PR here: #37

tmke8 added 4 commits February 4, 2026 21:58
For error reporting, this span type can still be transformed to a
different span type.
Add multi-line string support for both quoted (""") and raw (#""") strings
per the KDL v2 specification:
- Opening delimiter must be followed by newline
- Closing delimiter must be on its own line
- Auto-dedentation based on closing line's whitespace prefix
- Newline normalization (CRLF → LF)
- Escape sequences processed after dedentation
@tmke8
tmke8 force-pushed the chumsky-update-kdl2 branch from c4fd14f to 2ef7a67 Compare February 4, 2026 21:00
@tmke8

tmke8 commented Feb 4, 2026

Copy link
Copy Markdown
Collaborator Author

@kleinesfilmroellchen I addressed your first two review comments over at #37 (I think the other comments are for later commits, but correct me if I'm wrong)

@tmke8 tmke8 mentioned this pull request Feb 5, 2026
@TheLostLambda

Copy link
Copy Markdown
Owner

Thanks @kleinesfilmroellchen for helping with the reviewing!

For either @tmke8 or @kleinesfilmroellchen , are you interested in bumping miette to the latest version whilst we're updating chumsky?

As far as I know, it should be much less of an effort / fewer breaking changes and would address #35!

@kleinesfilmroellchen

Copy link
Copy Markdown

Unfortunately I am busy at the moment. But miette should be quite easy as their breaking changes are never that large from my experience. (Unlike what chumsky apparently did at 0.10)

@tmke8

tmke8 commented Feb 5, 2026

Copy link
Copy Markdown
Collaborator Author

Yeah, I can give it a try.

Would you also be open to switching the error tests from the current JSON matching approach to an approach based on snapshots (with e.g. insta) where we just verify the printed error? Currently it's a bit annoying to update the error tests and it's hard to see what the errors will actually look like for the user.

@TheLostLambda

Copy link
Copy Markdown
Owner

@tmke8 Absolutely!

I use insta in all of my projects, so that sounds great!

Thanks a ton!!

@tmke8

tmke8 commented Mar 28, 2026

Copy link
Copy Markdown
Collaborator Author

@tmke8 tmke8 closed this Mar 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants