Skip to content

Parser rework#9

Merged
sean-parent merged 58 commits into
mainfrom
sean-parent/parser-actions
Jun 18, 2026
Merged

Parser rework#9
sean-parent merged 58 commits into
mainfrom
sean-parent/parser-actions

Conversation

@sean-parent

@sean-parent sean-parent commented Feb 20, 2026

Copy link
Copy Markdown
Member

This pull request makes several improvements to Rust development workflows and code quality standards for the project. The most significant changes include the removal of the cel-parser crate, enhancements to documentation and allocation guidelines, expanded VSCode build/test tasks, and updates to dependencies and macro usage to reflect the new crate structure.

Project structure and dependencies:

  • Removed the cel-parser crate from the workspace and dependencies, consolidating parsing functionality into cel-runtime. All references and dependencies on cel-parser have been eliminated from Cargo.toml files and the workspace configuration. ([[1]](https://github.com/stlab/cel-rs/pull/9/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542L9-R15), [[2]](https://github.com/stlab/cel-rs/pull/9/files#diff-b9957dc1b6aedbcdeff4fbce5b4221da59ef2dd8884b80c051f4fe6ee3108f8dL1-L12), [[3]](https://github.com/stlab/cel-rs/pull/9/files#diff-3823bfb8141007195f06dc4ffabcc9e2db3fd93d37624332d46e88bcfd20c48bL14-R14))
  • Updated the cel-rs-macros crate to depend on cel-runtime instead of cel-parser, and refactored macro code to use parsing and error types from cel-runtime. ([[1]](https://github.com/stlab/cel-rs/pull/9/files#diff-3823bfb8141007195f06dc4ffabcc9e2db3fd93d37624332d46e88bcfd20c48bL14-R14), [[2]](https://github.com/stlab/cel-rs/pull/9/files#diff-c187b9b22359c2452046a640b14f4e3748f18cc98c62ce1ed00bfc82ca709c20L33-R36), [[3]](https://github.com/stlab/cel-rs/pull/9/files#diff-c187b9b22359c2452046a640b14f4e3748f18cc98c62ce1ed00bfc82ca709c20L48-L52))

Development tooling:

  • Added comprehensive VSCode tasks for building, testing, documentation generation, linting (Clippy), and running sanitizers for both macOS and Linux/WSL2, improving the developer experience and CI integration. ([.vscode/tasks.jsonR4-R352](https://github.com/stlab/cel-rs/pull/9/files#diff-7d76d7533653c23b753fc7ce638cf64bdb5e419927d276af836d3a03fdf1745aR4-R352))
  • Updated the spelling dictionary in VSCode settings to recognize "Peekable". ([.vscode/settings.jsonL10-R11](https://github.com/stlab/cel-rs/pull/9/files#diff-a5de3e5871ffcc383a2294845bd3df25d3eeff6c29ad46e3a396577c413bf357L10-R11))

Code quality and documentation standards:

  • Introduced a rule to avoid unnecessary heap allocations in Rust code, with best practices for using references, generics, and slices instead of heap-allocated types. ([.cursor/rules/avoid-heap-allocations.mdcR1-R87](https://github.com/stlab/cel-rs/pull/9/files#diff-d13685b400c5d4957f64a3d3644543afc23c9f80bff059aa3542cfe5adc431aeR1-R87))
  • Expanded and enforced Rust documentation comment standards for all code, requiring structured and concise doc comments for public and private items, with detailed guidelines and examples. ([.cursor/rules/doc-comments.mdcL2-R154](https://github.com/stlab/cel-rs/pull/9/files#diff-bc71edfe8cc02c0facfc56fc6371c9bf2d90055cb84af398575c9f3dc2c204ddL2-R154))

Dependency updates:

  • Added several dependencies to cel-runtime (e.g., proc-macro2, quote, owo-colors, syn, phf, once_cell) to support the consolidated parsing and macro infrastructure. ([cel-runtime/Cargo.tomlL10-R15](https://github.com/stlab/cel-rs/pull/9/files#diff-b401c530f9e812ac243c304a31f8c72e20f1a285e1fdbd95fdb804581cdd1bf2L10-R15))

Macro and parsing improvements:

  • Refactored the expression macro in cel-rs-macros to use the new parser interface from cel-runtime, improving error reporting and aligning with the new crate structure. Also updated the print_tokens! macro documentation example. ([[1]](https://github.com/stlab/cel-rs/pull/9/files#diff-c187b9b22359c2452046a640b14f4e3748f18cc98c62ce1ed00bfc82ca709c20L33-R36), [[2]](https://github.com/stlab/cel-rs/pull/9/files#diff-c187b9b22359c2452046a640b14f4e3748f18cc98c62ce1ed00bfc82ca709c20L48-L52), [[3]](https://github.com/stlab/cel-rs/pull/9/files#diff-c187b9b22359c2452046a640b14f4e3748f18cc98c62ce1ed00bfc82ca709c20L61-R74))

Detailed list of changes:

Project structure and dependencies

  • Removed the cel-parser crate from the workspace and all related dependencies, consolidating parsing functionality into cel-runtime. ([[1]](https://github.com/stlab/cel-rs/pull/9/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542L9-R15), [[2]](https://github.com/stlab/cel-rs/pull/9/files#diff-b9957dc1b6aedbcdeff4fbce5b4221da59ef2dd8884b80c051f4fe6ee3108f8dL1-L12))
  • Updated cel-rs-macros to depend on cel-runtime instead of cel-parser, and refactored macro code accordingly. ([[1]](https://github.com/stlab/cel-rs/pull/9/files#diff-3823bfb8141007195f06dc4ffabcc9e2db3fd93d37624332d46e88bcfd20c48bL14-R14), [[2]](https://github.com/stlab/cel-rs/pull/9/files#diff-c187b9b22359c2452046a640b14f4e3748f18cc98c62ce1ed00bfc82ca709c20L33-R36), [[3]](https://github.com/stlab/cel-rs/pull/9/files#diff-c187b9b22359c2452046a640b14f4e3748f18cc98c62ce1ed00bfc82ca709c20L48-L52))

Development tooling

  • Added and organized a comprehensive set of VSCode tasks for building, testing, documentation, linting, and sanitizer runs on both macOS and Linux/WSL2. ([.vscode/tasks.jsonR4-R352](https://github.com/stlab/cel-rs/pull/9/files#diff-7d76d7533653c23b753fc7ce638cf64bdb5e419927d276af836d3a03fdf1745aR4-R352))
  • Updated VSCode spell checker dictionary to include "Peekable". ([.vscode/settings.jsonL10-R11](https://github.com/stlab/cel-rs/pull/9/files#diff-a5de3e5871ffcc383a2294845bd3df25d3eeff6c29ad46e3a396577c413bf357L10-R11))

Code quality and documentation

  • Added a rule to avoid heap allocations in Rust code, promoting the use of references, generics, and slices. ([.cursor/rules/avoid-heap-allocations.mdcR1-R87](https://github.com/stlab/cel-rs/pull/9/files#diff-d13685b400c5d4957f64a3d3644543afc23c9f80bff059aa3542cfe5adc431aeR1-R87))
  • Expanded and enforced standards for Rust doc comments, requiring structured documentation for all code. ([.cursor/rules/doc-comments.mdcL2-R154](https://github.com/stlab/cel-rs/pull/9/files#diff-bc71edfe8cc02c0facfc56fc6371c9bf2d90055cb84af398575c9f3dc2c204ddL2-R154))

Dependencies

  • Added new dependencies to cel-runtime to support parsing, macro expansion, and colorized output. ([cel-runtime/Cargo.tomlL10-R15](https://github.com/stlab/cel-rs/pull/9/files#diff-b401c530f9e812ac243c304a31f8c72e20f1a285e1fdbd95fdb804581cdd1bf2L10-R15))

Macros and parsing

  • Refactored the expression macro to use the new CELParser interface from cel-runtime, improving error handling and reporting. Updated documentation for the print_tokens! macro. ([[1]](https://github.com/stlab/cel-rs/pull/9/files#diff-c187b9b22359c2452046a640b14f4e3748f18cc98c62ce1ed00bfc82ca709c20L33-R36), [[2]](https://github.com/stlab/cel-rs/pull/9/files#diff-c187b9b22359c2452046a640b14f4e3748f18cc98c62ce1ed00bfc82ca709c20L48-L52), [[3]](https://github.com/stlab/cel-rs/pull/9/files#diff-c187b9b22359c2452046a640b14f4e3748f18cc98c62ce1ed00bfc82ca709c20L61-R74))

Note

High Risk
Large behavioral change to the core parser, macro compile-time validation, and a new type-dispatch op table; mistakes affect expression parsing, runtime semantics, and proc-macro diagnostics across the workspace.

Overview
Reworks cel-parser so parsing produces an executable DynSegment via cel-runtime, instead of only validating token trees and emitting compile_error! into an output stream.

Parser pipeline and API: Adds LexLexer (flatten groups, combine compound punct, syn::Lit literals) and rewires CELParser to take OpLookup, expose parse_str / parse_tokens / set_tokens, and return Result<_, ParseError> while building ops on a DynSegment. Grammar gains postfix calls (parameter_list, "()" lookup) and richer literal suffix handling.

Errors: New error.rs with SourceSpan, CELError, ParseError, and format_rustc_style via annotate-snippets (replacing owo-colors/litrs manual formatting).

Operations: New op_table.rs with phf built-ins (checked signed math, div/mod by zero, wide shift matrix), scope stack, and typed “no operation” / undefined identifier messages with expression range spans.

Macros & CI: expression! validates through parse_tokens and spawns span-aware compile_error! (plus optional end-span diagnostic); trybuild UI tests added. CI adds cargo fmt --check and workflow_dispatch. Workspace adds playground feature, missing_docs lint, and expanded editor/AI guidance (CLAUDE.md, Cursor rules, VS Code tasks).

Reviewed by Cursor Bugbot for commit 4064d1d. Bugbot is set up for automated code reviews on this repo. Configure here.

sean-parent and others added 19 commits August 14, 2025 10:35
Introduces a new op_table module with a scope-based operation lookup system for dynamic operator dispatch in cel-parser. The parser now supports custom operation scopes, efficient built-in operator tables using phf, and exposes op_lookup_mut for extensibility. Includes new tests and documentation, and updates dependencies to add phf and once_cell.
Enhanced literal parsing in LexLexer to use syn::parse2 for better span preservation and added the 'parsing' feature to the syn dependency. Expanded documentation for parser structures and methods in lib.rs, including detailed doc comments for primary expressions, probe results, error formatting, and parsing entry points. Improved test assertions and error reporting in parser tests.
Refactored conditional logic in cel-parser to use && let chains for improved readability and conciseness. Expanded .vscode/tasks.json with comprehensive build, test, documentation, clippy, and sanitizer tasks for multiple platforms to streamline development workflows.
Added and reorganized tasks in .vscode/tasks.json for improved Rust workflow. Expanded sanitizer support for Linux/WSL2 and Windows, clarified platform-specific notes, and added a comprehensive test task for Windows. This update enhances cross-platform development and testing capabilities.
Updated .vscode/tasks.json to streamline task definitions by changing command types to 'cargo' for clippy and adjusting environment variables for better clarity. Removed experimental Windows tasks to focus on macOS and Linux/WSL2 support, enhancing the overall development experience.
Updated the error handling logic in the expression function to use the is_err() method for improved clarity and conciseness. This change enhances the readability of the parser code.
The lexer (`LexLexer`) no longer returns `Result` and is now infallible, as all input is pre-validated by `proc_macro2`. The parser (`CELParser`) is updated to remove error handling for lexer errors, simplifying token peeking and advancement. Documentation is updated to clarify error handling in both lexer and parser. Macro now only reports errors for non-matching expressions, not parse errors.
Move the CEL parser implementation from the separate cel-parser crate into cel-runtime::parser, remove the cel-parser crate and its example docs, and update workspace/Cargo manifests. CELParser API changed: constructor now takes an OpLookup, tokens are set via set_tokens/parse_tokens/parse_str, postfix/call parsing added, and error/reporting helpers adjusted. Updated lex_lexer to use a concrete TokenStreamIter type and relocated files under cel-runtime/src/parser; cel-rs-macros now depends on cel-runtime and uses CELParser/OpLookup accordingly. Also export parser types and implement FromStr for DynSegment to allow parsing from strings; tests updated to the new API.
Add a .cursor rule encouraging no-heap allocations and implement several refactors to reduce temporaries and allocations. Introduce PunctOp (no-heap punctuation token) and replace String-based punct tokens. Extend StackInfo with Cow<'static, str> type_name, padding flag, dropper, and AssociatedType to record nested types; store argument_names in DynSegment. Replace peek_types_vec with peek_stack_infos(&[StackInfo]) returning a slice to avoid allocating Vec<TypeId>. Change ScopeFn/OpLookup APIs to accept &mut DynSegment and a num_operands so scopes can inspect the stack without extra allocations; update builtin lookup and parser call sites and tests accordingly. Improve error messages to include human-readable type names.
Add a dedicated CELError type (with message and proc_macro2::Span) and a rustc-style formatter, and refactor the parser to return CELError instead of anyhow::Error. Removed the parser's compile_error token-output plumbing and helper extraction/formatting methods; replaced report_error with error_at that constructs CELError at the current span. Updated CEL macro to emit compile_error! using CELError spans and literals. Re-export CELError from runtime, adjusted FromStr and parser APIs/Result type, and updated tests to work with the new error type and formatting.
Introduce a Send+Sync SourceSpan and switch CELError to store it (with helper from_proc_macro2) so parse errors can be used across threads. Adjust proc-macro handling to emit compile_error! at call_site and use SourceSpan defaults. Add fallible operation support: DynSegment.op1r/op2r with stack-unwinding droppers and RawSegment::raw2 helper to run fallible binary ops. Update op_table to use op1r/op2r for checked integer arithmetic and shifts (returning errors on overflow/invalid shifts) and add tests for overflow and unwind behavior.
Drop manual unsafe impl Send/Sync for SourceSpan and CELError in parser/error.rs. Reformat cel-runtime/src/parser/op_table.rs: reorder anyhow import, reflow many long closure expressions across multiple lines for readability, compact some signature arrays onto single lines, and tidy minor test formatting. These are mostly non-functional formatting/import changes; only behavioral change is removal of the explicit unsafe Send/Sync impls.
Comment thread .vscode/tasks.json
Comment thread cel-runtime/src/parser/lex_lexer.rs Outdated
Comment thread cel-parser/src/lib.rs
Comment thread src/lib.rs
Comment thread cel-parser/src/op_table.rs
Comment thread cel-runtime/src/parser/mod.rs Outdated
Comment thread cel-runtime/src/dyn_segment.rs
Replace standalone matches! calls with assert!(matches!(...)) in cel-runtime/src/parser/lex_lexer.rs tests. The previous matches! usages were no-ops and did not fail the test on mismatch; wrapping them with assert! ensures the tests actually verify token shapes and will fail when tokens differ from expectations.
Comment thread cel-runtime/src/parser/op_table.rs Outdated
Comment thread cel-runtime/src/parser/mod.rs
Comment thread cel-runtime/src/parser/mod.rs
Use integer checked_div/checked_rem wrapped with op2r for all integer division and modulo signatures so division-by-zero returns an error instead of panicking; float behavior remains unchanged. Add explanatory comments and unit tests (division_by_zero and modulo_by_zero) that assert the error contains "division by zero". This prevents runtime panics and surfaces a clear error to callers.
Added a workflow_dispatch trigger to the CI configuration for manual workflow execution. Included a new job to check Rust code formatting using `cargo fmt --check`. Additionally, refactored several function signatures in the `cel-runtime` module for improved readability by removing unnecessary line breaks and ensuring consistent formatting across the codebase.
@sean-parent sean-parent requested a review from Copilot May 27, 2026 22:06
Comment thread cel-runtime/src/parser/mod.rs Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR consolidates CEL parsing into cel-runtime by removing the standalone cel-parser crate, updating macro validation to use the new runtime parser, and expanding developer tooling/docs to match the new workspace structure.

Changes:

  • Remove cel-parser from the workspace and route parsing APIs through cel-runtime (including new parser modules and DynSegment: FromStr).
  • Refactor cel-rs-macros to validate expressions via cel-runtime parser/error types.
  • Add/expand developer workflow assets (VSCode tasks, CI fmt check, and Cursor/CLAUDE guidance).

Reviewed changes

Copilot reviewed 22 out of 23 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
src/lib.rs Updates crate-level docs/re-exports to reflect parser living in cel-runtime.
notes.txt Adds a cargo clippy --fix --workspace workflow note.
CLAUDE.md Documents workspace architecture, parser pipeline, and workflow commands.
cel-runtime/src/segment.rs Aligns type-id tracking with updated StackInfo shape.
cel-runtime/src/raw_segment.rs Adds fallible stack-aware binary op support (raw2).
cel-runtime/src/parser/op_table.rs Introduces operator lookup/dispatch table with built-in ops and custom scopes.
cel-runtime/src/parser/mod.rs Adds the recursive-descent parser producing executable segments.
cel-runtime/src/parser/lex_lexer.rs Adds lexer layer that flattens/normalizes token streams for the parser.
cel-runtime/src/parser/error.rs Adds CELError + rustc-style diagnostic formatting.
cel-runtime/src/lib.rs Exposes parser APIs and implements DynSegment: FromStr.
cel-runtime/src/dyn_segment.rs Enhances stack metadata, adds fallible ops (op1r/op2r), improves error reporting.
cel-runtime/Cargo.toml Adds dependencies needed for parser/diagnostics and removes old macro dependency coupling.
cel-rs-macros/src/lib.rs Switches expression! validation to the runtime parser and updates docs.
cel-rs-macros/Cargo.toml Depends on cel-runtime instead of removed cel-parser.
cel-parser/src/lib.rs Removed (parser implementation moved into cel-runtime).
cel-parser/Cargo.toml Removed (crate removed from workspace).
Cargo.toml Removes cel-parser from workspace members/dependencies.
Cargo.lock Updates lockfile for removed/added crates.
.vscode/tasks.json Adds comprehensive build/test/lint/sanitizer tasks (needs update for removed crate).
.vscode/settings.json Updates spellchecker dictionary.
.github/workflows/ci.yml Adds workflow_dispatch and cargo fmt --check.
.cursor/rules/doc-comments.mdc Expands/enforces Rust doc comment conventions.
.cursor/rules/avoid-heap-allocations.mdc Adds allocation-avoidance guidelines.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cel-runtime/src/parser/mod.rs Outdated
Comment thread cel-parser/src/error.rs
Comment thread cel-runtime/src/parser/error.rs Outdated
Comment thread cel-rs-macros/src/lib.rs
Comment thread .vscode/tasks.json
Updated the push_literal function to return a Result type, allowing for better error handling when parsing integer and float literals with invalid suffixes. Added unit tests to verify correct parsing of valid suffixes and proper error reporting for invalid ones. This change improves the robustness of the parser by ensuring that suffix-related errors are caught and reported accurately.
sean-parent and others added 19 commits June 3, 2026 18:36
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rror

- Added new dependencies: `annotate-snippets` and `anstyle`.
- Removed unused dependencies: `hermit-abi`, `is-terminal`, `is_ci`, `owo-colors`, `supports-color`.
- Refactored `line_col_to_byte_offset` to `span_to_byte_range` for improved clarity and functionality in error reporting.
- Enhanced test coverage for function calls in the CEL parser, including handling of empty argument lists and error cases.
Introduces ParseError (not Send+Sync) for use as the parser return type,
carrying the original proc_macro2::Span for precise IDE diagnostics.
From<ParseError> for CELError extracts the SourceSpan for async/runtime use.
CELError::with_proc_macro_span is retained for Task 2 migration.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Updated CELParser to return ParseError instead of CELError, allowing for more precise diagnostics with proc_macro2::Span.
- Enhanced CELError to include a message method for better error reporting.
- Modified the expression macro to include tests for error handling scenarios.
- Adjusted VSCode task configuration to improve testing command options.

These changes improve error handling and diagnostics in the CEL parser, enhancing the developer experience.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces unwrap_or(self.last_span) with a join_spans() helper that
falls back to the start span when proc_macro2::Span::join returns None.
On stable Rust inside a real proc-macro, proc_macro2 gates Span::join
behind the nightly-only proc_macro_span feature and returns None; the
helper gracefully degrades to the start span rather than panicking or
silently swallowing the failure with the wrong fallback.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…in()

proc_macro::Span::join() is not yet stable. Revised design stores
start and end spans separately in ParseError::new_range(), merges
them at the SourceSpan layer for runtime, and emits two compile_error!()
diagnostics for compile-time.
…asks

Replaces the prior plan that used Span::join() with an approach that
stores start and end spans separately, requiring no unstable APIs.
…rrors

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… to improve error handling in expression macro. The changes include new permissions for various scripts and a refined approach to compile_error!() handling in the expression! macro, ensuring both start and end errors are reported correctly.
…dependencies. Add trybuild as a dev-dependency in cel-rs-macros and enhance error handling in CELParser with new tests for identifier scope errors.
Comment thread cel-parser/src/lib.rs
…rror message and adding check for undefined identifiers with appropriate error reporting.
…e redundant prefixes, and wrap identifiers in backticks across various modules.
…and spacing for improved readability in parse_str and test functions.
Comment thread cel-parser/src/op_table.rs

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit ffa26c9. Configure here.

Comment thread cel-parser/src/error.rs
@sean-parent sean-parent merged commit 975d223 into main Jun 18, 2026
2 checks passed
@sean-parent sean-parent deleted the sean-parent/parser-actions branch June 20, 2026 00:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants