Skip to content

Harden email extraction boundary handling and UTF-16 regressions #620

Description

@simsong

Follow up on the unresolved Copilot review findings in PR #593.

Investigate and add focused regression coverage for all four concerns together:

  1. Preserve extraction of ASCII emails before sentence punctuation without accepting a partial email/TLD match.
  2. Treat a preceding non-ASCII UTF-16LE code unit as a left boundary even when its low byte is ASCII alphanumeric.
  3. Define and test the UTF-16 trailing-context behavior for unaligned following bytes and sentence punctuation; compare it with the corresponding UTF-16 URL rule before changing the lexer rule.
  4. Strengthen accepted UTF-16 length-limit tests to assert the exact emitted email.txt feature and offset, not merely non-empty output.

Document the boundary cases chosen and verify that the change does not increase partial-email false positives.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions