From becd9bcb51d717b784742d8f6f63d508f405a757 Mon Sep 17 00:00:00 2001 From: Mark Scherer Date: Sat, 18 Jul 2026 14:24:20 +0200 Subject: [PATCH 1/2] Stamp nested blocks and list items with source lines Source-line tracking (sourceLines: true) previously stamped only the document's top-level blocks. Nested blocks parsed through rebuilt, re-indexed line arrays (blockquotes, divs, list items, footnotes, definition lists) could not be stamped because the local loop index no longer matched document coordinates. Thread an optional per-line map (local index to 0-based document line, -1 for synthetic lines) through parseBlocks() recursion and build it at every extraction site. Stamp nested blocks plus li/dt/dd nodes; keep raw HTML blocks and comments unstamped and never overwrite an author-set data-source-line attribute. Content parsed via the public parseBlockContent() helper carries no document positions, so it is mapped as unknown instead of mis-stamping local indices. Default output is unchanged; the map bookkeeping in the default case is below measurement noise on container-heavy documents. --- README.md | 2 +- docs/reference/api.md | 22 ++- src/DjotConverter.php | 2 +- src/Parser/BlockParser.php | 189 +++++++++++++++++----- tests/TestCase/SourceLineTrackingTest.php | 142 ++++++++++++++++ 5 files changed, 308 insertions(+), 49 deletions(-) diff --git a/README.md b/README.md index 42cec6a..75a2a77 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,7 @@ $html = $converter->convert('Hello *world*!'); - **Multiple renderers**: HTML, plain text, Markdown, ANSI terminal output - **Extensions**: Built-in extensions for external links, TOC, heading permalinks, @mentions, autolinks, default attributes, and citations - **Extensible**: Custom inline/block patterns, render events -- **Editor integration**: Opt-in `data-source-line` stamping on top-level blocks for live-preview scroll-sync (`new DjotConverter(sourceLines: true)`) — see [Source-line tracking](https://php-collective.github.io/djot-php/reference/api#source-line-tracking) +- **Editor integration**: Opt-in `data-source-line` stamping on blocks, nested blocks, list items, and definition list entries for live-preview scroll-sync (`new DjotConverter(sourceLines: true)`) — see [Source-line tracking](https://php-collective.github.io/djot-php/reference/api#source-line-tracking) - **File support**: Parse and convert files directly - **CLI tools**: `bin/djot` (one-shot convert) and `bin/djot-watch` (live-reload preview server) — see [CLI reference](https://php-collective.github.io/djot-php/reference/cli) diff --git a/docs/reference/api.md b/docs/reference/api.md index e087aa7..8daadfb 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -45,7 +45,7 @@ public function __construct( - `$nestedBlocksInLists`: **Deprecated.** When `true`, indentation alone introduces nested blocks of **any** type inside list items without a blank line (broad, eager - no lone-marker lookahead), while top-level paragraph interruption stays spec-compliant (see [Nested Blocks in Lists Mode](/guide/parser-options#nested-blocks-in-lists-mode)). Prefer `$blocksInterruptParagraphs` + `$nestedListsWithoutBlankLine`. No longer implied by `$significantNewlines`. - `$blocksInterruptParagraphs`: When `true`, top-level block elements (lists, blockquotes, headings, tables, thematic breaks, and code/div/comment fences) can interrupt a paragraph without a preceding blank line. It **also** interrupts a list item's lead paragraph, so an indented non-list block nests inside the item without a blank line, using the **same** lone-marker lookahead as the top level: unambiguous openers (`#`, fenced code, `:::`, `---`) and real multi-line blocks nest, while a single ambiguous marker line (`>`, `|`) stays literal. It does not nest sublists (see [Block Interrupts Paragraphs Mode](/guide/parser-options#block-interrupts-paragraphs-mode)). Implied by `$significantNewlines`. - `$nestedListsWithoutBlankLine`: When `true`, a sublist nests inside a list item without a blank line. Only sublists nest; non-list blocks under the item stay literal and top-level paragraph interruption is unaffected (see [Nested Lists Without Blank Line Mode](/guide/parser-options#nested-lists-without-blank-line-mode)). Implied by `$significantNewlines`. -- `$sourceLines`: When `true`, each top-level block element is stamped with a `data-source-line` attribute holding the **1-based** source line where the block started. Off by default, so normal output is unchanged. Intended for editor live-preview scroll-sync (map a rendered block back to the source textarea). Ignored when a pre-configured `$parser` is supplied (pass `new BlockParser(trackSourceLines: true)` instead). See [Source-line tracking](#source-line-tracking). +- `$sourceLines`: When `true`, block elements, nested blocks, list items, and definition terms/descriptions are stamped with a `data-source-line` attribute holding the **1-based** source line where the block started. Off by default, so normal output is unchanged. Intended for editor live-preview scroll-sync (map a rendered block back to the source textarea). Ignored when a pre-configured `$parser` is supplied (pass `new BlockParser(trackSourceLines: true)` instead). See [Source-line tracking](#source-line-tracking). ### Factory Methods @@ -411,12 +411,28 @@ echo $converter->convert("# Heading\n\nA paragraph.\n"); ``` -- Only **top-level** blocks are stamped (a paragraph nested inside a blockquote - is not). The value is the **1-based** source line where the block starts. +- Top-level and nested blocks are stamped. List items (`
  • `) and definition + list terms/descriptions (`
    ` / `
    `) are stamped too. The value is the + **1-based** source line where the block starts in the original document. - The attribute renders **after** any author attributes, e.g. `

    `. - Raw HTML blocks and comments are not stamped (no reliable element tag). +For example: + +```php +echo $converter->convert("- first\n\n second\n"); +``` + +```html +

      +
    • +

      first

      +

      second

      +
      • +
    +``` + The typical use is editor live-preview scroll synchronization: read the line of the block at the top of the source pane, find the element whose `data-source-line` matches in the rendered pane, and scroll it into view. The diff --git a/src/DjotConverter.php b/src/DjotConverter.php index 18dc41b..35fcccf 100644 --- a/src/DjotConverter.php +++ b/src/DjotConverter.php @@ -131,7 +131,7 @@ public static function ansi(?BlockParser $parser = null): self * @param bool $nestedBlocksInLists Allow nested blocks in list items without blank lines (deprecated; prefer blocksInterruptParagraphs + nestedListsWithoutBlankLine) * @param bool $blocksInterruptParagraphs Allow top-level block elements to interrupt paragraphs without a blank line * @param bool $nestedListsWithoutBlankLine Allow sublists to nest in list items without a blank line - * @param bool $sourceLines Stamp top-level block elements with a `data-source-line` attribute (1-based source line). Opt-in, for editor scroll-sync; ignored when a pre-configured $parser is supplied. + * @param bool $sourceLines Stamp block elements, nested blocks, list items, and definition list entries with a `data-source-line` attribute (1-based source line). Opt-in, for editor scroll-sync; ignored when a pre-configured $parser is supplied. */ public function __construct( bool $xhtml = false, diff --git a/src/Parser/BlockParser.php b/src/Parser/BlockParser.php index 110958d..4d805ed 100644 --- a/src/Parser/BlockParser.php +++ b/src/Parser/BlockParser.php @@ -221,10 +221,10 @@ class BlockParser protected ?Closure $headingIdTransformer = null; /** - * When true, top-level block nodes are stamped with a `data-source-line` - * attribute holding the 1-based source line where the block started. - * Opt-in (default off): used by editor live-preview to sync scroll to the - * source textarea. Off by default so normal rendering output is unchanged. + * When true, block nodes are stamped with a `data-source-line` attribute + * holding the 1-based source line where the block started. Opt-in (default + * off): used by editor live-preview to sync scroll to the source textarea. + * Off by default so normal rendering output is unchanged. * * @var bool */ @@ -421,12 +421,17 @@ public function getBlockPatterns(): array /** * Parse block content (for use in custom block callbacks) * + * The extracted lines carry no positions in the original document, so + * source-line tracking skips them (a null map would mis-stamp local + * indices as document lines); the custom block node itself is still + * stamped by the caller. + * * @param \Djot\Node\Node $parent * @param array $lines */ public function parseBlockContent(Node $parent, array $lines): void { - $this->parseBlocks($parent, $lines, 0); + $this->parseBlocks($parent, $lines, 0, array_fill(0, count($lines), -1)); } /** @@ -645,8 +650,10 @@ protected function extractFootnotes(array $lines): void // Collect continuation lines (indented or blank) $contentLines = []; + $contentLineMap = []; if (trim($content) !== '') { $contentLines[] = $content; + $contentLineMap[] = $i; } $j = $i + 1; $hasContent = false; @@ -655,6 +662,7 @@ protected function extractFootnotes(array $lines): void if (IndentationHelper::isBlankLine($nextLine)) { // Add blank line to preserve structure $contentLines[] = ''; + $contentLineMap[] = $j; $j++; continue; @@ -662,11 +670,13 @@ protected function extractFootnotes(array $lines): void // Check if line has at least base indentation (2 spaces or 1 tab) if (preg_match('/^(?:[ ]{' . $baseIndent . '}|\t)(.*)$/', $nextLine, $contMatch)) { $contentLines[] = $contMatch[1]; + $contentLineMap[] = $j; $hasContent = true; $j++; } elseif (!$hasContent && preg_match('/^\s+(.+)$/', $nextLine, $contMatch)) { // Allow flexible indentation for first content line $contentLines[] = $contMatch[1]; + $contentLineMap[] = $j; $hasContent = true; $j++; } else { @@ -678,12 +688,13 @@ protected function extractFootnotes(array $lines): void $lineCount = count($contentLines); while ($lineCount > 0 && $contentLines[$lineCount - 1] === '') { array_pop($contentLines); + array_pop($contentLineMap); $lineCount--; } $footnote = new Footnote($label); if ($contentLines) { - $this->parseBlocks($footnote, $contentLines, 0); + $this->parseBlocks($footnote, $contentLines, 0, $contentLineMap); } $this->footnotes[$label] = $footnote; } @@ -935,8 +946,9 @@ protected function retargetHeadingLinks(Node $node, array $newUrlByLabel, Headin * @param \Djot\Node\Node $parent * @param array $lines * @param int $indent + * @param array|null $lineMap */ - protected function parseBlocks(Node $parent, array $lines, int $indent): void + protected function parseBlocks(Node $parent, array $lines, int $indent, ?array $lineMap = null): void { if ($this->nestingDepth >= self::MAX_NESTING_DEPTH) { $text = implode("\n", $lines); @@ -951,7 +963,7 @@ protected function parseBlocks(Node $parent, array $lines, int $indent): void $this->nestingDepth++; try { - $this->parseBlocksImpl($parent, $lines, $indent); + $this->parseBlocksImpl($parent, $lines, $indent, $lineMap); } finally { $this->nestingDepth--; } @@ -961,8 +973,9 @@ protected function parseBlocks(Node $parent, array $lines, int $indent): void * @param \Djot\Node\Node $parent * @param array $lines * @param int $indent + * @param array|null $lineMap */ - private function parseBlocksImpl(Node $parent, array $lines, int $indent): void + private function parseBlocksImpl(Node $parent, array $lines, int $indent, ?array $lineMap): void { $i = 0; $count = count($lines); @@ -986,17 +999,16 @@ private function parseBlocksImpl(Node $parent, array $lines, int $indent): void } // Source-line tracking (opt-in): remember where this block starts and - // how many children the parent had, so newly appended top-level blocks - // can be stamped with `data-source-line` after the dispatch below. - // Only the document's direct children are stamped (nested blocks are - // skipped via the -1 sentinel). + // how many children the parent had, so newly appended blocks can be + // stamped with `data-source-line` after the dispatch below. $blockStart = $i; - $childrenBefore = ($this->trackSourceLines && $parent instanceof Document) ? count($parent->getChildren()) : -1; + $sourceLine = $this->sourceLineFor($lineMap, $blockStart); + $childrenBefore = ($this->trackSourceLines && $sourceLine >= 0) ? count($parent->getChildren()) : -1; // Try custom block patterns first (before built-in syntax) $customConsumed = $this->tryCustomBlockPatterns($parent, $lines, $i); if ($customConsumed !== null) { - $this->stampSourceLine($parent, $childrenBefore, $blockStart); + $this->stampSourceLine($parent, $childrenBefore, $sourceLine); $i += $customConsumed; continue; @@ -1015,7 +1027,7 @@ private function parseBlocksImpl(Node $parent, array $lines, int $indent): void && !($marker >= 'A' && preg_match('/^[ \t]*[A-Za-z]+[.)](?:\{[^{}]+\})?([ \t]|$)/', $line) === 1) ) { $consumedFast = $this->tryParseParagraph($parent, $lines, $i); - $this->stampSourceLine($parent, $childrenBefore, $blockStart); + $this->stampSourceLine($parent, $childrenBefore, $sourceLine); $i += $consumedFast; continue; @@ -1029,11 +1041,11 @@ private function parseBlocksImpl(Node $parent, array $lines, int $indent): void ?? $this->tryParseComment($parent, $lines, $i) ?? $this->tryParseRawBlock($parent, $lines, $i) ?? $this->tryParseCodeBlock($parent, $lines, $i) - ?? $this->tryParseDiv($parent, $lines, $i) + ?? $this->tryParseDiv($parent, $lines, $i, $lineMap) ?? $this->tryParseHeading($parent, $lines, $i) ?? $this->tryParseThematicBreak($parent, $line, $i) - ?? $this->tryParseBlockQuote($parent, $lines, $i) - ?? $this->tryParseList($parent, $lines, $i) + ?? $this->tryParseBlockQuote($parent, $lines, $i, $lineMap) + ?? $this->tryParseList($parent, $lines, $i, $lineMap) ?? $this->tryParseLineBlock($parent, $lines, $i) ?? $this->tryParseTable($parent, $lines, $i) ?? $this->tryParseFootnoteDefinition($lines, $i) @@ -1042,7 +1054,7 @@ private function parseBlocksImpl(Node $parent, array $lines, int $indent): void ?? $this->tryParseCaption($parent, $lines, $i) ?? $this->tryParseParagraph($parent, $lines, $i); - $this->stampSourceLine($parent, $childrenBefore, $blockStart); + $this->stampSourceLine($parent, $childrenBefore, $sourceLine); $i += $consumed; } } @@ -1054,11 +1066,11 @@ private function parseBlocksImpl(Node $parent, array $lines, int $indent): void * * @param \Djot\Node\Node $parent * @param int $childrenBefore Child count before the block was parsed, or -1 when disabled. - * @param int $start 0-indexed source line index; emitted as 1-based (+1). + * @param int $sourceLine 0-indexed source line index; emitted as 1-based (+1). * * @return void */ - private function stampSourceLine(Node $parent, int $childrenBefore, int $start): void + private function stampSourceLine(Node $parent, int $childrenBefore, int $sourceLine): void { if ($childrenBefore < 0) { return; @@ -1067,12 +1079,27 @@ private function stampSourceLine(Node $parent, int $childrenBefore, int $start): $children = $parent->getChildren(); $total = count($children); for ($k = $childrenBefore; $k < $total; $k++) { - if ($children[$k]->getAttribute('data-source-line') === null) { - $children[$k]->setAttribute('data-source-line', (string)($start + 1)); + if ( + !$children[$k] instanceof RawBlock + && !$children[$k] instanceof Comment + && $children[$k]->getAttribute('data-source-line') === null + ) { + $children[$k]->setAttribute('data-source-line', (string)($sourceLine + 1)); } } } + /** + * Resolve a local line index to the original document's 0-based line index. + * + * @param array|null $lineMap + * @param int $index + */ + private function sourceLineFor(?array $lineMap, int $index): int + { + return $lineMap === null ? $index : ($lineMap[$index] ?? -1); + } + /** * Try to match custom block patterns at the current position * @@ -1492,8 +1519,9 @@ protected function tryParseRawBlock(Node $parent, array $lines, int $start): ?in * @param \Djot\Node\Node $parent * @param array $lines * @param int $start + * @param array|null $lineMap */ - protected function tryParseDiv(Node $parent, array $lines, int $start): ?int + protected function tryParseDiv(Node $parent, array $lines, int $start, ?array $lineMap = null): ?int { $line = $lines[$start]; @@ -1523,6 +1551,7 @@ protected function tryParseDiv(Node $parent, array $lines, int $start): ?int $this->pendingAttributeSourceLines = []; $innerLines = []; + $innerLineMap = []; $sourceLines = [...$divAttributeSourceLines, $line]; $i = $start + 1; $count = count($lines); @@ -1543,6 +1572,7 @@ protected function tryParseDiv(Node $parent, array $lines, int $start): ?int $codeBlockFenceLength = $codeFenceInfo['length']; $sourceLines[] = $currentLine; $innerLines[] = $currentLine; + $innerLineMap[] = $this->sourceLineFor($lineMap, $i); $i++; continue; @@ -1555,6 +1585,7 @@ protected function tryParseDiv(Node $parent, array $lines, int $start): ?int } $sourceLines[] = $currentLine; $innerLines[] = $currentLine; + $innerLineMap[] = $this->sourceLineFor($lineMap, $i); $i++; continue; @@ -1571,6 +1602,7 @@ protected function tryParseDiv(Node $parent, array $lines, int $start): ?int $sourceLines[] = $currentLine; $innerLines[] = $currentLine; + $innerLineMap[] = $this->sourceLineFor($lineMap, $i); $i++; } @@ -1583,7 +1615,7 @@ protected function tryParseDiv(Node $parent, array $lines, int $start): ?int // Parse inner content as blocks (track line offset for nested content) $previousOffset = $this->lineOffset; $this->lineOffset = $previousOffset + $start + 1; - $this->parseBlocks($div, $innerLines, 0); + $this->parseBlocks($div, $innerLines, 0, $innerLineMap); $this->lineOffset = $previousOffset; // Apply opener-line attributes (from `::: {#id .class key=value}` syntax) @@ -1709,8 +1741,9 @@ protected function tryParseThematicBreak(Node $parent, string $line, int $start) * @param \Djot\Node\Node $parent * @param array $lines * @param int $start + * @param array|null $lineMap */ - protected function tryParseBlockQuote(Node $parent, array $lines, int $start): ?int + protected function tryParseBlockQuote(Node $parent, array $lines, int $start, ?array $lineMap = null): ?int { $line = $lines[$start]; @@ -1733,6 +1766,7 @@ protected function tryParseBlockQuote(Node $parent, array $lines, int $start): ? $this->pendingAttributeSourceLines = []; $innerLines = []; + $innerLineMap = []; $lazyState = [ 'inFence' => false, 'fenceChar' => '', @@ -1744,9 +1778,11 @@ protected function tryParseBlockQuote(Node $parent, array $lines, int $start): ? if (preg_match('/^>[ \t](.*)$/', $line, $matches)) { $innerLines[] = $matches[1]; + $innerLineMap[] = $this->sourceLineFor($lineMap, $start); $this->trackBlockQuoteLazyState($matches[1], $lazyState); } elseif (preg_match('/^>$/', $line)) { $innerLines[] = ''; + $innerLineMap[] = $this->sourceLineFor($lineMap, $start); $this->trackBlockQuoteLazyState('', $lazyState); } @@ -1763,11 +1799,13 @@ protected function tryParseBlockQuote(Node $parent, array $lines, int $start): ? // Continue with a "> " or ">\t" prefix (space or tab delimiter) if (preg_match('/^>[ \t](.*)$/', $currentLine, $matches)) { $innerLines[] = $matches[1]; + $innerLineMap[] = $this->sourceLineFor($lineMap, $i); $this->trackBlockQuoteLazyState($matches[1], $lazyState); $i++; } elseif (preg_match('/^>$/', $currentLine)) { // Empty block quote line (just >) $innerLines[] = ''; + $innerLineMap[] = $this->sourceLineFor($lineMap, $i); $this->trackBlockQuoteLazyState('', $lazyState); $i++; } elseif ($lazyState['paragraphOpen'] && !$this->startsNewBlock($currentLine)) { @@ -1776,6 +1814,7 @@ protected function tryParseBlockQuote(Node $parent, array $lines, int $start): ? // block that left no open paragraph (a just-opened div, a closed // fence), terminates the quote instead of being swallowed. $innerLines[] = $currentLine; + $innerLineMap[] = $this->sourceLineFor($lineMap, $i); $this->trackBlockQuoteLazyState($currentLine, $lazyState); $i++; } else { @@ -1783,7 +1822,7 @@ protected function tryParseBlockQuote(Node $parent, array $lines, int $start): ? } } - $this->parseBlocks($blockQuote, $innerLines, 0); + $this->parseBlocks($blockQuote, $innerLines, 0, $innerLineMap); // Apply the saved attributes to the blockquote if ($quoteAttributes !== []) { @@ -1885,8 +1924,9 @@ private function trackBlockQuoteLazyState(string $content, array &$state): void * @param \Djot\Node\Node $parent * @param array $lines * @param int $start + * @param array|null $lineMap */ - protected function tryParseList(Node $parent, array $lines, int $start): ?int + protected function tryParseList(Node $parent, array $lines, int $start, ?array $lineMap = null): ?int { $line = $lines[$start]; @@ -1903,7 +1943,7 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int // Definition lists are handled separately in djot if ($listInfo['type'] === ListBlock::TYPE_DEFINITION) { - return $this->tryParseDjotDefinitionList($parent, $lines, $start); + return $this->tryParseDjotDefinitionList($parent, $lines, $start, $lineMap); } // Disambiguate roman vs alphabetical for single-letter markers @@ -1978,6 +2018,7 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int // Collect all indented content at this new level $subLines = []; + $subLineMap = []; $subIndent = $currentIndent; // Track the maximum content indent we've seen (for detecting drop-back to marker level) $maxContentIndent = $currentIndent; @@ -1987,6 +2028,7 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int $subLine = $lines[$i]; if (IndentationHelper::isBlankLine($subLine)) { $subLines[] = ''; + $subLineMap[] = $this->sourceLineFor($lineMap, $i); $sawBlankLine = true; $i++; @@ -2013,6 +2055,7 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int } // Remove subIndent worth of indentation (handling tabs) $subLines[] = IndentationHelper::stripLeadingIndent($subLine, $subIndent); + $subLineMap[] = $this->sourceLineFor($lineMap, $i); $sawBlankLine = false; $i++; } elseif ($lineIndent === $baseIndent) { @@ -2039,6 +2082,7 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int } // Otherwise it's lazy continuation at base level - include in nested content $subLines[] = $trimmedLine; + $subLineMap[] = $this->sourceLineFor($lineMap, $i); $sawBlankLine = false; $i++; } elseif ($lineIndent > $baseIndent) { @@ -2054,11 +2098,12 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int $subLineCount = count($subLines); while ($subLineCount > 0 && $subLines[$subLineCount - 1] === '') { array_pop($subLines); + array_pop($subLineMap); $subLineCount--; } // Parse nested content if ($subLines !== []) { - $this->parseBlocks($lastItem, $subLines, 0); + $this->parseBlocks($lastItem, $subLines, 0, $subLineMap); } // In djot, blank lines within nested content don't make the parent list loose // The list is only loose if there's a blank line directly after item content @@ -2099,6 +2144,7 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int /** @var string|null $taskMarker */ $taskMarker = $itemInfo['taskMarker'] ?? null; $listItem = new ListItem($taskMarker); + $listItemSourceLine = $this->sourceLineFor($lineMap, $i); /** @var string $itemContent */ $itemContent = $itemInfo['content']; @@ -2117,6 +2163,7 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int // Collect item content lines (without blank line = tight continuation) /** @var array $itemLines */ $itemLines = [$itemContent]; + $itemLineMap = [$listItemSourceLine]; $i++; $lastItemHadBlankAfter = false; $hasNonMarkerContinuation = false; @@ -2195,9 +2242,11 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int // beyond the content column is not significant for plain text, // matching the reference implementation. $itemLines[] = $nextTrimmed; + $itemLineMap[] = $this->sourceLineFor($lineMap, $i); } else { // Lazy continuation (not properly indented but not at base level either) $itemLines[] = $nextTrimmed; + $itemLineMap[] = $this->sourceLineFor($lineMap, $i); } $hasNonMarkerContinuation = true; $i++; @@ -2218,11 +2267,13 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int if ($markerLineSublist) { $j = $i; $gathered = []; + $gatheredLineMap = []; $consumedThrough = $i; while ($j < $count) { $subLine = $lines[$j]; if (IndentationHelper::isBlankLine($subLine)) { $gathered[] = ''; + $gatheredLineMap[] = $this->sourceLineFor($lineMap, $j); $j++; continue; @@ -2247,19 +2298,23 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int } } $gathered[] = IndentationHelper::stripLeadingIndent($subLine, $contentIndent); + $gatheredLineMap[] = $this->sourceLineFor($lineMap, $j); $j++; $consumedThrough = $j; } // Drop trailing blank lines collected past the last real content. while ($gathered !== [] && $gathered[array_key_last($gathered)] === '') { array_pop($gathered); + array_pop($gatheredLineMap); } if ($gathered !== []) { if ($itemLines !== [] && $itemLines[array_key_last($itemLines)] !== '') { $itemLines[] = ''; + $itemLineMap[] = -1; } - foreach ($gathered as $gatheredLine) { + foreach ($gathered as $gatheredIndex => $gatheredLine) { $itemLines[] = $gatheredLine; + $itemLineMap[] = $gatheredLineMap[$gatheredIndex]; } // Resume after the last consumed content line, not past blank // lines that precede outer content (the outer loop needs them @@ -2341,6 +2396,7 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int if ($contIndent >= $contentIndent) { if ($itemLines !== [] && $itemLines[array_key_last($itemLines)] !== '') { $itemLines[] = ''; + $itemLineMap[] = -1; } $hasNonMarkerContinuation = true; $parseItemLinesAsBlocks = true; @@ -2352,6 +2408,7 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int $contIndent = IndentationHelper::getLeadingSpaces($contLine); if ($contIndent >= $contentIndent) { $itemLines[] = IndentationHelper::stripLeadingIndent($contLine, $contentIndent); + $itemLineMap[] = $this->sourceLineFor($lineMap, $i); $i++; } else { break; @@ -2372,8 +2429,10 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int // interrupted without a blank line). if ($itemLines !== [] && $itemLines[array_key_last($itemLines)] !== '') { $itemLines[] = ''; + $itemLineMap[] = -1; } $itemLines[] = IndentationHelper::stripLeadingIndent($potentialAttrLine, $contentIndent); + $itemLineMap[] = $this->sourceLineFor($lineMap, $i); $hasNonMarkerContinuation = true; $parseItemLinesAsBlocks = true; $i++; @@ -2388,6 +2447,7 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int $contIndent = IndentationHelper::getLeadingSpaces($contLine); if ($contIndent >= $contentIndent) { $itemLines[] = IndentationHelper::stripLeadingIndent($contLine, $contentIndent); + $itemLineMap[] = $this->sourceLineFor($lineMap, $i); $i++; } else { break; @@ -2407,6 +2467,7 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int // (blockquote, code fence, ...) be recognized instead of escaped. if ($itemContent === '' && count($itemLines) > 1 && $itemLines[0] === '') { array_shift($itemLines); + array_shift($itemLineMap); } // For tight lists with continuation lines, check if content starts with @@ -2419,7 +2480,7 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int // Content starts with a block element (blockquote, code fence, // etc.) or we pushed a {...} back into itemLines that must be // recognized as a block attribute for the next block. - $this->parseBlocks($listItem, $itemLines, 0); + $this->parseBlocks($listItem, $itemLines, 0, $itemLineMap); } else { $paragraph = new Paragraph(); // Plain-paragraph item: continuation lines strip ALL leading @@ -2434,7 +2495,7 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int } elseif ($itemLines !== ['']) { // A bare marker with no content is an empty item: leave it childless // so it renders as `
  • \n
    • ` rather than wrapping a blank line. - $this->parseBlocks($listItem, $itemLines, 0); + $this->parseBlocks($listItem, $itemLines, 0, $itemLineMap); } // When a list-nesting mode is active, check for immediate nested @@ -2457,12 +2518,14 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int // If there's indented content that could be a nested block if ($enterNesting) { $subLines = []; + $subLineMap = []; $nestedIndent = $nextIndent; while ($i < $count) { $subLine = $lines[$i]; if (IndentationHelper::isBlankLine($subLine)) { // Continue across blank lines (same as standard nesting path) $subLines[] = ''; + $subLineMap[] = $this->sourceLineFor($lineMap, $i); $i++; continue; @@ -2485,10 +2548,11 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int // in the item rather than letting them escape. $strip = $lineIndent >= $nestedIndent ? $nestedIndent : $contentIndent; $subLines[] = IndentationHelper::stripLeadingIndent($subLine, $strip); + $subLineMap[] = $this->sourceLineFor($lineMap, $i); $i++; } if ($subLines !== []) { - $this->parseBlocks($listItem, $subLines, 0); + $this->parseBlocks($listItem, $subLines, 0, $subLineMap); } } } @@ -2499,6 +2563,9 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int $listItem->setAttribute($key, $value); } } + if ($this->trackSourceLines && $listItemSourceLine >= 0 && $listItem->getAttribute('data-source-line') === null) { + $listItem->setAttribute('data-source-line', (string)($listItemSourceLine + 1)); + } $list->appendChild($listItem); } @@ -2517,8 +2584,9 @@ protected function tryParseList(Node $parent, array $lines, int $start): ?int * @param \Djot\Node\Node $parent * @param array $lines * @param int $start + * @param array|null $lineMap */ - protected function tryParseDjotDefinitionList(Node $parent, array $lines, int $start): ?int + protected function tryParseDjotDefinitionList(Node $parent, array $lines, int $start, ?array $lineMap = null): ?int { $defList = new DefinitionList(); @@ -2546,6 +2614,7 @@ protected function tryParseDjotDefinitionList(Node $parent, array $lines, int $s } // Collect all consecutive terms (multiple `: term` lines can share one definition) + /** @var array, attributes: array, sourceLine: int}> $terms */ $terms = []; $codeFenceInfo = null; @@ -2577,13 +2646,14 @@ protected function tryParseDjotDefinitionList(Node $parent, array $lines, int $s if ($termStartsWithCodeFence) { // Code fence starts definition - create empty term and break $codeFenceInfo = $termContent; - $terms[] = ['lines' => [], 'attributes' => []]; + $terms[] = ['lines' => [], 'attributes' => [], 'sourceLine' => $this->sourceLineFor($lineMap, $i)]; $i++; break; } $termLines = [$termContent]; + $termLineMap = [$this->sourceLineFor($lineMap, $i)]; $termAttributes = []; $i++; @@ -2596,6 +2666,7 @@ protected function tryParseDjotDefinitionList(Node $parent, array $lines, int $s // Single space continuation is part of term if (preg_match('/^ ([^ ].*)$/', $nextLine, $contMatch)) { $termLines[] = $contMatch[1]; + $termLineMap[] = $this->sourceLineFor($lineMap, $i); $i++; } else { break; @@ -2611,7 +2682,7 @@ protected function tryParseDjotDefinitionList(Node $parent, array $lines, int $s } } - $terms[] = ['lines' => $termLines, 'attributes' => $termAttributes]; + $terms[] = ['lines' => $termLines, 'attributes' => $termAttributes, 'sourceLine' => $termLineMap[0]]; // Check if next non-blank line is another term or definition content $peekIdx = $i; @@ -2638,17 +2709,28 @@ protected function tryParseDjotDefinitionList(Node $parent, array $lines, int $s $term->setAttribute($key, $value); } } + if ( + $this->trackSourceLines + && $termData['sourceLine'] >= 0 + && $term->getAttribute('data-source-line') === null + ) { + $term->setAttribute('data-source-line', (string)($termData['sourceLine'] + 1)); + } $defList->appendChild($term); } // Now collect definition content (after blank line, 2-space indent) // Use `: +` marker to create additional dd elements for the same term $defLines = []; + $defLineMap = []; + /** @var array, map: array}> $allDefBlocks */ $allDefBlocks = []; // If term started with code fence, add it to definition content if ($codeFenceInfo !== null) { $defLines[] = $codeFenceInfo; + $lastTerm = end($terms); + $defLineMap[] = $lastTerm === false ? -1 : $lastTerm['sourceLine']; } while ($i < $count) { @@ -2656,6 +2738,7 @@ protected function tryParseDjotDefinitionList(Node $parent, array $lines, int $s if (IndentationHelper::isBlankLine($defLine)) { $defLines[] = ''; + $defLineMap[] = $this->sourceLineFor($lineMap, $i); $i++; continue; @@ -2664,8 +2747,9 @@ protected function tryParseDjotDefinitionList(Node $parent, array $lines, int $s // Check for continuation marker `: +` - creates new dd for same term if ($defLine === ': +') { if ($defLines !== []) { - $allDefBlocks[] = $defLines; + $allDefBlocks[] = ['lines' => $defLines, 'map' => $defLineMap]; $defLines = []; + $defLineMap = []; } $i++; @@ -2680,6 +2764,7 @@ protected function tryParseDjotDefinitionList(Node $parent, array $lines, int $s // Definition content must be indented by 2 spaces if (preg_match('/^ (.*)$/', $defLine, $defMatch)) { $defLines[] = $defMatch[1]; + $defLineMap[] = $this->sourceLineFor($lineMap, $i); $i++; } else { break; @@ -2688,21 +2773,25 @@ protected function tryParseDjotDefinitionList(Node $parent, array $lines, int $s // Add final block if ($defLines !== []) { - $allDefBlocks[] = $defLines; + $allDefBlocks[] = ['lines' => $defLines, 'map' => $defLineMap]; } // Create definition node(s) if ($allDefBlocks !== []) { - foreach ($allDefBlocks as $block) { + foreach ($allDefBlocks as $blockData) { $def = new DefinitionDescription(); $defAttributes = []; + $block = $blockData['lines']; + $blockLineMap = $blockData['map']; // Skip leading/trailing blank lines while ($block !== [] && $block[0] === '') { array_shift($block); + array_shift($blockLineMap); } while ($block !== [] && end($block) === '') { array_pop($block); + array_pop($blockLineMap); } // Check if last line is a standalone attribute block for the dd @@ -2710,10 +2799,11 @@ protected function tryParseDjotDefinitionList(Node $parent, array $lines, int $s if ($blockCount > 0 && preg_match('/^\{([^{}]+)\}\s*$/', $block[$blockCount - 1], $attrMatch)) { $defAttributes = AttributeParser::parse($attrMatch[1]); array_pop($block); + array_pop($blockLineMap); } if ($block !== []) { - $this->parseBlocks($def, $block, 0); + $this->parseBlocks($def, $block, 0, $blockLineMap); } // Apply definition attributes @@ -2722,11 +2812,22 @@ protected function tryParseDjotDefinitionList(Node $parent, array $lines, int $s $def->setAttribute($key, $value); } } + $defSourceLine = $blockLineMap[0] ?? -1; + if ($this->trackSourceLines && $defSourceLine >= 0 && $def->getAttribute('data-source-line') === null) { + $def->setAttribute('data-source-line', (string)($defSourceLine + 1)); + } $defList->appendChild($def); } } else { - // Term with no definition content - create empty dd - $defList->appendChild(new DefinitionDescription()); + // Term with no definition content - create empty dd, anchored + // to its term's line (there is no content line of its own). + $def = new DefinitionDescription(); + $lastTerm = end($terms); + $defSourceLine = $lastTerm === false ? -1 : $lastTerm['sourceLine']; + if ($this->trackSourceLines && $defSourceLine >= 0) { + $def->setAttribute('data-source-line', (string)($defSourceLine + 1)); + } + $defList->appendChild($def); } } diff --git a/tests/TestCase/SourceLineTrackingTest.php b/tests/TestCase/SourceLineTrackingTest.php index 929c683..74b3704 100644 --- a/tests/TestCase/SourceLineTrackingTest.php +++ b/tests/TestCase/SourceLineTrackingTest.php @@ -5,6 +5,7 @@ namespace Djot\Test\TestCase; use Djot\DjotConverter; +use Djot\Node\Block\Div; use PHPUnit\Framework\TestCase; class SourceLineTrackingTest extends TestCase @@ -39,6 +40,118 @@ public function testEnabledStampsListAndBlockquote(): void $this->assertMatchesRegularExpression('/]*data-source-line="6"/', $html); } + public function testEnabledStampsBlocksInsideBlockquote(): void + { + $converter = new DjotConverter(sourceLines: true); + $html = $converter->convert("> First paragraph.\n>\n> Second paragraph.\n"); + + $this->assertMatchesRegularExpression('/]*data-source-line="1"/', $html); + $this->assertStringContainsString('

    First paragraph.

    ', $html); + $this->assertStringContainsString('

    Second paragraph.

    ', $html); + } + + public function testBlockquoteLazyContinuationKeepsOriginalLine(): void + { + $converter = new DjotConverter(sourceLines: true); + $html = $converter->convert("> quoted\nlazy continuation\n\nAfter\n"); + + $this->assertStringContainsString("

    quoted\nlazy continuation

    ", $html); + $this->assertStringContainsString('

    After

    ', $html); + } + + public function testEnabledStampsBlocksInsideDiv(): void + { + $converter = new DjotConverter(sourceLines: true); + $html = $converter->convert("::: note\nNested paragraph.\n\n```\ncode\n```\n:::\n"); + + $this->assertMatchesRegularExpression('/
    assertStringContainsString('

    Nested paragraph.

    ', $html); + $this->assertStringContainsString('
    code', $html);
+    }
+
+    public function testEnabledStampsListItemsLooseParagraphsAndNestedSublists(): void
+    {
+        $converter = new DjotConverter(sourceLines: true, nestedListsWithoutBlankLine: true);
+        $html = $converter->convert("- first\n\n  second\n\n  - nested\n    - deep\n");
+
+        $this->assertMatchesRegularExpression('/
  • /', $html);
+        $this->assertStringContainsString('
    first
    ', $html);
+        $this->assertStringContainsString('
    second
    ', $html);
+        $this->assertMatchesRegularExpression('/
      /', $html);
+        $this->assertMatchesRegularExpression('/
    • /', $html);
+        $this->assertMatchesRegularExpression('/
        /', $html);
+        $this->assertMatchesRegularExpression('/
      • /', $html);
+    }
+
+    public function testListSyntheticSeparatorDoesNotBecomeSourceLine(): void
+    {
+        $converter = new DjotConverter(sourceLines: true);
+        $html = $converter->convert("- first\n  {.note}\n  second\n");
+
+        $this->assertStringContainsString('
        first
        ', $html);
+        $this->assertStringContainsString('
        second
        ', $html);
+        $this->assertStringNotContainsString('data-source-line="0"', $html);
+    }
+
+    public function testListInsideBlockquoteComposesLineMap(): void
+    {
+        $converter = new DjotConverter(sourceLines: true, nestedListsWithoutBlankLine: true);
+        $html = $converter->convert("> - quoted item\n>   - nested item\n");
+
+        $this->assertMatchesRegularExpression('/
          /', $html);
+        $this->assertMatchesRegularExpression('/
        • /', $html);
+        $this->assertStringContainsString('
          quoted item
          ', $html);
+        $this->assertMatchesRegularExpression('/
            /', $html);
+        $this->assertMatchesRegularExpression('/
          • /', $html);
+        $this->assertStringContainsString('
            nested item
            ', $html);
+    }
+
+    public function testFootnoteContentBlocksAreStamped(): void
+    {
+        $converter = new DjotConverter(sourceLines: true);
+        $html = $converter->convert("Text[^a]\n\n[^a]: foot one\n\n  foot two\n");
+
+        $this->assertStringContainsString('
            foot one
            ', $html);
+        $this->assertStringContainsString('
            foot two', $html);
+    }
+
+    public function testDefinitionListTermsAndDescriptionsAreStamped(): void
+    {
+        $converter = new DjotConverter(sourceLines: true);
+        $html = $converter->convert(": term\n  definition\n");
+
+        $this->assertMatchesRegularExpression('/
            /', $html);
+        $this->assertStringContainsString('
            term
            ', $html);
+        $this->assertMatchesRegularExpression('/
            /', $html);
+        $this->assertStringContainsString('
            definition
            ', $html);
+    }
+
+    public function testNestedAuthorSourceLineAttributeIsNotOverwritten(): void
+    {
+        $converter = new DjotConverter(sourceLines: true);
+        $html = $converter->convert("> {data-source-line=99}\n> Nested\n");
+
+        $this->assertStringContainsString('
            Nested
            ', $html);
+    }
+
+    public function testDisabledOutputHasNoNestedSourceLineAttributes(): void
+    {
+        $converter = new DjotConverter();
+        $html = $converter->convert("> - item\n>   - nested\n");
+
+        $this->assertStringNotContainsString('data-source-line', $html);
+        $this->assertSame("
            \n
              \n
            • \nitem\n- nested\n
              • \n
            \n
            \n", $html);
+    }
+
+    public function testCrlfInputKeepsCorrectSourceLines(): void
+    {
+        $converter = new DjotConverter(sourceLines: true);
+        $html = $converter->convert("> first\r\n>\r\n> second\r\n");
+
+        $this->assertStringContainsString('
            first
            ', $html);
+        $this->assertStringContainsString('
            second
            ', $html);
+    }
+
     public function testSourceLineRendersAfterAuthorAttributes(): void
     {
         $converter = new DjotConverter(sourceLines: true);
@@ -47,4 +160,33 @@ public function testSourceLineRendersAfterAuthorAttributes(): void
         // The attribute is appended after author attributes (attribute order).
         $this->assertStringContainsString('
            ', $html);
     }
+
+    public function testCustomBlockContentIsNotStampedWithLocalLines(): void
+    {
+        $converter = new DjotConverter(sourceLines: true);
+        $parser = $converter->getParser();
+        $parser->addBlockPattern('/^!!!\s*$/', function ($lines, $start, $parent, $p) {
+            $content = [];
+            $i = $start + 1;
+            $count = count($lines);
+            while ($i < $count && preg_match('/^\s+(.*)$/', $lines[$i], $match)) {
+                $content[] = $match[1];
+                $i++;
+            }
+
+            $div = new Div();
+            $p->parseBlockContent($div, $content);
+            $parent->appendChild($div);
+
+            return $i - $start;
+        });
+        $html = $converter->convert("Intro\n\n!!!\n  nested\n");
+
+        // The custom block node itself is stamped with its document line, but
+        // its extracted content has no document positions - the nested
+        // paragraph must not be mis-stamped with a local index (line 1).
+        $this->assertStringContainsString('
            Intro
            ', $html);
+        $this->assertStringContainsString('
            ', $html);
+        $this->assertStringContainsString('
            nested
            ', $html);
+    }
 }

From a64e7c3a484f6f79f31fc34f49ce190d61395b9b Mon Sep 17 00:00:00 2001
From: Mark Scherer 
Date: Sat, 18 Jul 2026 14:34:24 +0200
Subject: [PATCH 2/2] Document source-line attribute stability and future
 position extensions

---
 docs/reference/api.md | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/docs/reference/api.md b/docs/reference/api.md
index 8daadfb..cfb8b3a 100644
--- a/docs/reference/api.md
+++ b/docs/reference/api.md
@@ -438,6 +438,16 @@ the block at the top of the source pane, find the element whose
 `data-source-line` matches in the rendered pane, and scroll it into view. The
 1-based value matches editor gutters (Monaco / CodeMirror).
 
+::: info Stability and future extensions
+`data-source-line` is the stable, lean tier of source mapping: a 1-based start
+line on block-level elements, intended for scroll-sync anchors. Its name, value
+format, and block-level scope will not change. Richer mappings (start/end
+ranges with column and offset, inline elements - what djot.js emits as
+`data-startpos` / `data-endpos` under its `sourcePositions` option) are
+deliberately **not** folded into this attribute; if added later, they will
+arrive as a separate opt-in option alongside it.
+:::
+
 When you construct the parser yourself, enable it there instead — the converter
 `sourceLines` flag is ignored once a pre-configured `$parser` is passed: