Honor TokenReader subrange bounds in peek and backtrack.

[Christopher Pruijsen (cpruijsen)]

TokenReader.peekToken(), peekPreviousTokenKind(), and backtrackToMarker() now honor the same [_readerStartIndex, _readerEndIndex) window that peekTokenKind() / peekTokenAfterKind() / peekTokenAfterAfterKind() already enforce. Fixes #481: an embedded reader could observe the token before its start or after its end, and backtrackToMarker could rewind into the outer stream. TokenReader is not a public export. Current NodeParser hot paths already call peekTokenKind() first, which is why existing parser snapshots did not fail.

Decision

peekToken() past the window returns the stream's EndOfInput token (tokens[tokens.length - 1]); peekPreviousTokenKind() treats _readerStartIndex as start-of-input; backtrackToMarker throws if marker < _readerStartIndex.

Alternative: throw from peekToken() (like readToken()), or clamp an out-of-range marker to the window start. Open PR #482 takes the throw-from-peekToken() option.

readToken() already states that peek must always return a valid Token, and the other peek methods already signal EndOfInput at the window edge. Expired markers already throw as a parser bug. Can switch to throw-on-peek or clamp-on-backtrack.

Test plan

• New tsdoc/src/parser/__tests__/TokenReader.test.ts fails without the source change and passes with it (embedded predecessor leak, next-token leak, empty TokenSequence.createEmpty, out-of-range backtrack).
• Full @microsoft/tsdoc Heft test run (typecheck / ESLint / API Extractor / 259 Jest tests).
• rush change --verify on CI (change file included).
• Optional: playground / api-demo smoke; no parser grammar change.

Change log

Rush change file: common/changes/@microsoft/tsdoc/fix-tokenreader-bounds_2026-09-11-13-00.json (patch).

[Sylvester Kaczmarek (sylvesterkaczmarek)]

At an embedded reader boundary, peekToken() now returns the tokenizer's global EndOfInput token. Its range is at the end of the whole comment, not readerEndIndex. NodeParser uses peekToken().range for diagnostics, so errors at an embedded boundary can point at the wrong location. Can this return an EOI token/range at the subrange end?

[Christopher Pruijsen (cpruijsen)]

True about the range, it is the end of the comment. I do not think it reaches a diagnostic today. The peekToken().range calls in NodeParser are in _parseBlockTag, _parseInlineTag, _parseHtmlEndTag and _parseFencedCode, which all run on the top-level reader, where the window ends at the real EndOfInput. The embedded reader only goes to _parseInheritDocTag and _parseLinkTag, which do not call peekToken(). If the maintainers want a zero-length EndOfInput at the subrange end anyway, it is a few lines and I can add it.

[Sylvester Kaczmarek (sylvesterkaczmarek)]

Thanks for checking the call paths. Agreed that the range concern is not currently observable in NodeParser diagnostics. I would keep this fix focused on preventing out-of-window reads and backtracking. A subrange-local EndOfInput token can be handled separately if a caller needs its range. No further change requested from me.