track(grammar): tree-sitter-cpp blockers behind mozcpp/deepspeech skips (#83) #86

Summary

Track the upstream tree-sitter-cpp blockers that are causing
tree-sitter-mozcpp to skip the test_fn_id_strings test and a set of
DeepSpeech files (see #83). The skips themselves are documented and have
correct FIXME references after #83. This issue exists so we have a single
place to record release / fix status for the seven upstream defects and
trigger a grammar bump when one of them becomes available.

This is a watch-and-bump issue, not a coding task. No code change is
expected unless / until a new tree-sitter-cpp release ships on
crates.io with one of these fixes.

Why we cannot fix this locally

tree-sitter-mozcpp is a thin overlay on top of tree-sitter-cpp. The
overlay (tree-sitter-mozcpp/grammar.js, ~378 lines) only adds
Mozilla-specific macro tokens such as MOZ_NONHEAP_CLASS. Every parse
failure in #83 comes from a structural defect in the underlying
tree-sitter-cpp grammar — binary_expression, preproc_if,
enumerator, etc. We cannot fix those rules in the overlay, and per
AGENTS.md we cannot fork the vendored grammar or repoint
tree-sitter-cpp at a git sha:

External grammar crates are version-pinned (=X.Y.Z) in the root
Cargo.toml. Treat the pinned version as fixed: do not loosen pins
to a range without explicit user approval. Bumping a grammar version
is a deliberate, separate change.

The build pipeline reflects this — generate-grammars/generate-mozcpp.sh
reads the pin from tree-sitter-mozcpp/Cargo.toml and downloads the
matching crate from crates.io. There is no supported path to a git sha.

Status of the seven upstream defects

Current pins:

• root Cargo.toml: tree-sitter-mozcpp = "=0.20.4"
• tree-sitter-mozcpp/Cargo.toml: tree-sitter-cpp = "0.23.4"
• crates.io tree-sitter-cpp max_stable_version: 0.23.4 (2024-11-11)

Six bugs and one feature request remain open upstream. One bug (#252) was
fixed on master in September 2025 but no tree-sitter-cpp release has
been cut since 2024-11-11 (v0.23.4), so the fix is unreachable through
crates.io today.

Acceptance criteria (any one of these unblocks an action)

1. A new tree-sitter-cpp release on crates.io that includes the feat(lib): per-language Cargo features for grammar selection #252
   fix. When that ships:
   • Bump tree-sitter-cpp in tree-sitter-mozcpp/Cargo.toml to the new
     version.
   • Bump tree-sitter-mozcpp major/minor in the root Cargo.toml.
   • Run ./generate-grammars/generate-mozcpp.sh and review the diff.
   • Remove the two fst_test.h entries from the exclusion list in
     tests/deepspeech_test.rs.
   • Re-run cargo insta test --review and accept the resulting snapshot
     drift.
   • Cross-check that no other previously-skipped DeepSpeech file now
     parses cleanly — if so, drop those exclusions too.
2. Any of test(metrics): tighten Npm/Npa annotation-type tests to also catch is_func_space revert #307–fix(cfg_predicate): slow-path whitespace collapser mangles non-ASCII UTF-8 #312 closed upstream and shipped in a release — repeat
   the same flow, dropping the corresponding entries.
3. All seven closed and shipped — remove the FIXME blocks and
   #[ignore] markers entirely (this is the acceptance criterion of
   fix(tests,mozcpp): tree-sitter-cpp parse failures skip mozcpp/deepspeech tests #83), close fix(tests,mozcpp): tree-sitter-cpp parse failures skip mozcpp/deepspeech tests #83 and close this tracking issue.

How to check status quickly

# Latest released tree-sitter-cpp on crates.io
curl -s https://crates.io/api/v1/crates/tree-sitter-cpp \
  | jq '.crate | {max_stable_version, updated_at}'

# Any tree-sitter-cpp release newer than what we pin
gh api repos/tree-sitter/tree-sitter-cpp/releases \
  --jq '.[] | {tag_name, published_at}' | head

# Status of the seven blockers
for n in 307 308 309 310 311 312 252; do
  gh issue view $n --repo tree-sitter/tree-sitter-cpp \
    --json number,title,state,closedAt
done

Related

• fix(tests,mozcpp): tree-sitter-cpp parse failures skip mozcpp/deepspeech tests #83 — local skip tracker for the affected tests (do not close until all
  seven are resolved and the FIXMEs are removed)
• Fix tree-sitter-mozcpp test failures introduced by #1141 mozilla/rust-code-analysis#1142 — the upstream Mozilla ancestor of fix(tests,mozcpp): tree-sitter-cpp parse failures skip mozcpp/deepspeech tests #83

Resolution Plan

Status refreshed 2026-07-31 — nothing has moved

Ran the issue's own status script:

Latest tree-sitter-cpp GitHub release is still v0.23.4, 2024-11-11. So
as of today that is ~20 months with no release cut, and ~10 months
since the #252 fix landed on master unreleased. The table in the issue body
is accurate; it is the waiting that has become the finding.

(The crates.io API call in the "how to check" block returns nulls now — it
needs a User-Agent header. The gh api releases call still works and is the
one to keep.)

Three stale facts in the body

1. The root pin line is out of date. The body says
   root Cargo.toml: tree-sitter-mozcpp = "=0.20.4". It is now
   tree-sitter-mozcpp = { package = "bca-tree-sitter-mozcpp", path = "./tree-sitter-mozcpp", version = "=2.1.0" } — a path dependency on the
   vendored fork, renamed and re-versioned.
2. The root now pins upstream tree-sitter-cpp directly —
   Cargo.toml:70, = "=0.23.4" — which it did not when this was filed.
   Since refactor(api): normalize &LANG-by-value and drop Java-style get_ getter prefixes #507 / refactor(lang): back Cpp with upstream tree-sitter-cpp, add mozcpp opt-in #720 that crate is the default C++ grammar and mozcpp is
   opt-in. So a tree-sitter-cpp release would need bumping in two places,
   not one, and the acceptance-criteria checklist only mentions
   tree-sitter-mozcpp/Cargo.toml.
3. fix(tests,mozcpp): tree-sitter-cpp parse failures skip mozcpp/deepspeech tests #83 is already CLOSED, but the Related section says "do not close until
   all seven are resolved and the FIXMEs are removed", and acceptance criterion
   3 says to close it as part of the final cleanup. tests/deepspeech_test.rs
   still carries a skip marker, so the FIXMEs were not removed. Reconcile:
   either reopen fix(tests,mozcpp): tree-sitter-cpp parse failures skip mozcpp/deepspeech tests #83, or drop it from criterion 3 and note that the skip
   tracking now lives here.

A pin that is not actually pinned

tree-sitter-mozcpp/Cargo.toml:44 reads tree-sitter-cpp = "0.23.4" — a
caret range, not the =0.23.4 form AGENTS.md requires for external grammar
crates. Cargo.lock is committed, so it is held in practice, but a plain
cargo update would silently move it to any 0.23.x that ships — which is
precisely the "deliberate, separate change" this issue exists to gate.

Two siblings have the same shape: tree-sitter-mozjs/Cargo.toml:46
(tree-sitter-javascript = "0.25.0") and tree-sitter-tcl/Cargo.toml:43
(tree-sitter-language = "0.1.0"). Worth tightening all three to = in one
small change — it is independent of any upstream movement and closes the path
by which this bump could happen accidentally rather than deliberately.

The real recommendation: stop watching manually

Every acceptance criterion here is gated on "a new release ships", the issue
has been open on that basis for a long time, and there is no mechanism that
notices when it happens — someone has to remember to run the script. Twenty
months of no release is strong evidence that nobody will.

This repository already has the pattern: mutation-test.yml and
benchmark.yml are quarterly crons with workflow_dispatch. Add
grammar-upstream-watch.yml on the same shape:

• Query gh api repos/tree-sitter/tree-sitter-cpp/releases for the newest tag.
• Compare against the pin parsed out of tree-sitter-mozcpp/Cargo.toml.
• If newer, gh issue comment on this issue with the new version and the
  current state of the seven blockers.
• Also re-query the seven and report any that closed, since a fix landing on
  master is the leading indicator.

That converts a watch issue into a triggered one and is the single change that
makes the rest of this plan actionable rather than aspirational. Extend it to
the other pinned grammars while you are there — the same staleness applies to
every =X.Y.Z in the root manifest.

Consider escalating upstream

Separately, and cheaply: #252's fix has sat unreleased for ten months. A
polite request on the tree-sitter-cpp repository for a release cut is
reasonable and costs nothing. #1058 is a second reason to open a conversation
with that project — worth combining rather than filing twice.

Steps

1. Update the three stale facts in the body; reconcile fix(tests,mozcpp): tree-sitter-cpp parse failures skip mozcpp/deepspeech tests #83.
2. Tighten the three caret-ranged grammar dependencies to =.
3. Fix the crates.io snippet (add a User-Agent) or drop it in favour of the
   gh api one.
4. Add grammar-upstream-watch.yml.
5. Add the two-place bump (tree-sitter-mozcpp/Cargo.toml and root
   Cargo.toml:70) to acceptance criterion 1.
6. Open the upstream release request; combine with fix(tree-sitter-mozcpp): bound the deserialize memcpy in scanner.c #1058's scanner report.
7. Leave the criteria otherwise as written — they are correct and complete.

Assessment

Difficulty — Low. Body edits, three one-character manifest changes, and a
cron workflow modelled on two that already exist. The grammar bump itself, if
a release ever ships, would be Medium — snapshot drift across the C++ corpus —
but that is not this issue's work; the criteria already describe it.

Complexity — Low. No source code, no metric computation, no public API.
The workflow addition needs make actionlint per AGENTS.md, which is the
only gate it touches.

Priority — Low. This is a watch-and-bump tracker by its own description,
and everything it tracks is blocked on a third party that has not cut a release
in twenty months. Nothing here is broken, the skips are documented with correct
references, and the affected grammar is the opt-in one. The automation in step
4 is the part worth doing soon — not because the outcome is urgent, but because
without it this issue's whole purpose depends on someone remembering.

low-priority applied: Priority is Low.

Decision (2026-07-31)

Automate the watch, escalate upstream, and reopen #83.

Status re-checked today with the script above: all six bugs still OPEN, #252
still CLOSED-but-unreleased, and the newest tree-sitter-cpp release is still
v0.23.4 (2024-11-11) — roughly twenty months with no release cut, ten
months since #252's fix landed on master. Three actions follow.

1. Add grammar-upstream-watch.yml. Every acceptance criterion here is
   gated on "a new release ships", and nothing notices when it does — someone
   has to remember to run the script. Twenty months of silence is good evidence
   nobody will. Model it on the existing quarterly crons
   (mutation-test.yml, benchmark.yml: schedule: plus workflow_dispatch):
   compare the newest upstream tag against the pin parsed from
   tree-sitter-mozcpp/Cargo.toml, gh issue comment here when it moves, and
   re-query the seven blockers each run since a master fix is the leading
   indicator. Extend to the other pinned grammars while there. make actionlint
   is required for any workflow edit.

2. Open an upstream release request. feat(lib): per-language Cargo features for grammar selection #252's fix has been unreleased for ten
   months; asking for a release cut is reasonable and costs nothing. fix(tree-sitter-mozcpp): bound the deserialize memcpy in scanner.c #1058
   is a second reason to open a conversation with that project (an unbounded
   memcpy in src/scanner.c, byte-identical in 0.23.4) — combine the two
   rather than filing twice.

3. fix(tests,mozcpp): tree-sitter-cpp parse failures skip mozcpp/deepspeech tests #83 is reopened. It was closed while its FIXMEs are still in the tree
   and all seven blockers remain unresolved, contradicting both the Related
   note here and acceptance criterion 3.

Body corrections applied by this decision

• The root pin line is stale: it now reads
  tree-sitter-mozcpp = { package = "bca-tree-sitter-mozcpp", path = "./tree-sitter-mozcpp", version = "=2.1.0" },
  a path dependency on the vendored fork.
• The root also pins upstream tree-sitter-cpp directly (Cargo.toml:70,
  ="0.23.4"), which it did not when this was filed — since refactor(api): normalize &LANG-by-value and drop Java-style get_ getter prefixes #507 / refactor(lang): back Cpp with upstream tree-sitter-cpp, add mozcpp opt-in #720 that
  crate is the default C++ grammar. Acceptance criterion 1 must bump two
  places, not one.
• The curl snippet returns nulls without a User-Agent header; the
  gh api .../releases call is the one to rely on.
• The caret-range pin in tree-sitter-mozcpp/Cargo.toml:44
  (tree-sitter-cpp = "0.23.4", not ="0.23.4") is tracked in chore(grammars): three vendored manifests use caret ranges instead of = pins #1151 — it
  is the path by which the bump this issue gates could happen accidentally.

The seven blockers' status table and the acceptance criteria are otherwise
correct and unchanged.