Skip to content

feat: review in the document's own language, and release extension 0.3.0 - #28

Merged
Nobbettt merged 10 commits into
mainfrom
feat/multi-language-support
Sep 29, 2026
Merged

Nobbettt merged 10 commits into
mainfrom
feat/multi-language-support

Conversation

@Nobbettt

@Nobbettt Nobbettt commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Summary

sharp-pen now reviews every document in its own language, detected automatically from the text, including documents that mix languages. This PR also releases VS Code extension 0.3.0.

Extension

  • Language: the prompt tells every AI client to detect the language from the text, review each passage by its own language's rules, keep the spelling variant, and never translate. Each note is in the same language as the text it's about. There is no language setting.
  • Protected text: quoted text, product names, technical terms, code-like text and deliberate fragments are never changed.
  • Quality:
    • fewer, confident suggestions
    • Level 1 errors are always listed at Level 1, even when a Level 2 rewrite covers the sentence
    • short or common words get a neighbouring word, so their anchor is unambiguous
  • Reliability: the prompt has a bilingual worked example, describes the response format once (by its schema), and repeats the output rule after the document.
  • Version 0.3.0, with a CHANGELOG.md entry.

Skill

  • Language: the skill identifies the draft's language or languages, reviews each passage by its own language's rules, and never translates. Its reply to the author is in the author's language.
  • changes.json: gains an optional, validated language tag, such as sv or en-GB. The page sets it as the lang of its text panes.
  • Wording: Level 2 is renamed to "sentence structure and clarity".

Tests and docs

  • npm run eval:languages: runs the real AI CLIs on four mixed-language documents: Hungarian/Spanish in both orders, Swedish/English/Hungarian, and Swedish/Norwegian/Danish. It scores recall and flags protected text that changed, Level 1 rewrites, and notes in the wrong language. It isn't part of CI; its scoring has unit tests that are.
  • READMEs: both say sharp-pen works in every language the chosen model supports.

After merging

  • Skill and plugin: v0.1.6 publishes automatically.
  • Extension: push the tag extension-v0.3.0 on main to publish 0.3.0 to the VS Code Marketplace.
  • Still to rename: "sentence construction" is still used in the plugin manifests, the extension README and the website.

Test plan

  • Unit tests: npm test, 270 tests, pass.
  • Language evaluation: npm run eval:languages, 3 runs per AI tool and document. Claude, Codex and OpenCode pass all four documents. Copilot passes two; it fails both Hungarian/Spanish documents at 86% recall, because its default model misses the Hungarian comma before "aki" and some Spanish agreement. No tool changed protected text or mixed up languages.
  • Skill evaluation: a throwaway test with the skill's real instructions and build.py, 2 runs per tool and document. Claude and Codex caught 100% of planted errors, Copilot 88–100%.
  • build.py: validates language, and the page sets lang on its text panes.
  • CI: green on all 20 jobs, covering Linux, macOS and Windows on x64 and ARM, and VS Code 1.85.0.

Nobbettt and others added 10 commits September 29, 2026 20:29
The analysis prompt now tells every AI client to detect the document's
language from the text (it is never supplied or set by the user),
review in it without translating, keep its spelling variant, leave
passages in other languages alone, and write notes in the main
language.

Also: Level 2 is described as sentence structure and clarity instead
of the translated term "sentence construction"; quoted text, names,
technical terms, code-like text and deliberate fragments are protected;
fewer, confident suggestions are preferred; one worked example is
included; the response shape is described once, by the schema; and the
output rule is repeated after the document.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The skill works out the draft's language and spelling variant from the
text, reviews by that language's own rules without translating, writes
notes in it, and names it in the reply so the author can correct it.
changes.json gains an optional, validated `language` BCP 47 tag, which
the page sets on its text panes for screen readers and spellcheck.

Level 2 is renamed to "sentence structure and clarity", and quoted
text, names, technical terms, code-like text and deliberate fragments
are left alone, matching the extension's prompt.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
"Leave passages in another language unchanged" made Codex skip a
document's whole second language, and notes followed whichever language
came first. The prompt now reviews each passage by its own language's
rules, applies both levels to every language, and leaves only short
insertions (quotations, loanwords, names) unchanged.

Also tightened after testing: each note is in the same language as its
from text, with a bilingual worked example showing it; Level 1 errors
are always listed at Level 1 even when a Level 2 rewrite covers the
sentence; and very short or common words get a neighbouring word so
their anchor is unambiguous.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Matches the extension's prompt: every passage is reviewed by its own
language's rules, only short foreign insertions are left alone, each
note is in the language of the text it is about, and `language` is the
language most of the prose is in.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`npm run eval:languages` runs the signed-in CLIs, several times each,
on four mixed-language documents with an answer key of planted errors:
Hungarian/Spanish in both orders, Swedish/English/Hungarian, and the
closely related Swedish/Norwegian/Danish. It scores recall, text that
must not change (quotations, names, UK spellings, control sentences),
Level 1 rewrites, and notes or replacements in the wrong language.
Raw responses are saved so a run can be re-scored with --rescore.

It makes real model calls and is not part of npm test or CI; its
scoring and language detector have unit tests that are.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Testing showed Codex answering an English request in Hungarian, Spanish
or Swedish, extending the notes-in-the-draft's-language rule to its
reply. Step 5 now says to reply in the language the author writes in.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… eval

A correct fix of a short word ("was" -> "were") changes most of it, so
the rewrite check now skips anchors under 6 characters.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Nobbettt Nobbettt changed the title feat: review in the document's own, auto-detected language feat: review in the document's own language, and release extension 0.3.0 Sep 29, 2026
@Nobbettt
Nobbettt marked this pull request as ready for review September 29, 2026 19:37
@Nobbettt
Nobbettt merged commit b9e520c into main Sep 29, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant