Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
3c04c62
Add typecheck script and clear tsc --noEmit baseline
oshorefueled Jul 13, 2026
e319bba
Exclude docs/research from eslint
oshorefueled Jul 13, 2026
20e0acc
Add vitest.config.ts for stable test resolution
oshorefueled Jul 13, 2026
3580e49
Add npm run verify aggregate gate
oshorefueled Jul 13, 2026
0fe8700
Add typecheck CI job and align test workflow Node version
oshorefueled Jul 13, 2026
6c271f1
Deprecate --mode agent and fall back to standard
oshorefueled Jul 13, 2026
a2e9f25
Append Phase 1 completion note to harness audit
oshorefueled Jul 13, 2026
888f027
Add root .vectorlint.ini for reproducible smoke validation
oshorefueled Jul 13, 2026
99f73c2
Add neutral review-domain contract module
oshorefueled Jul 13, 2026
e4259e7
Extract single finding evidence verifier
oshorefueled Jul 14, 2026
25ec49a
Add shared finding severity and scoring helpers
oshorefueled Jul 14, 2026
20d42ff
Implement shared finding processor contract
oshorefueled Jul 14, 2026
671da9c
Route standard check evaluation through shared processor
oshorefueled Jul 14, 2026
9fe6440
Reject judge prompt type at the loader boundary
oshorefueled Jul 14, 2026
d36d06d
Remove legacy judge review path
oshorefueled Jul 14, 2026
d4ea9a7
Remove orphaned orchestrator param types from CLI contract
oshorefueled Jul 14, 2026
15ec7a3
Document finding-processing behavior and clean stale judge docs
oshorefueled Jul 14, 2026
4fa0701
Sanitize prompt criteria at the findings boundary
oshorefueled Jul 14, 2026
99e33d6
Reframe agent-mode messaging as unreleased internal path
oshorefueled Jul 14, 2026
716a77c
refactor(vectorlint): remove committed planning references
oshorefueled Jul 14, 2026
8f0680b
refactor(cli): reframe agent mode as internal
oshorefueled Jul 14, 2026
4c9519d
Remove planning artifact references from review docs
oshorefueled Jul 14, 2026
eb6c938
Merge remote-tracking branch 'origin/codex/ci/harness-stop-the-bleedi…
oshorefueled Jul 14, 2026
fbc9a76
Merge remote-tracking branch 'origin/main' into codex/feat/harness-re…
oshorefueled Jul 17, 2026
f17dc67
Merge parent harness review contract into finding processing
oshorefueled Jul 17, 2026
67b8a5a
Merge remote-tracking branch 'origin/main' into codex/refactor/harnes…
oshorefueled Jul 22, 2026
af25452
refactor(vectorlint): remove evaluation modes
oshorefueled Jul 22, 2026
f2da13f
refactor(vectorlint): harden finding processing boundaries
oshorefueled Jul 23, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ This repository implements VectorLint — a prompt‑driven, structured‑output
- `providers/` — LLM abstractions (OpenAI, Anthropic, Azure, Gemini), request builder, provider factory
- `scan/` — file discovery (fast‑glob) honoring config and exclusions
- `schemas/` — Zod schemas for all external data (API responses, config, CLI, env)
- `scoring/` — score calculation (density-based for check, rubric-based for judge)
- `scoring/` — density-based score calculation
- `types/` — TypeScript type definitions
- `presets/` — bundled rule packs (e.g., `VectorLint/`)
- `tests/` — Vitest specs for config, scanning, evaluation, providers
Expand Down Expand Up @@ -182,7 +182,7 @@ For documents >600 words, VectorLint automatically chunks content:
- Type safety: strict TypeScript with no `any`; use `unknown` + schema validation for external data
- Dependency inversion: depend on `LLMProvider` and `SearchProvider` interfaces; keep providers thin (transport only)
- Dependency injection: inject `RequestBuilder` via provider constructor to avoid coupling
- Separation of concerns: rules define rubric; schemas enforce structure; CLI orchestrates; evaluators process; reporters format
- Separation of concerns: rules define criteria; schemas enforce structure; CLI orchestrates; evaluators process; reporters format
- Separation of concerns: when a file starts combining contracts, orchestration, and utility logic, extract shared helpers and types into focused modules
- Extensibility: add providers by implementing `LLMProvider` or `SearchProvider`; add evaluators via registry pattern
- Shared domain constants: avoid magic strings for core runtime concepts; define shared constants, enums, or types and import them where needed
Expand Down
2 changes: 1 addition & 1 deletion CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ CONFIDENCE_THRESHOLD=0.75
```

- Default: `0.75`
- Applies to surfaced violations in check and judge evaluations
- Applies to surfaced violations
- Invalid values gracefully fall back to the default

---
Expand Down
229 changes: 18 additions & 211 deletions CREATING_RULES.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,7 @@ A comprehensive guide to creating powerful, reusable content evaluations using V

- [Overview](#overview)
- [Rule Anatomy](#rule-anatomy)
- [Evaluation Modes](#evaluation-modes)
- [Check Rules](#check-rules)
- [Judge Rules](#judge-rules)
- [How Rules Work](#how-rules-work)
- [Target Specification](#target-specification)
- [Configuration Reference](#configuration-reference)
- [Best Practices](#best-practices)
Expand All @@ -25,7 +23,7 @@ VectorLint rules are Markdown files with YAML frontmatter that define how your c
- **Rule = Prompt file** (`.md` file organized in rule packs)
- **Pack** = Subdirectory containing related rules (typically named after a company/style guide)
- **Criteria** = Individual quality checks within a rule
- **Score** = LLM-assigned rating (1-4 scale for judge, density-based for check)
- **Score** = Density-based quality score derived from violation count

- **Severity** = How failures are reported (`error` or `warning`)

Expand All @@ -41,7 +39,6 @@ Every rule is a Markdown file with two parts:
id: MyEval
name: My Content Evaluator
evaluator: base
type: check
severity: error
---

Expand Down Expand Up @@ -72,41 +69,16 @@ project/

---

## Evaluation Modes
## How Rules Work

VectorLint uses a single **Base Evaluator** (`evaluator: base`) that operates in two distinct modes, determined by the `type` field:

| Mode | `type` | Use Case | Scoring | Output |
| ---------- | ------- | ------------------------------------- | --------------------------- | ----------------------- |
| **Check** | `check` | Pass/fail checks, counting violations | 10 points - 1 per violation | List of specific issues |
| **Judge** | `judge` | Multi-dimensional quality scoring | 0-4 scale per criterion | Weighted average score |

### When to Use Each

**Use Check when:**

- You need to find specific errors (e.g., "Find all grammar mistakes")
- The check is binary (Pass/Fail) for each item
- You want a list of specific violations to fix

**Use Judge when:**

- You're measuring quality on a spectrum (e.g., "How engaging is this?")
- You have multiple dimensions (Clarity, Tone, Depth)
- You need weighted importance (some criteria matter more)

---

## Check Rules

Check rules are perfect for finding specific issues. The LLM lists violations, and the score is calculated based on the count of violations.
The LLM lists specific violations, and VectorLint calculates a density-based
score from the verified findings.

### Minimal Example

```markdown
---
evaluator: base
type: check
id: GrammarChecker
name: Grammar Checker
severity: error
Expand Down Expand Up @@ -141,71 +113,6 @@ Check this content for grammar issues, spelling errors, and punctuation mistakes

---

## Judge Rules

Judge rules use weighted criteria and a 1-4 rubric for sophisticated quality measurement.

### Structure

```markdown
---
specVersion: 1.0.0
evaluator: base
type: judge
id: HeadlineEvaluator
name: Headline Evaluator

severity: error
criteria:
- name: Value Communication
id: ValueCommunication
weight: 12
- name: Curiosity Gap
id: CuriosityGap
weight: 2
---

You are a headline evaluator... [Your detailed instructions]

## RUBRIC

# Value Communication <weight=12>

### Excellent <score=4>

Specific, immediately appealing benefit

### Good <score=3>

Clear benefit but less specific impact

...
```

### The 1-4 Scoring Scale

VectorLint uses a **1-4 scale** for all judge criteria, which is then normalized to a 1-10 scale:

| LLM Rating | Meaning | Normalized Score |
| :--------- | :-------- | :--------------- |
| **4** | Excellent | **10.0** |
| **3** | Good | **7.0** |
| **2** | Fair | **4.0** |
| **1** | Poor | **1.0** |

### Score Calculation

1. **Normalization**: We map the 1-4 rating to a 1-10 score using the formula: `1 + ((Rating - 1) / 3) * 9`.
2. **Weighted Average**: The final score is the weighted average of all normalized criterion scores.

**Example:**

- Criterion: "Value Communication" (weight=12)
- Rating: 3 (Good) -> Normalized: 7.0
- Weighted Points: 7.0 \* 12 = 84 points

---

## Target Specification

The `target` field allows you to:
Expand Down Expand Up @@ -243,27 +150,16 @@ target:

### Frontmatter Fields

| Field | Type | Required | Description |
| ------------- | ------------- | -------- | ------------------------------------------------------------------ |
| `specVersion` | string/number | No | Rule specification version (use `1.0.0`) |
| `evaluator` | string | No | Evaluator type: `base`, `technical-accuracy` (default: `base`) |
| `type` | string | No | Mode: `judge` or `check` (default: `check`) |
| `id` | string | **Yes** | Unique identifier (used in error reporting) |
| `name` | string | **Yes** | Human-readable name |

| `severity` | string | No | `error` or `warning` (default: `warning`) |
| `evaluateAs` | string | No | `document` or `chunk` - whether to evaluate content as a whole or in chunks (default: `chunk`) |
| `target` | object | No | Content matching specification |
| `criteria` | array | **Yes\*** | List of evaluation criteria (\*required for judge) |

### Criterion Fields

| Field | Type | Required | Description |
| -------- | ------ | -------- | ------------------------------------------ |
| `name` | string | **Yes** | Human-readable criterion name |
| `id` | string | **Yes** | Unique identifier (PascalCase recommended) |
| `weight` | number | No | Importance weight (default: 1) |
| `target` | object | No | Criterion-specific content matching |
| Field | Type | Required | Description |
| ------------- | ------------- | -------- | -------------------------------------------------------------- |
| `specVersion` | string/number | No | Rule specification version (use `1.0.0`) |
| `evaluator` | string | No | Evaluator type: `base`, `technical-accuracy` (default: `base`) |
| `id` | string | **Yes** | Unique identifier (used in error reporting) |
| `name` | string | **Yes** | Human-readable name |
| `severity` | string | No | `error` or `warning` (default: `warning`) |
| `strictness` | number/string | No | Density penalty: a positive number, `lenient`, `standard`, or `strict` |
| `evaluateAs` | string | No | `document` or `chunk` (default: `chunk`) |
| `target` | object | No | Content matching specification |

---

Expand All @@ -288,25 +184,10 @@ You are a headline evaluator for developer blog posts. Assess whether the headli
2. Uses natural, conversational language (avoid buzzwords)
3. Creates curiosity without being clickbait

For each criterion, provide a score (0-4) and specific examples from the text.
```

### 2. **Use Meaningful Weights (Subjective)**

Scale weights to reflect real-world importance:

```yaml
criteria:
# Technical accuracy is critical
- name: Technical Accuracy
weight: 40

# Readability is important
- name: Readability
weight: 30
For each violation, quote the exact text and suggest a concrete fix.
```

### 3. **Provide Context in Prompts**
### 2. **Provide Context in Prompts**

Help the LLM understand your domain:

Expand All @@ -323,12 +204,11 @@ Help the LLM understand your domain:

## Examples

### Example 1: Simple Grammar Check (Check)
### Example 1: Simple Grammar Rule

```markdown
---
evaluator: base
type: check
id: GrammarChecker
name: Grammar Checker
severity: error
Expand All @@ -338,79 +218,6 @@ Check this content for grammar issues, spelling errors, and punctuation mistakes
Report any errors found with specific examples.
```

### Example 2: Headline Evaluator (Judge)

```markdown
---
specVersion: 1.0.0
evaluator: base
type: judge
id: Headline
name: Headline Evaluator

severity: error
target:
regex: '^#\s+(.+)$'
flags: "mu"
group: 1
required: true
suggestion: Add an H1 headline for the article.
criteria:
- name: Value Communication
id: ValueCommunication
weight: 10
- name: Language Authenticity
id: LanguageAuthenticity
weight: 5
---

You are a headline evaluator. Assess the H1 headline for:

1. **Value Communication** (10 points): Does it clearly state what the reader gains?
2. **Language Authenticity** (5 points): Does it use natural, conversational language?

## RUBRIC

# Value Communication <weight=10>

### Excellent <score=4>

Specific, immediately appealing benefit clearly stated

...
```

### Example 3: AI Pattern Detector (Judge)

```markdown
---
specVersion: 1.0.0
evaluator: base
type: judge
id: AIPatterns
name: AI Pattern Detector

severity: warning
criteria:
- name: Language Authenticity
id: LanguageAuthenticity
weight: 40
- name: Structural Naturalness
id: StructuralNaturalness
weight: 30
---

Detect AI-generated writing patterns in this content.

## INSTRUCTION

Scan for common AI patterns:

1. **Buzzwords**: leverage, synergy, elevate
2. **Formulaic transitions**: Moreover, Furthermore
...
```

## Resources

- [VectorLint README](./README.md) - Installation and basic usage
Expand Down
5 changes: 2 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# VectorLint: Prompt it, Lint it! [![npm version](https://img.shields.io/npm/v/vectorlint.svg)](https://www.npmjs.com/package/vectorlint) [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

VectorLint is a command-line tool that evaluates and scores content using LLMs. It uses [LLM-as-a-Judge](https://en.wikipedia.org/wiki/LLM-as-a-Judge) to catch terminology, technical accuracy, and style issues that require contextual understanding.
VectorLint is a command-line tool that uses LLMs to find and score terminology, technical accuracy, and style issues that require contextual understanding.

![VectorLint Screenshot](./assets/VectorLint_screenshot.jpeg)

Expand Down Expand Up @@ -45,10 +45,9 @@ If you can write a prompt for it, you can lint it with VectorLint.

## Quality Scores

VectorLint scores your content using error density and a rubric-based system, enabling you to measure quality across documentation. This gives your team a shared understanding of which content needs attention and helps track improvements over time.
VectorLint scores your content using error density, enabling you to measure quality across documentation. This gives your team a shared understanding of which content needs attention and helps track improvements over time.

- **Density-Based Scoring:** For errors that can be counted, scores are calculated based on **error density (errors per 100 words)**, making quality assessment fair across documents of any length.
- **Rubric-Based Scoring:** For more nuanced quality standards, like flow and completeness, scores are graded on a 1-4 rubric system and then normalized to a **1-10 scale**.

## How VectorLint Reduces False Positives

Expand Down
18 changes: 0 additions & 18 deletions docs/best-practices.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -51,24 +51,6 @@ LLMs evaluate content relative to an implied standard. Make that standard explic

A grammar rule without context produces generic grammar findings. The same rule with a developer audience context produces findings calibrated to technical writing conventions.

## Use weights that reflect real priorities

In judge rules, weights are the single most important configuration decision. They determine which criteria actually control the final score. Treat them as a statement of your content team's values — not arbitrary numbers.

```yaml
criteria:
- name: Technical Accuracy
weight: 40 # Factual errors erode user trust — this matters most
- name: Clarity
weight: 30 # Unclear docs generate support tickets
- name: Tone
weight: 20 # Important but recoverable in editing
- name: SEO
weight: 10 # Nice to have, never at the expense of the above
```

If everything has the same weight, nothing is prioritized.

## Tier strictness by content type

Not all content deserves the same quality bar. Apply strictness in proportion to how much a failure costs — measured in user trust, support load, or brand impact.
Expand Down
Loading
Loading