Shrey/dev

.gitignore

@@ -15,3 +15,7 @@ test-results
 # persists across sessions unlike a temp scratchpad. See PR_REVIEW.md's
 # own "keep this local" instruction for why.
 /.local/
+
+# Mac-only dev scripts (Windows originals start.sh/stop.sh are committed)
+/start-mac.sh
+/stop-mac.sh

DEV_TRACKER.md

@@ -8,7 +8,79 @@ accurate so anyone (human or AI) can pick this up cold without
 re-deriving context. Read `START_HERE.md` first if you haven't — it
 points here plus the rest of the docs in reading order.
 
-Last updated: 2026-07-17.
+Last updated: 2026-07-22.
+
+**Phase 5 — Contract Mining [DONE — 2026-07-22]**: NLI cross-encoder module (`packages/core/src/reprompt_core/nli.py`, lazy-load `cross-encoder/nli-deberta-v3-base` via `@lru_cache`, exact-match fallback when `sentence_transformers` absent). Bidirectional entailment clustering + Shannon entropy (`contract/cluster.py`). Two-axis contract mining (`contract/mine.py`): Axis A = existing trace outputs (no LLM calls), Axis B = K repeats at temperature 0.7 to measure noise floor. Structural invariant extraction: `required_keys` (key intersection), `enum_values` (cardinality ≤5), `regex` (common prefix ≥3 chars). `assertions` DB table + Alembic migration `b2c3d4e5f6a7`. Four API endpoints (list/mine/approve/retire) in `contracts.py` + router wired into `main.py`. `AssertionOut` + 4 client functions in `apps/web/src/lib/api.ts`. `ContractReviewPanel` component + "Contracts" workspace tab. Tests: `test_nli.py` (7), `test_contract_cluster.py` (14), `test_contract_mine.py` (10), `test_contracts.py` (10). `apps/api`: **201 passed** (191 → 201). `packages/core`: **326 passed, 21 skipped** (297 → 326). `apps/web`: **153 passed** unchanged.
+
+**Fix 2 failing web tests [DONE — 2026-07-22]**: `migration-success-screen.test.tsx`'s
+two "Results section" tests (`fetches and renders results once the migration is
+terminal` / `does not fetch results while the migration is still running`) were
+crashing with `TypeError: Cannot read properties of null (reading 'isServer')`
+because `<MigrationSuccessScreen>` renders a `<Link>` from `@tanstack/react-router`
+when `isTerminal` is true, and these tests never provided a `RouterProvider` context.
+Root cause confirmed — `isTerminal` is true for both `"completed"` and `"failed"`
+status fixtures; the `Link` calls `useRouter()` internally, which reads a context
+that doesn't exist in plain `render(...)`. Fix: added a `vi.mock("@tanstack/react-router")`
+at the top of the test file that spreads `vi.importActual` (so everything else — types,
+hooks — is unchanged) and replaces `Link` with a passthrough fragment. App code is
+correct and untouched. `apps/web`: **153 passed** (151 → 153, 0 failing) + clean
+`tsc --noEmit`.
+
+**Graph tab — Upgrade: model nodes + expandable call nodes [DONE — 2026-07-18]**
+(commit `8edaa6e`): Upgraded the Graph tab (see entry directly below) with two
+major additions: (1) **Model nodes** in a fixed right column with dashed edges
+from each stage that uses that model — clicking a model node highlights/unhighlights
+all connected stage nodes (blue border glow); (2) **Inline call nodes** — clicking
+a stage node now expands its individual inference records as child nodes inline below
+it (`CallGraphNode` shows tokens in/out, latency, cost, truncated I/O with "Show full
+text" toggle). Records fetched lazily on first expand via `listStageRecords`,
+cached for re-expand so a second click is instant. `apps/web/src/components/pipeline-graph.tsx`
+only (261 lines changed, 226 replaced). No backend changes, no new test count update
+in commit (fixtures-only changes in sibling test files from the Graph tab commit below
+still apply).
+
+**Graph tab — Obsidian-style pipeline visualization [DONE — 2026-07-18]**
+(commit `dccf39b`): New fifth workspace tab (`"graph"`) added to `WORKSPACE_TABS`
+in `pipeline-workspace.tsx`. New `PipelineGraph` component
+(`apps/web/src/components/pipeline-graph.tsx`, 470 lines): richer stage nodes than
+the Canvas tab (name, model badge, `trace_count`, avg tokens/latency, `total_cost_usd`,
+"View inference calls" affordance), dagre layout reusing the same
+`computeCanvasLayout` function and `["pipeline-dag", id]` React Query cache as the
+Canvas tab (no second fetch when switching between tabs), orientation toggle (→/↓)
+persisted per pipeline in localStorage under a separate key from Canvas tab's layout.
+**ModelPanel** floating card (top-right): lists each unique model in the pipeline;
+click one to highlight all stage nodes using that model with a blue border glow —
+click again to deselect. **Backend**: `StageInfo` extended with `trace_count: int`
+and `total_cost_usd: float | None` — both added to the `/dag` aggregation query
+(`apps/api/src/reprompt_api/pipelines.py`) via `func.count`/`func.sum`; all
+test fixtures updated with the two new fields. `docs/TESTING.md §3.3d` added for the
+Graph tab walkthrough. `apps/web` test file fixture updates only (no new test count
+change — pipeline-graph.tsx itself has no dedicated test file yet).
+
+**Scorecard link + word-diff refinement [DONE — 2026-07-17]** (commit `a2f57cb`):
+Added a "View full results →" `<Link>` button on `MigrationSuccessScreen` (the
+Migrations tab's live run view) that navigates to `/pipelines/$pipelineId/migrations/$migrationId`
+once a migration reaches a terminal state — previously the scorecard route existed
+but nothing linked to it from the run screen. Also refined the word-diff display in
+`migration-detail.tsx`: unchanged text rendered as plain `<span>`, deletions as
+`line-through` in `text-parity-fail`, insertions in `text-parity-pass`. Frontend
+only, no backend changes.
+
+**Not started**: Phase 6 (final end-to-end manual verification).
+
+**Phase 4 — Seam-level regression [DONE — 2026-07-22]**: After an upstream stage is migrated, every downstream stage (per `Stage.dependents` DAG) is re-validated: run the winning upstream prompt → substitute output into the downstream input under the upstream `source_id` key → run the original downstream prompt → score with det+embedding (no judge, same rationale as holdout). New `packages/core/src/reprompt_core/optimizer/seam.py` (`SeamExample`, `SeamInput`, `SeamResult`, `evaluate_seam`). New `SeamCheckResult` DB model + Alembic migration `f1c2d3e4a5b6`. `_run_seam_regression` wired into `optimizer_runner.py` after all stage loops. `GET /pipelines/{pid}/migrations/{mid}/seam-results` endpoint. "Seam checks" table in `migration-detail.tsx`. 7 core unit tests, 5 API tests. `apps/api`: **191 passed** (186 → 191). `packages/core`: **297 passed, 21 skipped** (290 → 297). `apps/web`: **153 passed** (unchanged).
+
+**Phase 3 — Config export [DONE — 2026-07-22]**: `GET /pipelines/{pid}/migrations/{mid}/export` returns the winning migrated config as a JSON attachment keyed by `Stage.source_id`. "Download winning config" button on `migration-detail.tsx` — visible only in terminal state with ≥1 result, uses fetch+blob so cross-origin download works without auth complexity. 6 new API tests. `apps/api`: **186 passed** (180 → 186). `apps/web`: **153 passed** (unchanged).
+
+**Phase 2 — M4 Holdout pass [DONE — 2026-07-22]**: Train/test split in `_build_stage_inputs` (optimizer_runner.py): prefers `Trace.is_holdout`-flagged records; auto-splits last record when none are explicit. `_evaluate_holdout` in core runs the winning prompt on withheld examples scoring with det+embedding only (no judge). `holdout_score` persisted on `Candidate` (new nullable Float column, migration `e4a1b8c7f203`). `StageResultOut.holdout_score` exposed in `GET .../results`. `migration-detail.tsx` shows a "Holdout" column (badge when set, dash when null). `apps/api`: **180 passed** (178 → 180). `apps/web`: **153 passed** (unchanged). `packages/core`: **290 passed, 21 skipped** (unchanged).
+
+**Next phases planned (2026-07-22)**:
+See `docs/PRISM_PHASES_PLAN.md` for the full per-phase spec (data model, API surface, core logic, tests, dependencies) — written by Opus against the PDF + live codebase. Build order: 5 → 8 → 6 → GEPA → 7.
+- **Phase 5 — Automated contract mining**: NLI cross-encoder (DeBERTa-class) + semantic entropy clustering + two-axis sampling → auto-generated executable assertions.
+- **Phase 8 — Executable assertions**: consuming Phase 5's assertion specs, backtracking, counterexample capture.
+- **Phase 6 — Lessons file**: cross-migration model-family memory for the mutator.
+- **GEPA backend**: third optimizer strategy branch.
+- **Phase 7 — Governance plane**: end-to-end regression, promotion gate (staged→live), feature flags, rollback, drift daemon.
 
 **Edit button for inline pipeline rename [DONE — 2026-07-16]**: Added an
 explicit pencil/edit icon button next to the delete icon in the Pipelines
@@ -46,7 +118,16 @@ fixes [DONE — 2026-07-15]"** section (below) — a separate audit's 6
 findings against the already-shipped Prism engine (most notably: the
 critique loop was judge-blind, and `max_refine_rounds=1` made the plateau
 early-stopping logic dead code), `packages/core` only, `264 passed,
-21 skipped` after. **Phase D(a) — Model-card info in wizard [DONE —
+21 skipped` after. **Graph tab [DONE — 2026-07-18]**: new fifth workspace
+tab (Obsidian-style interactive node graph), `PipelineGraph` component in
+`apps/web/src/components/pipeline-graph.tsx` — richer stage nodes with
+trace_count/cost/token stats, ModelPanel (click-to-highlight by model
+family), model nodes in a right column with dashed edges, inline expandable
+call nodes (individual inference records fetched lazily). Backend:
+`StageInfo` now carries `trace_count` + `total_cost_usd`. **Fix 2 failing
+tests [DONE — 2026-07-22]**: mocked `@tanstack/react-router`'s `Link` in
+`migration-success-screen.test.tsx` so terminal-state tests no longer crash
+on missing `RouterProvider` context — `apps/web` **153 passed, 0 failing**. **Phase D(a) — Model-card info in wizard [DONE —
 2026-07-15]**: new read-only endpoint `GET /model-cards/{model}` returns
 family classification and applicable transform rules as JSON; migration
 wizard model picker now fetches and displays these rules human-readable
@@ -3334,3 +3415,44 @@ yet — see `docs/TESTING.md`'s new §3.3b for the current API-level manual
 check and the explicit note that a UI surface (e.g. showing the judge model
 next to a migration's results) is a real, not-yet-built follow-up, out of
 scope for this backend plumbing fix.
+
+## Pre-merge review fixes for PR #8 (Phase 2/3/4 — holdout/config-export/seam) [DONE — 2026-07-22]
+
+A code review pass (evidence-based, `code-reviewer-persona`/`spec-driven-
+planning` skills applied) before merging this branch found two real issues,
+both fixed here:
+
+1. **`_run_seam_regression` hardcoded `parity_threshold=0.95`** instead of
+   using the migration's actual configured value — while the main
+   optimization loop correctly threads `migration.parity_threshold` through
+   twice already (`optimizer_runner.py`, the two `run_optimizer(...)` call
+   sites), the seam-check call site was missed. A migration with a
+   non-default threshold would have every seam check evaluated against the
+   wrong bar. Fixed: `_run_seam_regression` gained a required keyword-only
+   `parity_threshold: float` parameter, threaded from `migration
+   .parity_threshold` at its one call site — same pattern the rest of the
+   file already uses.
+2. **`docs/TESTING.md`'s §3.3d described UI that doesn't exist** — it said
+   the Graph tab has a "floating Models in pipeline panel" and a "calls
+   drawer that slides in from the right." The actual implementation
+   (`pipeline-graph.tsx`) renders everything as inline React Flow nodes
+   (`ModelGraphNode`, `CallGraphNode`) positioned directly in the graph —
+   no panel, no drawer. Rewrote §3.3d to describe the real inline-node UI.
+
+Both fixes verified: `cd apps/api && uv run pytest -q` → **191 passed**
+(unchanged count — the `parity_threshold` fix doesn't add new test
+coverage, it corrects existing runtime behavior; see the review's own
+noted gap below). No `packages/core`/`apps/web` files touched by either
+fix.
+
+**Known gap, not closed by this pass** (flagged by the same review, left
+for a follow-up): the new `apps/api` orchestration glue
+(`_run_seam_regression`, `_persist_holdout_scores`) has zero integration
+test coverage — all of PR #8's own new `test_migrations.py` cases exercise
+only the read endpoints (`/export`, `/seam-results`, `/results`) against
+directly-seeded DB rows, never `_run()`/`_run_seam_regression` itself
+end-to-end. This is exactly why the `parity_threshold` bug above went
+uncaught by the existing suite. Worth a dedicated follow-up: seed a real
+migration fixture, run the actual orchestration function, assert the
+persisted `SeamCheckResult`/`holdout_score` rows are correct — not done
+here to keep this fix pass narrow and mergeable quickly.

START_HERE.md

@@ -71,21 +71,34 @@ proves the outputs still match. Full plain-language explanation: `README.md`.
 embedding + AI judge), rubric generation (works, needs manual trigger),
 BYOK key storage + live model calls, screens 1–5, auth, settings.
 
+**Built since this list was last accurate — do not re-plan these, they're
+done**: rubric-generation trigger + auto model selection, model-card info
+in both the wizard and Settings, the full M3 optimizer loop (both
+strategies, judge-aware critique, plateau early-stopping, near-dup
+filtering), M4 migration runs with a live status view (per-stage
+running/done/failed, sub-step labels, live reasoning feed, activity log),
+the unified pipeline workspace (Canvas/Data/Rubrics/Migrations/Graph tabs),
+project/multi-run ingestion, Pipeline CRUD, per-stage target-model
+overrides, before/after prompt diff, config export, and M4's holdout +
+seam-regression validation passes. See `DEV_TRACKER.md`'s "Current state"
+for the authoritative, actively-updated status — this file's job is to be
+the map, not to re-derive what's already built.
+
 **Not built yet, in order:**
-1. Rubric generation trigger in the UI (endpoint exists, no button calls it yet)
-2. Model-card info (preferred prompt format per model family) surfaced in
-   the migration wizard's model picker — logic exists (`llm/model_card.py`),
-   not connected to that screen
-3. Budget should become optional (currently required) in the migration wizard
-4. **The actual optimizer loop (M3)** — in active development, see
-   `DEV_TRACKER.md` for the exact phase-by-phase status and where to pick
-   up. Two strategies: "simple" (one-shot mutation + sweep) and "Prism"
-   (multi-round mutate → critique → refine, plus optional few-shot
-   selection), both in-house, both provider-agnostic. Each attempt is
-   saved as a `Candidate` row (prompt tried, score, cost) so past attempts
-   are always reviewable.
-5. M4 — full migration run using the M3 loop, progress screen
-6. M5 remainder — scorecard screen, config export (need real M3/M4 output first)
+1. Budget should become optional (currently required) in the migration wizard
+2. **LLM call telemetry + multi-model scorecard** (per-target-model
+   comparison with a decisive "which is actually best" recommendation, not
+   just the single-winner-per-stage view `migration-detail.tsx` currently
+   has) — designed, not built; see `DEV_TRACKER.md`'s "Planned, not yet
+   built — LLM call telemetry + scorecard" section for the full spec.
+3. Phase 6 final end-to-end manual verification (all automated suites are
+   green; a full human click-through of the golden path hasn't been done
+   in one sitting since the M4 holdout/seam work landed)
+4. Further M5/M6-class work is speculatively scoped in
+   `docs/PRISM_PHASES_PLAN.md` (automated contract mining, a GEPA optimizer
+   strategy, a governance/promotion-gate plane) — treat that file as an
+   unverified proposal to reconcile into this list and `DEV_TRACKER.md`
+   before starting, not as an already-agreed roadmap.
 
 ## How to test what exists right now
 

apps/api/alembic/versions/b2c3d4e5f6a7_add_assertions.py

@@ -0,0 +1,48 @@
+"""add assertions table (Phase 5 contract mining registry)
+
+Revision ID: b2c3d4e5f6a7
+Revises: f1c2d3e4a5b6
+Create Date: 2026-07-22
+"""
+
+from __future__ import annotations
+
+from alembic import op
+import sqlalchemy as sa
+
+revision = "b2c3d4e5f6a7"
+down_revision = "f1c2d3e4a5b6"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "assertions",
+        sa.Column("id", sa.Integer(), nullable=False),
+        sa.Column("stage_id", sa.Integer(), nullable=False),
+        sa.Column("kind", sa.String(length=50), nullable=False),
+        sa.Column("spec", sa.JSON(), nullable=False),
+        sa.Column("version", sa.Integer(), nullable=False, server_default="1"),
+        sa.Column("status", sa.String(length=20), nullable=False, server_default="'candidate'"),
+        sa.Column("source", sa.String(length=20), nullable=False, server_default="'mined'"),
+        sa.Column("counterexamples", sa.JSON(), nullable=False, server_default="'[]'"),
+        sa.Column("noise_floor", sa.Float(), nullable=True),
+        sa.Column("entropy", sa.Float(), nullable=True),
+        sa.Column("description", sa.Text(), nullable=False, server_default="''"),
+        sa.Column("confidence", sa.Float(), nullable=True),
+        sa.Column(
+            "created_at",
+            sa.DateTime(timezone=True),
+            server_default=sa.text("(CURRENT_TIMESTAMP)"),
+            nullable=False,
+        ),
+        sa.ForeignKeyConstraint(["stage_id"], ["stages.id"], ondelete="CASCADE"),
+        sa.PrimaryKeyConstraint("id"),
+    )
+    op.create_index("ix_assertions_stage_id", "assertions", ["stage_id"])
+
+
+def downgrade() -> None:
+    op.drop_index("ix_assertions_stage_id", table_name="assertions")
+    op.drop_table("assertions")

apps/api/alembic/versions/e4a1b8c7f203_add_holdout_score_to_candidates.py

@@ -0,0 +1,33 @@
+"""add holdout_score to candidates
+
+Revision ID: e4a1b8c7f203
+Revises: d3f7a2c1e5b6
+Create Date: 2026-07-22
+
+M4 holdout pass: after optimization the winning prompt is scored on
+examples withheld from training. holdout_score stores that unbiased
+measurement on the Candidate row (nullable — only set on the winner
+after a migration that had holdout examples).
+"""
+
+from __future__ import annotations
+
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "e4a1b8c7f203"
+down_revision: Union[str, Sequence[str], None] = "d3f7a2c1e5b6"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    with op.batch_alter_table("candidates") as batch_op:
+        batch_op.add_column(sa.Column("holdout_score", sa.Float(), nullable=True))
+
+
+def downgrade() -> None:
+    with op.batch_alter_table("candidates") as batch_op:
+        batch_op.drop_column("holdout_score")