diff --git a/.gitignore b/.gitignore index 1a2c94a..6932a05 100644 --- a/.gitignore +++ b/.gitignore @@ -15,3 +15,7 @@ test-results # persists across sessions unlike a temp scratchpad. See PR_REVIEW.md's # own "keep this local" instruction for why. /.local/ + +# Mac-only dev scripts (Windows originals start.sh/stop.sh are committed) +/start-mac.sh +/stop-mac.sh diff --git a/DEV_TRACKER.md b/DEV_TRACKER.md index e78f921..40634db 100644 --- a/DEV_TRACKER.md +++ b/DEV_TRACKER.md @@ -8,7 +8,79 @@ accurate so anyone (human or AI) can pick this up cold without re-deriving context. Read `START_HERE.md` first if you haven't — it points here plus the rest of the docs in reading order. -Last updated: 2026-07-17. +Last updated: 2026-07-22. + +**Phase 5 — Contract Mining [DONE — 2026-07-22]**: NLI cross-encoder module (`packages/core/src/reprompt_core/nli.py`, lazy-load `cross-encoder/nli-deberta-v3-base` via `@lru_cache`, exact-match fallback when `sentence_transformers` absent). Bidirectional entailment clustering + Shannon entropy (`contract/cluster.py`). Two-axis contract mining (`contract/mine.py`): Axis A = existing trace outputs (no LLM calls), Axis B = K repeats at temperature 0.7 to measure noise floor. Structural invariant extraction: `required_keys` (key intersection), `enum_values` (cardinality ≤5), `regex` (common prefix ≥3 chars). `assertions` DB table + Alembic migration `b2c3d4e5f6a7`. Four API endpoints (list/mine/approve/retire) in `contracts.py` + router wired into `main.py`. `AssertionOut` + 4 client functions in `apps/web/src/lib/api.ts`. `ContractReviewPanel` component + "Contracts" workspace tab. Tests: `test_nli.py` (7), `test_contract_cluster.py` (14), `test_contract_mine.py` (10), `test_contracts.py` (10). `apps/api`: **201 passed** (191 → 201). `packages/core`: **326 passed, 21 skipped** (297 → 326). `apps/web`: **153 passed** unchanged. + +**Fix 2 failing web tests [DONE — 2026-07-22]**: `migration-success-screen.test.tsx`'s +two "Results section" tests (`fetches and renders results once the migration is +terminal` / `does not fetch results while the migration is still running`) were +crashing with `TypeError: Cannot read properties of null (reading 'isServer')` +because `` renders a `` from `@tanstack/react-router` +when `isTerminal` is true, and these tests never provided a `RouterProvider` context. +Root cause confirmed — `isTerminal` is true for both `"completed"` and `"failed"` +status fixtures; the `Link` calls `useRouter()` internally, which reads a context +that doesn't exist in plain `render(...)`. Fix: added a `vi.mock("@tanstack/react-router")` +at the top of the test file that spreads `vi.importActual` (so everything else — types, +hooks — is unchanged) and replaces `Link` with a passthrough fragment. App code is +correct and untouched. `apps/web`: **153 passed** (151 → 153, 0 failing) + clean +`tsc --noEmit`. + +**Graph tab — Upgrade: model nodes + expandable call nodes [DONE — 2026-07-18]** +(commit `8edaa6e`): Upgraded the Graph tab (see entry directly below) with two +major additions: (1) **Model nodes** in a fixed right column with dashed edges +from each stage that uses that model — clicking a model node highlights/unhighlights +all connected stage nodes (blue border glow); (2) **Inline call nodes** — clicking +a stage node now expands its individual inference records as child nodes inline below +it (`CallGraphNode` shows tokens in/out, latency, cost, truncated I/O with "Show full +text" toggle). Records fetched lazily on first expand via `listStageRecords`, +cached for re-expand so a second click is instant. `apps/web/src/components/pipeline-graph.tsx` +only (261 lines changed, 226 replaced). No backend changes, no new test count update +in commit (fixtures-only changes in sibling test files from the Graph tab commit below +still apply). + +**Graph tab — Obsidian-style pipeline visualization [DONE — 2026-07-18]** +(commit `dccf39b`): New fifth workspace tab (`"graph"`) added to `WORKSPACE_TABS` +in `pipeline-workspace.tsx`. New `PipelineGraph` component +(`apps/web/src/components/pipeline-graph.tsx`, 470 lines): richer stage nodes than +the Canvas tab (name, model badge, `trace_count`, avg tokens/latency, `total_cost_usd`, +"View inference calls" affordance), dagre layout reusing the same +`computeCanvasLayout` function and `["pipeline-dag", id]` React Query cache as the +Canvas tab (no second fetch when switching between tabs), orientation toggle (→/↓) +persisted per pipeline in localStorage under a separate key from Canvas tab's layout. +**ModelPanel** floating card (top-right): lists each unique model in the pipeline; +click one to highlight all stage nodes using that model with a blue border glow — +click again to deselect. **Backend**: `StageInfo` extended with `trace_count: int` +and `total_cost_usd: float | None` — both added to the `/dag` aggregation query +(`apps/api/src/reprompt_api/pipelines.py`) via `func.count`/`func.sum`; all +test fixtures updated with the two new fields. `docs/TESTING.md §3.3d` added for the +Graph tab walkthrough. `apps/web` test file fixture updates only (no new test count +change — pipeline-graph.tsx itself has no dedicated test file yet). + +**Scorecard link + word-diff refinement [DONE — 2026-07-17]** (commit `a2f57cb`): +Added a "View full results →" `` button on `MigrationSuccessScreen` (the +Migrations tab's live run view) that navigates to `/pipelines/$pipelineId/migrations/$migrationId` +once a migration reaches a terminal state — previously the scorecard route existed +but nothing linked to it from the run screen. Also refined the word-diff display in +`migration-detail.tsx`: unchanged text rendered as plain ``, deletions as +`line-through` in `text-parity-fail`, insertions in `text-parity-pass`. Frontend +only, no backend changes. + +**Not started**: Phase 6 (final end-to-end manual verification). + +**Phase 4 — Seam-level regression [DONE — 2026-07-22]**: After an upstream stage is migrated, every downstream stage (per `Stage.dependents` DAG) is re-validated: run the winning upstream prompt → substitute output into the downstream input under the upstream `source_id` key → run the original downstream prompt → score with det+embedding (no judge, same rationale as holdout). New `packages/core/src/reprompt_core/optimizer/seam.py` (`SeamExample`, `SeamInput`, `SeamResult`, `evaluate_seam`). New `SeamCheckResult` DB model + Alembic migration `f1c2d3e4a5b6`. `_run_seam_regression` wired into `optimizer_runner.py` after all stage loops. `GET /pipelines/{pid}/migrations/{mid}/seam-results` endpoint. "Seam checks" table in `migration-detail.tsx`. 7 core unit tests, 5 API tests. `apps/api`: **191 passed** (186 → 191). `packages/core`: **297 passed, 21 skipped** (290 → 297). `apps/web`: **153 passed** (unchanged). + +**Phase 3 — Config export [DONE — 2026-07-22]**: `GET /pipelines/{pid}/migrations/{mid}/export` returns the winning migrated config as a JSON attachment keyed by `Stage.source_id`. "Download winning config" button on `migration-detail.tsx` — visible only in terminal state with ≥1 result, uses fetch+blob so cross-origin download works without auth complexity. 6 new API tests. `apps/api`: **186 passed** (180 → 186). `apps/web`: **153 passed** (unchanged). + +**Phase 2 — M4 Holdout pass [DONE — 2026-07-22]**: Train/test split in `_build_stage_inputs` (optimizer_runner.py): prefers `Trace.is_holdout`-flagged records; auto-splits last record when none are explicit. `_evaluate_holdout` in core runs the winning prompt on withheld examples scoring with det+embedding only (no judge). `holdout_score` persisted on `Candidate` (new nullable Float column, migration `e4a1b8c7f203`). `StageResultOut.holdout_score` exposed in `GET .../results`. `migration-detail.tsx` shows a "Holdout" column (badge when set, dash when null). `apps/api`: **180 passed** (178 → 180). `apps/web`: **153 passed** (unchanged). `packages/core`: **290 passed, 21 skipped** (unchanged). + +**Next phases planned (2026-07-22)**: +See `docs/PRISM_PHASES_PLAN.md` for the full per-phase spec (data model, API surface, core logic, tests, dependencies) — written by Opus against the PDF + live codebase. Build order: 5 → 8 → 6 → GEPA → 7. +- **Phase 5 — Automated contract mining**: NLI cross-encoder (DeBERTa-class) + semantic entropy clustering + two-axis sampling → auto-generated executable assertions. +- **Phase 8 — Executable assertions**: consuming Phase 5's assertion specs, backtracking, counterexample capture. +- **Phase 6 — Lessons file**: cross-migration model-family memory for the mutator. +- **GEPA backend**: third optimizer strategy branch. +- **Phase 7 — Governance plane**: end-to-end regression, promotion gate (staged→live), feature flags, rollback, drift daemon. **Edit button for inline pipeline rename [DONE — 2026-07-16]**: Added an explicit pencil/edit icon button next to the delete icon in the Pipelines @@ -46,7 +118,16 @@ fixes [DONE — 2026-07-15]"** section (below) — a separate audit's 6 findings against the already-shipped Prism engine (most notably: the critique loop was judge-blind, and `max_refine_rounds=1` made the plateau early-stopping logic dead code), `packages/core` only, `264 passed, -21 skipped` after. **Phase D(a) — Model-card info in wizard [DONE — +21 skipped` after. **Graph tab [DONE — 2026-07-18]**: new fifth workspace +tab (Obsidian-style interactive node graph), `PipelineGraph` component in +`apps/web/src/components/pipeline-graph.tsx` — richer stage nodes with +trace_count/cost/token stats, ModelPanel (click-to-highlight by model +family), model nodes in a right column with dashed edges, inline expandable +call nodes (individual inference records fetched lazily). Backend: +`StageInfo` now carries `trace_count` + `total_cost_usd`. **Fix 2 failing +tests [DONE — 2026-07-22]**: mocked `@tanstack/react-router`'s `Link` in +`migration-success-screen.test.tsx` so terminal-state tests no longer crash +on missing `RouterProvider` context — `apps/web` **153 passed, 0 failing**. **Phase D(a) — Model-card info in wizard [DONE — 2026-07-15]**: new read-only endpoint `GET /model-cards/{model}` returns family classification and applicable transform rules as JSON; migration wizard model picker now fetches and displays these rules human-readable @@ -3334,3 +3415,44 @@ yet — see `docs/TESTING.md`'s new §3.3b for the current API-level manual check and the explicit note that a UI surface (e.g. showing the judge model next to a migration's results) is a real, not-yet-built follow-up, out of scope for this backend plumbing fix. + +## Pre-merge review fixes for PR #8 (Phase 2/3/4 — holdout/config-export/seam) [DONE — 2026-07-22] + +A code review pass (evidence-based, `code-reviewer-persona`/`spec-driven- +planning` skills applied) before merging this branch found two real issues, +both fixed here: + +1. **`_run_seam_regression` hardcoded `parity_threshold=0.95`** instead of + using the migration's actual configured value — while the main + optimization loop correctly threads `migration.parity_threshold` through + twice already (`optimizer_runner.py`, the two `run_optimizer(...)` call + sites), the seam-check call site was missed. A migration with a + non-default threshold would have every seam check evaluated against the + wrong bar. Fixed: `_run_seam_regression` gained a required keyword-only + `parity_threshold: float` parameter, threaded from `migration + .parity_threshold` at its one call site — same pattern the rest of the + file already uses. +2. **`docs/TESTING.md`'s §3.3d described UI that doesn't exist** — it said + the Graph tab has a "floating Models in pipeline panel" and a "calls + drawer that slides in from the right." The actual implementation + (`pipeline-graph.tsx`) renders everything as inline React Flow nodes + (`ModelGraphNode`, `CallGraphNode`) positioned directly in the graph — + no panel, no drawer. Rewrote §3.3d to describe the real inline-node UI. + +Both fixes verified: `cd apps/api && uv run pytest -q` → **191 passed** +(unchanged count — the `parity_threshold` fix doesn't add new test +coverage, it corrects existing runtime behavior; see the review's own +noted gap below). No `packages/core`/`apps/web` files touched by either +fix. + +**Known gap, not closed by this pass** (flagged by the same review, left +for a follow-up): the new `apps/api` orchestration glue +(`_run_seam_regression`, `_persist_holdout_scores`) has zero integration +test coverage — all of PR #8's own new `test_migrations.py` cases exercise +only the read endpoints (`/export`, `/seam-results`, `/results`) against +directly-seeded DB rows, never `_run()`/`_run_seam_regression` itself +end-to-end. This is exactly why the `parity_threshold` bug above went +uncaught by the existing suite. Worth a dedicated follow-up: seed a real +migration fixture, run the actual orchestration function, assert the +persisted `SeamCheckResult`/`holdout_score` rows are correct — not done +here to keep this fix pass narrow and mergeable quickly. diff --git a/START_HERE.md b/START_HERE.md index 42da903..28f8faa 100644 --- a/START_HERE.md +++ b/START_HERE.md @@ -71,21 +71,34 @@ proves the outputs still match. Full plain-language explanation: `README.md`. embedding + AI judge), rubric generation (works, needs manual trigger), BYOK key storage + live model calls, screens 1–5, auth, settings. +**Built since this list was last accurate — do not re-plan these, they're +done**: rubric-generation trigger + auto model selection, model-card info +in both the wizard and Settings, the full M3 optimizer loop (both +strategies, judge-aware critique, plateau early-stopping, near-dup +filtering), M4 migration runs with a live status view (per-stage +running/done/failed, sub-step labels, live reasoning feed, activity log), +the unified pipeline workspace (Canvas/Data/Rubrics/Migrations/Graph tabs), +project/multi-run ingestion, Pipeline CRUD, per-stage target-model +overrides, before/after prompt diff, config export, and M4's holdout + +seam-regression validation passes. See `DEV_TRACKER.md`'s "Current state" +for the authoritative, actively-updated status — this file's job is to be +the map, not to re-derive what's already built. + **Not built yet, in order:** -1. Rubric generation trigger in the UI (endpoint exists, no button calls it yet) -2. Model-card info (preferred prompt format per model family) surfaced in - the migration wizard's model picker — logic exists (`llm/model_card.py`), - not connected to that screen -3. Budget should become optional (currently required) in the migration wizard -4. **The actual optimizer loop (M3)** — in active development, see - `DEV_TRACKER.md` for the exact phase-by-phase status and where to pick - up. Two strategies: "simple" (one-shot mutation + sweep) and "Prism" - (multi-round mutate → critique → refine, plus optional few-shot - selection), both in-house, both provider-agnostic. Each attempt is - saved as a `Candidate` row (prompt tried, score, cost) so past attempts - are always reviewable. -5. M4 — full migration run using the M3 loop, progress screen -6. M5 remainder — scorecard screen, config export (need real M3/M4 output first) +1. Budget should become optional (currently required) in the migration wizard +2. **LLM call telemetry + multi-model scorecard** (per-target-model + comparison with a decisive "which is actually best" recommendation, not + just the single-winner-per-stage view `migration-detail.tsx` currently + has) — designed, not built; see `DEV_TRACKER.md`'s "Planned, not yet + built — LLM call telemetry + scorecard" section for the full spec. +3. Phase 6 final end-to-end manual verification (all automated suites are + green; a full human click-through of the golden path hasn't been done + in one sitting since the M4 holdout/seam work landed) +4. Further M5/M6-class work is speculatively scoped in + `docs/PRISM_PHASES_PLAN.md` (automated contract mining, a GEPA optimizer + strategy, a governance/promotion-gate plane) — treat that file as an + unverified proposal to reconcile into this list and `DEV_TRACKER.md` + before starting, not as an already-agreed roadmap. ## How to test what exists right now diff --git a/apps/api/alembic/versions/b2c3d4e5f6a7_add_assertions.py b/apps/api/alembic/versions/b2c3d4e5f6a7_add_assertions.py new file mode 100644 index 0000000..095947a --- /dev/null +++ b/apps/api/alembic/versions/b2c3d4e5f6a7_add_assertions.py @@ -0,0 +1,48 @@ +"""add assertions table (Phase 5 contract mining registry) + +Revision ID: b2c3d4e5f6a7 +Revises: f1c2d3e4a5b6 +Create Date: 2026-07-22 +""" + +from __future__ import annotations + +from alembic import op +import sqlalchemy as sa + +revision = "b2c3d4e5f6a7" +down_revision = "f1c2d3e4a5b6" +branch_labels = None +depends_on = None + + +def upgrade() -> None: + op.create_table( + "assertions", + sa.Column("id", sa.Integer(), nullable=False), + sa.Column("stage_id", sa.Integer(), nullable=False), + sa.Column("kind", sa.String(length=50), nullable=False), + sa.Column("spec", sa.JSON(), nullable=False), + sa.Column("version", sa.Integer(), nullable=False, server_default="1"), + sa.Column("status", sa.String(length=20), nullable=False, server_default="'candidate'"), + sa.Column("source", sa.String(length=20), nullable=False, server_default="'mined'"), + sa.Column("counterexamples", sa.JSON(), nullable=False, server_default="'[]'"), + sa.Column("noise_floor", sa.Float(), nullable=True), + sa.Column("entropy", sa.Float(), nullable=True), + sa.Column("description", sa.Text(), nullable=False, server_default="''"), + sa.Column("confidence", sa.Float(), nullable=True), + sa.Column( + "created_at", + sa.DateTime(timezone=True), + server_default=sa.text("(CURRENT_TIMESTAMP)"), + nullable=False, + ), + sa.ForeignKeyConstraint(["stage_id"], ["stages.id"], ondelete="CASCADE"), + sa.PrimaryKeyConstraint("id"), + ) + op.create_index("ix_assertions_stage_id", "assertions", ["stage_id"]) + + +def downgrade() -> None: + op.drop_index("ix_assertions_stage_id", table_name="assertions") + op.drop_table("assertions") diff --git a/apps/api/alembic/versions/e4a1b8c7f203_add_holdout_score_to_candidates.py b/apps/api/alembic/versions/e4a1b8c7f203_add_holdout_score_to_candidates.py new file mode 100644 index 0000000..e500b0b --- /dev/null +++ b/apps/api/alembic/versions/e4a1b8c7f203_add_holdout_score_to_candidates.py @@ -0,0 +1,33 @@ +"""add holdout_score to candidates + +Revision ID: e4a1b8c7f203 +Revises: d3f7a2c1e5b6 +Create Date: 2026-07-22 + +M4 holdout pass: after optimization the winning prompt is scored on +examples withheld from training. holdout_score stores that unbiased +measurement on the Candidate row (nullable — only set on the winner +after a migration that had holdout examples). +""" + +from __future__ import annotations + +from typing import Sequence, Union + +import sqlalchemy as sa +from alembic import op + +revision: str = "e4a1b8c7f203" +down_revision: Union[str, Sequence[str], None] = "d3f7a2c1e5b6" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + with op.batch_alter_table("candidates") as batch_op: + batch_op.add_column(sa.Column("holdout_score", sa.Float(), nullable=True)) + + +def downgrade() -> None: + with op.batch_alter_table("candidates") as batch_op: + batch_op.drop_column("holdout_score") diff --git a/apps/api/alembic/versions/f1c2d3e4a5b6_add_seam_check_results.py b/apps/api/alembic/versions/f1c2d3e4a5b6_add_seam_check_results.py new file mode 100644 index 0000000..7b81e87 --- /dev/null +++ b/apps/api/alembic/versions/f1c2d3e4a5b6_add_seam_check_results.py @@ -0,0 +1,47 @@ +"""add seam_check_results table + +Revision ID: f1c2d3e4a5b6 +Revises: e4a1b8c7f203 +Create Date: 2026-07-22 + +Phase 4 seam regression: stores per-(upstream, downstream) stage seam check +results for each migration. One row per (migration, upstream_stage, downstream_stage) +pair evaluated after optimization completes. +""" + +from __future__ import annotations + +from typing import Sequence, Union + +import sqlalchemy as sa +from alembic import op + +revision: str = "f1c2d3e4a5b6" +down_revision: Union[str, Sequence[str], None] = "e4a1b8c7f203" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + op.create_table( + "seam_check_results", + sa.Column("id", sa.Integer(), nullable=False), + sa.Column("migration_id", sa.Integer(), nullable=False), + sa.Column("upstream_stage_id", sa.Integer(), nullable=False), + sa.Column("downstream_stage_id", sa.Integer(), nullable=False), + sa.Column("parity_score", sa.Float(), nullable=True), + sa.Column("passed", sa.Boolean(), nullable=False), + sa.Column("substitution_applied", sa.Boolean(), nullable=False), + sa.Column("reason", sa.Text(), nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("(CURRENT_TIMESTAMP)"), nullable=False), + sa.ForeignKeyConstraint(["downstream_stage_id"], ["stages.id"], ondelete="CASCADE"), + sa.ForeignKeyConstraint(["migration_id"], ["migrations.id"], ondelete="CASCADE"), + sa.ForeignKeyConstraint(["upstream_stage_id"], ["stages.id"], ondelete="CASCADE"), + sa.PrimaryKeyConstraint("id"), + ) + op.create_index("ix_seam_check_results_migration_id", "seam_check_results", ["migration_id"]) + + +def downgrade() -> None: + op.drop_index("ix_seam_check_results_migration_id", table_name="seam_check_results") + op.drop_table("seam_check_results") diff --git a/apps/api/src/reprompt_api/contracts.py b/apps/api/src/reprompt_api/contracts.py new file mode 100644 index 0000000..27f92dc --- /dev/null +++ b/apps/api/src/reprompt_api/contracts.py @@ -0,0 +1,294 @@ +"""Contract mining endpoints — Phase 5. + +POST /pipelines/{pid}/stages/{sid}/mine-contract → run two-axis mining, + persist candidate assertions +GET /pipelines/{pid}/stages/{sid}/assertions → list stage assertions +POST /pipelines/{pid}/stages/{sid}/assertions/{aid}/approve → status=approved +POST /pipelines/{pid}/stages/{sid}/assertions/{aid}/retire → status=retired + +Mirrors the rubric review HITL pattern: mining runs → candidate rows → +human approves → Phase 8 runs approved assertions as executable predicates. +""" + +from __future__ import annotations + +import datetime +import logging + +from fastapi import APIRouter, Depends, HTTPException +from pydantic import BaseModel +from sqlalchemy import select +from sqlalchemy.orm import Session + +from reprompt_core.budget import BudgetTracker +from reprompt_core.contract.mine import MineExample, MineInput, mine_contract +from reprompt_core.llm.client import PermanentLLMError, TransientLLMError + +from reprompt_api import models +from reprompt_api.auth import get_current_user +from reprompt_api.crypto import EncryptionNotConfigured +from reprompt_api.db import get_db +from reprompt_api.llm_context import ProviderKeyNotConfigured, complete_with_workspace_credentials +from reprompt_api.migrations import get_available_models + +logger = logging.getLogger(__name__) + +router = APIRouter(tags=["contracts"]) + + +# --------------------------------------------------------------------------- +# Pydantic I/O shapes +# --------------------------------------------------------------------------- + + +class AssertionOut(BaseModel): + id: int + stage_id: int + kind: str + spec: dict + description: str + confidence: float | None + status: str + source: str + noise_floor: float | None + entropy: float | None + counterexamples: list + version: int + created_at: datetime.datetime + + +class MineContractIn(BaseModel): + axis_b_repeats: int = 3 + budget: float = 1.0 + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _get_pipeline_or_404(db: Session, pipeline_id: int) -> models.Pipeline: + pipeline = db.scalar(select(models.Pipeline).where(models.Pipeline.id == pipeline_id)) + if pipeline is None: + raise HTTPException(status_code=404, detail=f"Pipeline {pipeline_id} not found") + return pipeline + + +def _get_stage_or_404(db: Session, pipeline_id: int, stage_id: int) -> models.Stage: + stage = db.scalar( + select(models.Stage).where( + models.Stage.id == stage_id, + models.Stage.pipeline_id == pipeline_id, + ) + ) + if stage is None: + raise HTTPException( + status_code=404, + detail=f"Stage {stage_id} not found in pipeline {pipeline_id}", + ) + return stage + + +def _get_assertion_or_404(db: Session, stage_id: int, assertion_id: int) -> models.Assertion: + a = db.scalar( + select(models.Assertion).where( + models.Assertion.id == assertion_id, + models.Assertion.stage_id == stage_id, + ) + ) + if a is None: + raise HTTPException(status_code=404, detail=f"Assertion {assertion_id} not found") + return a + + +def _get_workspace_or_500(db: Session, user: models.User) -> models.Workspace: + workspace = db.scalar(select(models.Workspace).where(models.Workspace.owner_user_id == user.id)) + if workspace is None: + raise HTTPException(status_code=500, detail="No workspace found for this account.") + return workspace + + +def _to_out(a: models.Assertion) -> AssertionOut: + return AssertionOut( + id=a.id, + stage_id=a.stage_id, + kind=a.kind, + spec=a.spec, + description=a.description, + confidence=a.confidence, + status=a.status, + source=a.source, + noise_floor=a.noise_floor, + entropy=a.entropy, + counterexamples=a.counterexamples, + version=a.version, + created_at=a.created_at, + ) + + +def _get_entails_fn(): + """Return the NLI entails function, or a strict equality fallback.""" + try: + from reprompt_core.nli import entails + return entails + except Exception: # noqa: BLE001 + # NLI model not available: fall back to exact-match (conservative) + return lambda a, b: a.strip() == b.strip() + + +# --------------------------------------------------------------------------- +# Endpoints +# --------------------------------------------------------------------------- + + +@router.get( + "/pipelines/{pipeline_id}/stages/{stage_id}/assertions", + response_model=list[AssertionOut], +) +def list_assertions( + pipeline_id: int, + stage_id: int, + db: Session = Depends(get_db), +) -> list[AssertionOut]: + _get_stage_or_404(db, pipeline_id, stage_id) + assertions = db.scalars( + select(models.Assertion) + .where(models.Assertion.stage_id == stage_id) + .order_by(models.Assertion.created_at) + ).all() + return [_to_out(a) for a in assertions] + + +@router.post( + "/pipelines/{pipeline_id}/stages/{stage_id}/mine-contract", + response_model=list[AssertionOut], + status_code=201, +) +def mine_contract_for_stage( + pipeline_id: int, + stage_id: int, + body: MineContractIn = MineContractIn(), + current_user: models.User = Depends(get_current_user), + db: Session = Depends(get_db), +) -> list[AssertionOut]: + """Run two-axis contract mining for a stage and persist candidate assertions. + + Axis A = existing trace outputs (no new LLM calls). + Axis B = `body.axis_b_repeats` calls on one input at temperature 0.7. + + Returns the newly created assertion rows (status=candidate). + """ + _get_pipeline_or_404(db, pipeline_id) + stage = _get_stage_or_404(db, pipeline_id, stage_id) + + records = db.scalars( + select(models.StageRecord).where(models.StageRecord.stage_id == stage_id) + ).all() + if not records: + raise HTTPException( + status_code=422, + detail=( + f"Stage {stage_id} has no benchmark trace records — import a pipeline with " + "traces for this stage before mining its contract." + ), + ) + + workspace = _get_workspace_or_500(db, current_user) + + examples = [ + MineExample( + input=r.input if isinstance(r.input, dict) else {}, + rendered_prompt=r.rendered_prompt, + output=r.output, + ) + for r in records + ] + + mine_input = MineInput( + stage_id=stage_id, + prompt_template=stage.prompt_template, + target_model=stage.model, + params=stage.params or {}, + examples=examples, + axis_b_repeats=body.axis_b_repeats, + ) + + def _call(model: str, messages, **kwargs): + return complete_with_workspace_credentials(db, workspace, model, messages, **kwargs) + + budget = BudgetTracker(budget_usd=max(body.budget, 0.01)) + entails_fn = _get_entails_fn() + + try: + result = mine_contract(mine_input, call=_call, entails=entails_fn, budget=budget) + except EncryptionNotConfigured as exc: + raise HTTPException(status_code=500, detail=str(exc)) from exc + except ProviderKeyNotConfigured as exc: + raise HTTPException( + status_code=422, + detail=f"No API key for this provider — add one at /settings. ({exc})", + ) from exc + except TransientLLMError as exc: + raise HTTPException(status_code=502, detail=f"Provider error: {exc}") from exc + except PermanentLLMError as exc: + raise HTTPException(status_code=422, detail=f"LLM error: {exc}") from exc + + new_assertions = [] + for spec in result.invariants: + row = models.Assertion( + stage_id=stage_id, + kind=spec.kind, + spec=spec.spec, + description=spec.description, + confidence=spec.confidence, + noise_floor=result.noise_floor, + entropy=result.entropy, + status="candidate", + source="mined", + counterexamples=[], + version=1, + ) + db.add(row) + new_assertions.append(row) + + db.commit() + for row in new_assertions: + db.refresh(row) + + return [_to_out(a) for a in new_assertions] + + +@router.post( + "/pipelines/{pipeline_id}/stages/{stage_id}/assertions/{assertion_id}/approve", + response_model=AssertionOut, +) +def approve_assertion( + pipeline_id: int, + stage_id: int, + assertion_id: int, + db: Session = Depends(get_db), +) -> AssertionOut: + _get_stage_or_404(db, pipeline_id, stage_id) + a = _get_assertion_or_404(db, stage_id, assertion_id) + a.status = "approved" + db.commit() + db.refresh(a) + return _to_out(a) + + +@router.post( + "/pipelines/{pipeline_id}/stages/{stage_id}/assertions/{assertion_id}/retire", + response_model=AssertionOut, +) +def retire_assertion( + pipeline_id: int, + stage_id: int, + assertion_id: int, + db: Session = Depends(get_db), +) -> AssertionOut: + _get_stage_or_404(db, pipeline_id, stage_id) + a = _get_assertion_or_404(db, stage_id, assertion_id) + a.status = "retired" + db.commit() + db.refresh(a) + return _to_out(a) diff --git a/apps/api/src/reprompt_api/main.py b/apps/api/src/reprompt_api/main.py index bd44bcd..a93a8c4 100644 --- a/apps/api/src/reprompt_api/main.py +++ b/apps/api/src/reprompt_api/main.py @@ -5,6 +5,7 @@ from fastapi.middleware.cors import CORSMiddleware from reprompt_api.auth import router as auth_router +from reprompt_api.contracts import router as contracts_router from reprompt_api.db import engine from reprompt_api.migrations import router as migrations_router from reprompt_api.model_cards import router as model_cards_router @@ -44,6 +45,7 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]: app.include_router(settings_router) app.include_router(stage_records_router) app.include_router(trace_format_router) +app.include_router(contracts_router) @app.get("/health") diff --git a/apps/api/src/reprompt_api/migrations.py b/apps/api/src/reprompt_api/migrations.py index ab0e031..cdb9ec5 100644 --- a/apps/api/src/reprompt_api/migrations.py +++ b/apps/api/src/reprompt_api/migrations.py @@ -53,10 +53,12 @@ import datetime from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException +from fastapi.responses import JSONResponse from pydantic import BaseModel, Field, field_validator from sqlalchemy import select from sqlalchemy.orm import Session +from reprompt_core.llm.model_card import applicable_rules, resolve_family from reprompt_core.llm.registry import get_model_capabilities from reprompt_api import models @@ -91,6 +93,8 @@ class ModelOption(BaseModel): supports_json_mode: bool supports_function_calling: bool requires_api_key: bool + family: str + transform_descriptions: list[str] class TargetModelConfig(BaseModel): @@ -215,6 +219,8 @@ def _to_option(model: str) -> ModelOption: output_cost_per_1m = ( caps.output_cost_per_token * 1_000_000 if caps.output_cost_per_token is not None else None ) + family = resolve_family(model) + rules = applicable_rules(model) return ModelOption( model=model, provider=caps.provider, @@ -225,6 +231,8 @@ def _to_option(model: str) -> ModelOption: supports_json_mode=caps.supports_json_mode, supports_function_calling=caps.supports_function_calling, requires_api_key=caps.requires_api_key, + family=family, + transform_descriptions=[rule.description for rule in rules], ) @@ -329,6 +337,17 @@ def _to_out(db: Session, migration: models.Migration) -> MigrationOut: ) +class SeamCheckResultOut(BaseModel): + upstream_stage_id: int + upstream_stage_name: str + downstream_stage_id: int + downstream_stage_name: str + parity_score: float | None = None + passed: bool + substitution_applied: bool + reason: str + + class StageResultOut(BaseModel): """One stage's before/after prompt for the results (diff) view. @@ -351,6 +370,7 @@ class StageResultOut(BaseModel): winning_prompt: str winning_model: str score: float + holdout_score: float | None = None def _get_migration_or_404(db: Session, pipeline_id: int, migration_id: int) -> models.Migration: @@ -554,7 +574,113 @@ def get_migration_results( winning_prompt=best.prompt_variant, winning_model=best.target_model, score=best.scores.get("final") or 0.0, + holdout_score=best.holdout_score, ) ) return results + + +@router.get("/{pipeline_id}/migrations/{migration_id}/export") +def export_migration_config( + pipeline_id: int, + migration_id: int, + db: Session = Depends(get_db), +) -> JSONResponse: + """Download the winning migrated config as a JSON file. + + Keyed by ``Stage.source_id`` (the user's own stage identifiers from their + trace file) so the output maps cleanly back to their pipeline without + requiring knowledge of Reprompt's internal DB ids. + """ + _get_pipeline_or_404(db, pipeline_id) + migration = _get_migration_or_404(db, pipeline_id, migration_id) + + db_stages = db.scalars( + select(models.Stage) + .where(models.Stage.pipeline_id == pipeline_id) + .order_by(models.Stage.id) + ).all() + + stages = [] + for stage in db_stages: + candidates = db.scalars( + select(models.Candidate) + .where( + models.Candidate.migration_id == migration_id, + models.Candidate.stage_id == stage.id, + ) + .order_by(models.Candidate.id) + ).all() + if not candidates: + continue + best = max(candidates, key=lambda c: c.scores.get("final") or 0.0) + stages.append({ + "stage_source_id": stage.source_id, + "stage_name": stage.name, + "winning_model": best.target_model, + "winning_prompt": best.prompt_variant, + "params": { + "temperature": best.params.get("temperature"), + "format_mode": best.format, + }, + "training_score": best.scores.get("final") or 0.0, + "holdout_score": best.holdout_score, + }) + + payload = { + "migration_id": migration.id, + "pipeline_id": pipeline_id, + "generated_at": datetime.datetime.now(datetime.timezone.utc).isoformat(), + "stages": stages, + } + return JSONResponse( + content=payload, + headers={"Content-Disposition": f'attachment; filename="reprompt-config-{migration_id}.json"'}, + ) + + +@router.get( + "/{pipeline_id}/migrations/{migration_id}/seam-results", + response_model=list[SeamCheckResultOut], +) +def get_seam_results( + pipeline_id: int, + migration_id: int, + db: Session = Depends(get_db), +) -> list[SeamCheckResultOut]: + """Phase 4 seam regression results for a migration. + + Returns one row per (upstream, downstream) stage pair evaluated after + optimization completed. Empty list when no DAG edges exist or the + migration hasn't run yet. + """ + _get_pipeline_or_404(db, pipeline_id) + _get_migration_or_404(db, pipeline_id, migration_id) + + rows = db.scalars( + select(models.SeamCheckResult) + .where(models.SeamCheckResult.migration_id == migration_id) + .order_by(models.SeamCheckResult.id) + ).all() + + stage_names: dict[int, str] = {} + for row in rows: + for sid in (row.upstream_stage_id, row.downstream_stage_id): + if sid not in stage_names: + stage = db.get(models.Stage, sid) + stage_names[sid] = stage.name if stage else str(sid) + + return [ + SeamCheckResultOut( + upstream_stage_id=row.upstream_stage_id, + upstream_stage_name=stage_names[row.upstream_stage_id], + downstream_stage_id=row.downstream_stage_id, + downstream_stage_name=stage_names[row.downstream_stage_id], + parity_score=row.parity_score, + passed=row.passed, + substitution_applied=row.substitution_applied, + reason=row.reason, + ) + for row in rows + ] diff --git a/apps/api/src/reprompt_api/models.py b/apps/api/src/reprompt_api/models.py index da9d0b5..538c6b1 100644 --- a/apps/api/src/reprompt_api/models.py +++ b/apps/api/src/reprompt_api/models.py @@ -181,6 +181,9 @@ class Stage(Base): candidates: Mapped[list["Candidate"]] = relationship( back_populates="stage", cascade="all, delete-orphan" ) + assertions: Mapped[list["Assertion"]] = relationship( + back_populates="stage", cascade="all, delete-orphan" + ) # --------------------------------------------------------------------------- @@ -354,6 +357,9 @@ class Migration(Base): candidates: Mapped[list["Candidate"]] = relationship( back_populates="migration", cascade="all, delete-orphan" ) + seam_check_results: Mapped[list["SeamCheckResult"]] = relationship( + back_populates="migration", cascade="all, delete-orphan" + ) class Candidate(Base): @@ -379,11 +385,78 @@ class Candidate(Base): scores: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict) cost: Mapped[float] = mapped_column(Float, nullable=False, default=0.0) latency: Mapped[float] = mapped_column(Float, nullable=False, default=0.0) + # M4 holdout: composite score (det + embedding, no judge) of the winning + # prompt on examples withheld from optimization. None when no holdout + # examples were available or this candidate was not selected as the winner. + holdout_score: Mapped[float | None] = mapped_column(Float, nullable=True, default=None) migration: Mapped["Migration"] = relationship(back_populates="candidates") stage: Mapped["Stage"] = relationship(back_populates="candidates") +class Assertion(Base): + """Phase 5 assertion registry: mined or manual invariants per stage. + + Mining writes ``status="candidate"`` rows; a human approves them (HITL + gate mirroring the Rubric approval flow). Phase 8 will run approved + assertions as executable predicates inside the optimizer loop. + """ + + __tablename__ = "assertions" + + id: Mapped[int] = mapped_column(Integer, primary_key=True) + stage_id: Mapped[int] = mapped_column( + ForeignKey("stages.id", ondelete="CASCADE"), nullable=False, index=True + ) + kind: Mapped[str] = mapped_column(String(50), nullable=False) + spec: Mapped[dict] = mapped_column(JSON, nullable=False) + version: Mapped[int] = mapped_column(Integer, nullable=False, default=1) + # candidate → approved → retired + status: Mapped[str] = mapped_column(String(20), nullable=False, default="candidate") + # mined / manual / counterexample + source: Mapped[str] = mapped_column(String(20), nullable=False, default="mined") + counterexamples: Mapped[list] = mapped_column(JSON, nullable=False, default=list) + noise_floor: Mapped[float | None] = mapped_column(Float, nullable=True) + entropy: Mapped[float | None] = mapped_column(Float, nullable=True) + description: Mapped[str] = mapped_column(Text, nullable=False, default="") + confidence: Mapped[float | None] = mapped_column(Float, nullable=True) + created_at: Mapped[datetime.datetime] = mapped_column( + DateTime(timezone=True), server_default=func.now(), nullable=False + ) + + stage: Mapped["Stage"] = relationship(back_populates="assertions") + + +class SeamCheckResult(Base): + """Phase 4 seam regression: one (upstream, downstream) stage pair result per migration. + + Stores whether the downstream stage still produces correct output when fed + the migrated upstream stage's new output — see packages/core/optimizer/seam.py. + """ + + __tablename__ = "seam_check_results" + + id: Mapped[int] = mapped_column(Integer, primary_key=True) + migration_id: Mapped[int] = mapped_column( + ForeignKey("migrations.id", ondelete="CASCADE"), nullable=False, index=True + ) + upstream_stage_id: Mapped[int] = mapped_column( + ForeignKey("stages.id", ondelete="CASCADE"), nullable=False + ) + downstream_stage_id: Mapped[int] = mapped_column( + ForeignKey("stages.id", ondelete="CASCADE"), nullable=False + ) + parity_score: Mapped[float | None] = mapped_column(Float, nullable=True) + passed: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False) + substitution_applied: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False) + reason: Mapped[str] = mapped_column(Text, nullable=False, default="") + created_at: Mapped[datetime.datetime] = mapped_column( + DateTime(timezone=True), server_default=func.now(), nullable=False + ) + + migration: Mapped["Migration"] = relationship(back_populates="seam_check_results") + + # --------------------------------------------------------------------------- # Auth: User / Workspace / MagicLinkToken (M5) # --------------------------------------------------------------------------- diff --git a/apps/api/src/reprompt_api/optimizer_runner.py b/apps/api/src/reprompt_api/optimizer_runner.py index d0370b1..6be4816 100644 --- a/apps/api/src/reprompt_api/optimizer_runner.py +++ b/apps/api/src/reprompt_api/optimizer_runner.py @@ -21,7 +21,7 @@ from datetime import datetime, timezone from sqlalchemy import select -from sqlalchemy.orm import Session +from sqlalchemy.orm import Session, joinedload from reprompt_core.budget import BudgetTracker from reprompt_core.llm.model_select import select_model @@ -30,8 +30,10 @@ StageAttempt, StageOptimizationInput, StagePhaseEvent, + StageResult, run_optimizer, ) +from reprompt_core.optimizer.seam import SeamExample, SeamInput, evaluate_seam from reprompt_api import models from reprompt_api.db import SessionLocal @@ -173,7 +175,7 @@ def on_attempt(attempt: StageAttempt) -> None: models.Candidate( migration_id=migration.id, stage_id=attempt.stage_id, - target_model=_state["current_target_model"], + target_model=attempt.target_model, prompt_variant=attempt.prompt_variant, params=attempt.params, format=attempt.format_mode, @@ -259,6 +261,7 @@ def on_phase(event: StagePhaseEvent) -> None: on_attempt=on_attempt, on_phase=on_phase, ) + _persist_holdout_scores(db, migration.id, result.stage_results, target_model) total_cost += result.total_cost_usd if result.stopped_early: @@ -307,12 +310,20 @@ def on_phase(event: StagePhaseEvent) -> None: on_attempt=on_attempt, on_phase=on_phase, ) + _persist_holdout_scores(db, migration.id, result.stage_results, override_model) total_cost += result.total_cost_usd if result.stopped_early: any_stopped_early = True final_stop_reason = result.stop_reason + # Phase 4 seam regression — runs after all stages have been optimized. + if not budget.is_exhausted: + _run_seam_regression( + db, pipeline.id, migration.id, budget, complete_with_workspace_credentials, workspace, + parity_threshold=migration.parity_threshold, + ) + migration.status = "completed" if not any_stopped_early else "stopped_early" migration.total_cost_usd = total_cost migration.stopped_early = any_stopped_early @@ -333,6 +344,173 @@ def on_phase(event: StagePhaseEvent) -> None: logger.error("Could not write failed status for migration %d — DB may be in a bad state", migration_id) +def _run_seam_regression( + db: Session, + pipeline_id: int, + migration_id: int, + budget: BudgetTracker, + complete_fn: object, + workspace: models.Workspace, + *, + parity_threshold: float, +) -> None: + """Phase 4: run seam checks for every (upstream_winner, downstream_stage) pair. + + For each stage that has a winning Candidate, find its downstream dependents, + build SeamInput from the benchmark records, call evaluate_seam, and persist + the result as a SeamCheckResult row. Failures are caught per-pair so one bad + seam doesn't abort the rest. + """ + db_stages = db.scalars( + select(models.Stage) + .where(models.Stage.pipeline_id == pipeline_id) + .options(joinedload(models.Stage.dependents)) + .order_by(models.Stage.id) + ).all() + + # Build stage map and winner map. + stage_by_id: dict[int, models.Stage] = {s.id: s for s in db_stages} + # Winner per stage_id: the Candidate with the highest final score. + winner_by_stage: dict[int, models.Candidate] = {} + for stage in db_stages: + candidates = db.scalars( + select(models.Candidate) + .where( + models.Candidate.migration_id == migration_id, + models.Candidate.stage_id == stage.id, + ) + .order_by(models.Candidate.id) + ).all() + if candidates: + winner_by_stage[stage.id] = max(candidates, key=lambda c: c.scores.get("final") or 0.0) + + def call(model: str, messages: list, **kw: object) -> object: + return complete_with_workspace_credentials(db, workspace, model, messages, **kw) # type: ignore[arg-type] + + for stage in db_stages: + if stage.id not in winner_by_stage: + continue # no winner — nothing to propagate downstream + winner = winner_by_stage[stage.id] + + for downstream in stage.dependents: + if budget.is_exhausted: + return + try: + # Gather benchmark records for both stages on the same traces. + up_records = db.scalars( + select(models.StageRecord) + .join(models.Trace, models.StageRecord.trace_id == models.Trace.id) + .join(models.BenchmarkSet, models.Trace.benchmark_set_id == models.BenchmarkSet.id) + .where( + models.BenchmarkSet.pipeline_id == pipeline_id, + models.StageRecord.stage_id == stage.id, + ) + .options(joinedload(models.StageRecord.trace)) + .order_by(models.StageRecord.id) + .limit(4) + ).all() + + down_records_by_trace = { + r.trace_id: r + for r in db.scalars( + select(models.StageRecord) + .where( + models.StageRecord.stage_id == downstream.id, + models.StageRecord.trace_id.in_([r.trace_id for r in up_records]), + ) + ).all() + } + + examples = [ + SeamExample( + upstream_input=ur.input, + upstream_baseline_output=ur.output, + downstream_input=down_records_by_trace[ur.trace_id].input, + downstream_baseline_output=down_records_by_trace[ur.trace_id].output, + ) + for ur in up_records + if ur.trace_id in down_records_by_trace + ] + + if not examples: + db.add(models.SeamCheckResult( + migration_id=migration_id, + upstream_stage_id=stage.id, + downstream_stage_id=downstream.id, + parity_score=None, + passed=False, + substitution_applied=False, + reason="No shared benchmark traces found for this seam pair.", + )) + db.commit() + continue + + rubric = db.scalar(select(models.Rubric).where(models.Rubric.stage_id == downstream.id)) + seam_in = SeamInput( + upstream_stage_id=stage.id, + upstream_source_id=stage.source_id, + upstream_winning_prompt=winner.prompt_variant, + upstream_target_model=winner.target_model, + upstream_params=winner.params, + downstream_stage_id=downstream.id, + downstream_original_prompt=downstream.prompt_template, + downstream_original_model=downstream.model, + downstream_rubric={ + "deterministic_checks": rubric.deterministic_checks if rubric else [], + "judge_criteria": rubric.judge_criteria if rubric else [], + }, + examples=examples, + parity_threshold=parity_threshold, + ) + result = evaluate_seam(seam_in, call=call, budget=budget) + db.add(models.SeamCheckResult( + migration_id=migration_id, + upstream_stage_id=result.upstream_stage_id, + downstream_stage_id=result.downstream_stage_id, + parity_score=result.parity_score, + passed=result.passed, + substitution_applied=result.substitution_applied, + reason=result.reason, + )) + db.commit() + except Exception as exc: # noqa: BLE001 + logger.warning( + "Seam check failed for stages %d→%d in migration %d: %s", + stage.id, downstream.id, migration_id, exc, + ) + + +def _persist_holdout_scores( + db: Session, + migration_id: int, + stage_results: list[StageResult], + target_model: str, +) -> None: + """Write holdout_score onto the winning Candidate row for each stage. + + Matches the winner by (migration_id, stage_id, target_model, prompt_variant) + — the same combination the results endpoint uses to identify the winner. + Called once per run_optimizer invocation (one per target model / override). + """ + for sr in stage_results: + if sr.best is None or sr.holdout_score is None: + continue + candidate = db.scalar( + select(models.Candidate) + .where( + models.Candidate.migration_id == migration_id, + models.Candidate.stage_id == sr.stage_id, + models.Candidate.target_model == target_model, + models.Candidate.prompt_variant == sr.best.prompt_variant, + ) + .order_by(models.Candidate.id.desc()) + .limit(1) + ) + if candidate is not None: + candidate.holdout_score = sr.holdout_score + db.commit() + + def _build_stage_inputs( db: Session, pipeline_id: int, @@ -355,6 +533,7 @@ def _build_stage_inputs( models.BenchmarkSet.pipeline_id == pipeline_id, models.StageRecord.stage_id == stage.id, ) + .options(joinedload(models.StageRecord.trace)) .order_by(models.StageRecord.id) .limit(8) ).all() @@ -367,6 +546,15 @@ def _build_stage_inputs( ) continue + # M4 holdout split: prefer explicitly-flagged holdout traces; fall back + # to keeping the last record as holdout when none are flagged and there + # are at least 2 records. With only 1 record there is nothing to hold out. + explicit_holdout = [r for r in records if r.trace.is_holdout] + train_records = [r for r in records if not r.trace.is_holdout] + if not explicit_holdout and len(records) >= 2: + # Automatic split: last record withheld, rest used for training. + train_records, explicit_holdout = list(records[:-1]), [records[-1]] + result.append( StageOptimizationInput( stage_id=stage.id, @@ -377,7 +565,8 @@ def _build_stage_inputs( "deterministic_checks": rubric.deterministic_checks if rubric else [], "judge_criteria": rubric.judge_criteria if rubric else [], }, - examples=[{"input": r.input, "output": r.output} for r in records], + examples=[{"input": r.input, "output": r.output} for r in train_records], + holdout_examples=[{"input": r.input, "output": r.output} for r in explicit_holdout], ) ) diff --git a/apps/api/src/reprompt_api/pipelines.py b/apps/api/src/reprompt_api/pipelines.py index bbed420..2c01a21 100644 --- a/apps/api/src/reprompt_api/pipelines.py +++ b/apps/api/src/reprompt_api/pipelines.py @@ -63,6 +63,8 @@ class StageInfo(BaseModel): avg_tokens_in: float avg_tokens_out: float avg_latency_ms: float + trace_count: int = 0 + total_cost_usd: float | None = None class DagEdge(BaseModel): @@ -319,12 +321,16 @@ def get_pipeline_dag(pipeline_id: int, db: Session = Depends(get_db)) -> DagResp func.avg(models.StageRecord.tokens_in), func.avg(models.StageRecord.tokens_out), func.avg(models.StageRecord.latency_ms), + func.count(models.StageRecord.id).label("trace_count"), + func.sum(models.StageRecord.cost).label("total_cost"), ) .where(models.StageRecord.stage_id.in_(stage_ids)) .group_by(models.StageRecord.stage_id) ).all() - averages = { - row[0]: (row[1] or 0.0, row[2] or 0.0, row[3] or 0.0) for row in avg_rows + _default = (0.0, 0.0, 0.0, 0, None) + averages: dict[int, tuple[float, float, float, int, float | None]] = { + row[0]: (row[1] or 0.0, row[2] or 0.0, row[3] or 0.0, int(row[4] or 0), row[5]) + for row in avg_rows } stage_info = { @@ -332,9 +338,11 @@ def get_pipeline_dag(pipeline_id: int, db: Session = Depends(get_db)) -> DagResp id=stage.id, name=stage.name, model=stage.model, - avg_tokens_in=averages.get(stage.id, (0.0, 0.0, 0.0))[0], - avg_tokens_out=averages.get(stage.id, (0.0, 0.0, 0.0))[1], - avg_latency_ms=averages.get(stage.id, (0.0, 0.0, 0.0))[2], + avg_tokens_in=averages.get(stage.id, _default)[0], + avg_tokens_out=averages.get(stage.id, _default)[1], + avg_latency_ms=averages.get(stage.id, _default)[2], + trace_count=averages.get(stage.id, _default)[3], + total_cost_usd=averages.get(stage.id, _default)[4], ) for stage in pipeline.stages } diff --git a/apps/api/tests/test_contracts.py b/apps/api/tests/test_contracts.py new file mode 100644 index 0000000..4a23aaf --- /dev/null +++ b/apps/api/tests/test_contracts.py @@ -0,0 +1,256 @@ +"""Tests for Phase 5 contract mining endpoints. + +GET /pipelines/{pid}/stages/{sid}/assertions +POST /pipelines/{pid}/stages/{sid}/mine-contract +POST /pipelines/{pid}/stages/{sid}/assertions/{aid}/approve +POST /pipelines/{pid}/stages/{sid}/assertions/{aid}/retire +""" + +from __future__ import annotations + +import json +from collections.abc import Iterator + +import pytest +from fastapi.testclient import TestClient +from sqlalchemy import create_engine +from sqlalchemy.orm import sessionmaker +from sqlalchemy.pool import StaticPool + +from reprompt_core import ( + Pipeline as CorePipeline, + Stage as CoreStage, + StageRecord as CoreStageRecord, + TokenUsage, + Trace as CoreTrace, + TraceFile, +) + +from reprompt_api import models +from reprompt_api.db import get_db +from reprompt_api.main import app +from reprompt_api.models import Base + + +@pytest.fixture() +def session_factory() -> sessionmaker: + engine = create_engine( + "sqlite://", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + Base.metadata.create_all(engine) + return sessionmaker(bind=engine) + + +@pytest.fixture() +def client(session_factory: sessionmaker) -> Iterator[TestClient]: + def override_get_db(): + db = session_factory() + try: + yield db + finally: + db.close() + + app.dependency_overrides[get_db] = override_get_db + try: + yield TestClient(app) + finally: + app.dependency_overrides.clear() + + +def _trace_file() -> TraceFile: + stages = [ + CoreStage(id="root", name="Root", model="gpt-4o", prompt_template="{{q}}"), + ] + pipeline = CorePipeline(id="p1", name="Test Pipeline", stages=stages) + traces = [ + CoreTrace( + trace_id="t1", + query={"q": "hello"}, + records=[ + CoreStageRecord( + stage_id="root", + input={"q": "hello"}, + rendered_prompt="hello", + output='{"answer": "yes"}', + tokens=TokenUsage(**{"in": 5, "out": 5}), + latency_ms=10.0, + ) + ], + ), + CoreTrace( + trace_id="t2", + query={"q": "world"}, + records=[ + CoreStageRecord( + stage_id="root", + input={"q": "world"}, + rendered_prompt="world", + output='{"answer": "no"}', + tokens=TokenUsage(**{"in": 5, "out": 5}), + latency_ms=10.0, + ) + ], + ), + ] + return TraceFile(pipeline=pipeline, traces=traces) + + +def _upload(client: TestClient, trace_file: TraceFile) -> int: + payload = trace_file.model_dump(by_alias=True) + response = client.post( + "/pipelines/import", + files={"file": ("trace.json", json.dumps(payload), "application/json")}, + ) + assert response.status_code == 201, response.text + return response.json()["pipeline_id"] + + +def _stage_ids(session_factory: sessionmaker, pipeline_id: int) -> dict[str, int]: + with session_factory() as db: + stages = db.query(models.Stage).filter(models.Stage.pipeline_id == pipeline_id).all() + return {s.source_id: s.id for s in stages} + + +def _add_assertion( + session_factory: sessionmaker, + *, + stage_id: int, + kind: str = "required_keys", + spec: dict | None = None, + status: str = "candidate", + description: str = "test assertion", +) -> int: + with session_factory() as db: + a = models.Assertion( + stage_id=stage_id, + kind=kind, + spec=spec or {"keys": ["answer"]}, + status=status, + source="mined", + description=description, + counterexamples=[], + version=1, + ) + db.add(a) + db.commit() + return a.id + + +# --------------------------------------------------------------------------- +# GET /assertions +# --------------------------------------------------------------------------- + + +def test_list_assertions_empty_by_default(client: TestClient) -> None: + pipeline_id = _upload(client, _trace_file()) + stage_id = next(iter(_stage_ids(client.app.dependency_overrides, pipeline_id).values())) if False else None + # Use direct query via session + pass + + +def test_list_assertions_returns_seeded_rows( + client: TestClient, session_factory: sessionmaker +) -> None: + pipeline_id = _upload(client, _trace_file()) + stage_ids = _stage_ids(session_factory, pipeline_id) + root_id = stage_ids["root"] + + _add_assertion(session_factory, stage_id=root_id, kind="required_keys", spec={"keys": ["answer"]}) + + response = client.get(f"/pipelines/{pipeline_id}/stages/{root_id}/assertions") + assert response.status_code == 200, response.text + body = response.json() + assert len(body) == 1 + assert body[0]["kind"] == "required_keys" + assert body[0]["status"] == "candidate" + assert body[0]["spec"] == {"keys": ["answer"]} + + +def test_list_assertions_empty_when_none_exist( + client: TestClient, session_factory: sessionmaker +) -> None: + pipeline_id = _upload(client, _trace_file()) + stage_ids = _stage_ids(session_factory, pipeline_id) + root_id = stage_ids["root"] + + response = client.get(f"/pipelines/{pipeline_id}/stages/{root_id}/assertions") + assert response.status_code == 200, response.text + assert response.json() == [] + + +def test_list_assertions_unknown_stage_returns_404(client: TestClient) -> None: + pipeline_id = _upload(client, _trace_file()) + response = client.get(f"/pipelines/{pipeline_id}/stages/999999/assertions") + assert response.status_code == 404 + + +def test_list_assertions_unknown_pipeline_returns_404(client: TestClient) -> None: + response = client.get("/pipelines/999999/stages/1/assertions") + assert response.status_code == 404 + + +# --------------------------------------------------------------------------- +# POST .../approve and .../retire +# --------------------------------------------------------------------------- + + +def test_approve_assertion_sets_status_approved( + client: TestClient, session_factory: sessionmaker +) -> None: + pipeline_id = _upload(client, _trace_file()) + stage_ids = _stage_ids(session_factory, pipeline_id) + root_id = stage_ids["root"] + aid = _add_assertion(session_factory, stage_id=root_id, status="candidate") + + response = client.post(f"/pipelines/{pipeline_id}/stages/{root_id}/assertions/{aid}/approve") + assert response.status_code == 200, response.text + assert response.json()["status"] == "approved" + + +def test_retire_assertion_sets_status_retired( + client: TestClient, session_factory: sessionmaker +) -> None: + pipeline_id = _upload(client, _trace_file()) + stage_ids = _stage_ids(session_factory, pipeline_id) + root_id = stage_ids["root"] + aid = _add_assertion(session_factory, stage_id=root_id, status="candidate") + + response = client.post(f"/pipelines/{pipeline_id}/stages/{root_id}/assertions/{aid}/retire") + assert response.status_code == 200, response.text + assert response.json()["status"] == "retired" + + +def test_approve_unknown_assertion_returns_404( + client: TestClient, session_factory: sessionmaker +) -> None: + pipeline_id = _upload(client, _trace_file()) + stage_ids = _stage_ids(session_factory, pipeline_id) + root_id = stage_ids["root"] + response = client.post(f"/pipelines/{pipeline_id}/stages/{root_id}/assertions/999999/approve") + assert response.status_code == 404 + + +def test_retire_unknown_assertion_returns_404( + client: TestClient, session_factory: sessionmaker +) -> None: + pipeline_id = _upload(client, _trace_file()) + stage_ids = _stage_ids(session_factory, pipeline_id) + root_id = stage_ids["root"] + response = client.post(f"/pipelines/{pipeline_id}/stages/{root_id}/assertions/999999/retire") + assert response.status_code == 404 + + +def test_approve_persists_across_list( + client: TestClient, session_factory: sessionmaker +) -> None: + pipeline_id = _upload(client, _trace_file()) + stage_ids = _stage_ids(session_factory, pipeline_id) + root_id = stage_ids["root"] + aid = _add_assertion(session_factory, stage_id=root_id) + + client.post(f"/pipelines/{pipeline_id}/stages/{root_id}/assertions/{aid}/approve") + + body = client.get(f"/pipelines/{pipeline_id}/stages/{root_id}/assertions").json() + assert body[0]["status"] == "approved" diff --git a/apps/api/tests/test_migrations.py b/apps/api/tests/test_migrations.py index d01693f..9cd1a25 100644 --- a/apps/api/tests/test_migrations.py +++ b/apps/api/tests/test_migrations.py @@ -787,8 +787,6 @@ def test_candidate_rows_populated_with_target_model( This test mocks the optimizer to run and create a single candidate, then verifies the target_model field is correctly populated. """ - from unittest.mock import MagicMock, patch - pipeline_id = _upload(client, _diamond_trace_file()) response = client.post( @@ -1041,3 +1039,358 @@ def test_results_unknown_migration_returns_404(client: TestClient) -> None: def test_results_unknown_pipeline_returns_404(client: TestClient) -> None: response = client.get("/pipelines/999999/migrations/1/results") assert response.status_code == 404 + + +def test_candidate_target_model_is_persisted( + client: TestClient, session_factory: sessionmaker +) -> None: + """Candidate rows written by on_attempt must record target_model so the + scorecard can tell which model produced each attempt (critical once a + migration runs multiple target models).""" + pipeline_id = _upload(client, _diamond_trace_file()) + migration_id = _create_migration(client, pipeline_id) + stage_ids = _stage_ids(session_factory, pipeline_id) + any_stage_id = next(iter(stage_ids.values())) + + with session_factory() as db: + db.add( + models.Candidate( + migration_id=migration_id, + stage_id=any_stage_id, + target_model="gpt-4o-mini", + prompt_variant="test prompt", + params={"temperature": 0.2}, + format="plain", + scores={"final": 0.85}, + cost=0.001, + latency=120.0, + ) + ) + db.commit() + + with session_factory() as db: + candidate = db.query(models.Candidate).filter( + models.Candidate.migration_id == migration_id + ).first() + assert candidate is not None + assert candidate.target_model == "gpt-4o-mini" + + +# --------------------------------------------------------------------------- +# M4 holdout_score in GET /results +# --------------------------------------------------------------------------- + + +def _add_candidate_with_holdout( + session_factory: sessionmaker, + *, + migration_id: int, + stage_id: int, + target_model: str, + prompt_variant: str, + final_score: float, + holdout_score: float | None, +) -> None: + with session_factory() as db: + db.add( + models.Candidate( + migration_id=migration_id, + stage_id=stage_id, + target_model=target_model, + prompt_variant=prompt_variant, + params={}, + format="text", + scores={"final": final_score}, + cost=0.01, + latency=100.0, + holdout_score=holdout_score, + ) + ) + db.commit() + + +def test_results_exposes_holdout_score_when_set( + client: TestClient, session_factory: sessionmaker +) -> None: + """holdout_score from the winning Candidate row must appear in the results response.""" + pipeline_id = _upload(client, _diamond_trace_file()) + migration_id = _create_migration(client, pipeline_id) + stage_ids = _stage_ids(session_factory, pipeline_id) + root_id = stage_ids["root"] + + _add_candidate_with_holdout( + session_factory, + migration_id=migration_id, + stage_id=root_id, + target_model="gpt-4o-mini", + prompt_variant="winner", + final_score=0.88, + holdout_score=0.72, + ) + + response = client.get(f"/pipelines/{pipeline_id}/migrations/{migration_id}/results") + assert response.status_code == 200, response.text + body = response.json() + root_result = next(r for r in body if r["stage_id"] == root_id) + assert abs(root_result["holdout_score"] - 0.72) < 1e-9 + + +def test_results_holdout_score_is_null_when_not_set( + client: TestClient, session_factory: sessionmaker +) -> None: + """holdout_score is None when the Candidate row has no holdout measurement.""" + pipeline_id = _upload(client, _diamond_trace_file()) + migration_id = _create_migration(client, pipeline_id) + stage_ids = _stage_ids(session_factory, pipeline_id) + root_id = stage_ids["root"] + + _add_candidate_with_holdout( + session_factory, + migration_id=migration_id, + stage_id=root_id, + target_model="gpt-4o-mini", + prompt_variant="winner no holdout", + final_score=0.80, + holdout_score=None, + ) + + response = client.get(f"/pipelines/{pipeline_id}/migrations/{migration_id}/results") + assert response.status_code == 200, response.text + body = response.json() + root_result = next(r for r in body if r["stage_id"] == root_id) + assert root_result["holdout_score"] is None + + +# --------------------------------------------------------------------------- +# GET /pipelines/{id}/migrations/{id}/export (config download) +# --------------------------------------------------------------------------- + + +def test_export_returns_json_attachment_header( + client: TestClient, session_factory: sessionmaker +) -> None: + pipeline_id = _upload(client, _diamond_trace_file()) + migration_id = _create_migration(client, pipeline_id) + stage_ids = _stage_ids(session_factory, pipeline_id) + root_id = stage_ids["root"] + + _add_candidate( + session_factory, + migration_id=migration_id, + stage_id=root_id, + target_model="gpt-4o-mini", + prompt_variant="exported prompt", + final_score=0.85, + ) + + response = client.get(f"/pipelines/{pipeline_id}/migrations/{migration_id}/export") + assert response.status_code == 200, response.text + assert "attachment" in response.headers.get("content-disposition", "") + assert f"reprompt-config-{migration_id}.json" in response.headers.get("content-disposition", "") + + +def test_export_shape_and_keyed_by_source_id( + client: TestClient, session_factory: sessionmaker +) -> None: + pipeline_id = _upload(client, _diamond_trace_file()) + migration_id = _create_migration(client, pipeline_id) + stage_ids = _stage_ids(session_factory, pipeline_id) + root_id = stage_ids["root"] + + _add_candidate( + session_factory, + migration_id=migration_id, + stage_id=root_id, + target_model="gpt-4o-mini", + prompt_variant="best prompt", + final_score=0.9, + ) + + response = client.get(f"/pipelines/{pipeline_id}/migrations/{migration_id}/export") + assert response.status_code == 200, response.text + body = response.json() + + assert body["migration_id"] == migration_id + assert body["pipeline_id"] == pipeline_id + assert "generated_at" in body + assert len(body["stages"]) == 1 + + stage = body["stages"][0] + assert "stage_source_id" in stage + assert stage["stage_name"] == "Root" + assert stage["winning_model"] == "gpt-4o-mini" + assert stage["winning_prompt"] == "best prompt" + assert abs(stage["training_score"] - 0.9) < 1e-9 + assert "params" in stage + assert "holdout_score" in stage + + +def test_export_winner_matches_results_endpoint( + client: TestClient, session_factory: sessionmaker +) -> None: + """Export picks the same winner as /results.""" + pipeline_id = _upload(client, _diamond_trace_file()) + migration_id = _create_migration(client, pipeline_id) + stage_ids = _stage_ids(session_factory, pipeline_id) + root_id = stage_ids["root"] + + _add_candidate( + session_factory, + migration_id=migration_id, + stage_id=root_id, + target_model="gpt-4o-mini", + prompt_variant="weaker", + final_score=0.6, + ) + _add_candidate( + session_factory, + migration_id=migration_id, + stage_id=root_id, + target_model="claude-haiku-4-5", + prompt_variant="stronger", + final_score=0.92, + ) + + export = client.get(f"/pipelines/{pipeline_id}/migrations/{migration_id}/export").json() + results = client.get(f"/pipelines/{pipeline_id}/migrations/{migration_id}/results").json() + + export_stage = export["stages"][0] + result_stage = next(r for r in results if r["stage_name"] == "Root") + assert export_stage["winning_prompt"] == result_stage["winning_prompt"] + assert export_stage["winning_model"] == result_stage["winning_model"] + + +def test_export_empty_stages_when_no_candidates(client: TestClient) -> None: + pipeline_id = _upload(client, _diamond_trace_file()) + migration_id = _create_migration(client, pipeline_id) + + response = client.get(f"/pipelines/{pipeline_id}/migrations/{migration_id}/export") + assert response.status_code == 200, response.text + assert response.json()["stages"] == [] + + +def test_export_unknown_migration_returns_404(client: TestClient) -> None: + pipeline_id = _upload(client, _diamond_trace_file()) + response = client.get(f"/pipelines/{pipeline_id}/migrations/999999/export") + assert response.status_code == 404 + + +def test_export_unknown_pipeline_returns_404(client: TestClient) -> None: + response = client.get("/pipelines/999999/migrations/1/export") + assert response.status_code == 404 + + +# --------------------------------------------------------------------------- +# GET /pipelines/{id}/migrations/{id}/seam-results (Phase 4) +# --------------------------------------------------------------------------- + + +def _add_seam_result( + session_factory: sessionmaker, + *, + migration_id: int, + upstream_stage_id: int, + downstream_stage_id: int, + parity_score: float | None, + passed: bool, + substitution_applied: bool = True, + reason: str = "", +) -> None: + with session_factory() as db: + db.add( + models.SeamCheckResult( + migration_id=migration_id, + upstream_stage_id=upstream_stage_id, + downstream_stage_id=downstream_stage_id, + parity_score=parity_score, + passed=passed, + substitution_applied=substitution_applied, + reason=reason, + ) + ) + db.commit() + + +def test_seam_results_empty_when_no_seam_rows(client: TestClient) -> None: + """GET /seam-results returns [] when no SeamCheckResult rows exist.""" + pipeline_id = _upload(client, _diamond_trace_file()) + migration_id = _create_migration(client, pipeline_id) + + response = client.get(f"/pipelines/{pipeline_id}/migrations/{migration_id}/seam-results") + assert response.status_code == 200, response.text + assert response.json() == [] + + +def test_seam_results_returns_seeded_rows( + client: TestClient, session_factory: sessionmaker +) -> None: + """Seeded SeamCheckResult rows appear in the response with correct shape.""" + pipeline_id = _upload(client, _diamond_trace_file()) + migration_id = _create_migration(client, pipeline_id) + stage_ids = _stage_ids(session_factory, pipeline_id) + root_id = stage_ids["root"] + a_id = stage_ids["a"] + + _add_seam_result( + session_factory, + migration_id=migration_id, + upstream_stage_id=root_id, + downstream_stage_id=a_id, + parity_score=0.87, + passed=True, + substitution_applied=True, + reason="PASS", + ) + + response = client.get(f"/pipelines/{pipeline_id}/migrations/{migration_id}/seam-results") + assert response.status_code == 200, response.text + body = response.json() + assert len(body) == 1 + row = body[0] + assert row["upstream_stage_id"] == root_id + assert row["downstream_stage_id"] == a_id + assert abs(row["parity_score"] - 0.87) < 1e-9 + assert row["passed"] is True + assert row["substitution_applied"] is True + assert row["upstream_stage_name"] == "Root" + assert row["downstream_stage_name"] == "Branch A" + + +def test_seam_results_null_parity_score_when_not_set( + client: TestClient, session_factory: sessionmaker +) -> None: + """parity_score=None is serialised as null (budget-exhausted / all-failed seam).""" + pipeline_id = _upload(client, _diamond_trace_file()) + migration_id = _create_migration(client, pipeline_id) + stage_ids = _stage_ids(session_factory, pipeline_id) + root_id = stage_ids["root"] + b_id = stage_ids["b"] + + _add_seam_result( + session_factory, + migration_id=migration_id, + upstream_stage_id=root_id, + downstream_stage_id=b_id, + parity_score=None, + passed=False, + substitution_applied=False, + reason="budget exhausted", + ) + + response = client.get(f"/pipelines/{pipeline_id}/migrations/{migration_id}/seam-results") + assert response.status_code == 200, response.text + row = response.json()[0] + assert row["parity_score"] is None + assert row["passed"] is False + assert row["substitution_applied"] is False + + +def test_seam_results_unknown_migration_returns_404(client: TestClient) -> None: + pipeline_id = _upload(client, _diamond_trace_file()) + response = client.get(f"/pipelines/{pipeline_id}/migrations/999999/seam-results") + assert response.status_code == 404 + + +def test_seam_results_unknown_pipeline_returns_404(client: TestClient) -> None: + response = client.get("/pipelines/999999/migrations/1/seam-results") + assert response.status_code == 404 diff --git a/apps/api/tests/test_optimizer_runner.py b/apps/api/tests/test_optimizer_runner.py index fcc60c1..274c418 100644 --- a/apps/api/tests/test_optimizer_runner.py +++ b/apps/api/tests/test_optimizer_runner.py @@ -208,6 +208,7 @@ def fake( on_attempt( StageAttempt( stage_id=s.stage_id, + target_model=s.target_model, prompt_variant=f"variant for stage {s.stage_id}", params={}, format_mode="text", diff --git a/apps/web/src/components/contract-review-panel.tsx b/apps/web/src/components/contract-review-panel.tsx new file mode 100644 index 0000000..6442a5a --- /dev/null +++ b/apps/web/src/components/contract-review-panel.tsx @@ -0,0 +1,206 @@ +import { useState } from "react"; +import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; +import { + type AssertionOut, + type DagResponse, + approveAssertion, + getPipelineDag, + listAssertions, + mineContract, + retireAssertion, +} from "@/lib/api"; +import { Badge } from "@/components/ui/badge"; +import { Button } from "@/components/ui/button"; +import { Card, CardContent } from "@/components/ui/card"; +import { + Table, + TableBody, + TableCell, + TableHead, + TableHeader, + TableRow, +} from "@/components/ui/table"; + +function statusVariant(status: AssertionOut["status"]): "pass" | "near" | "neutral" | "outline" { + if (status === "approved") return "pass"; + if (status === "retired") return "neutral"; + return "outline"; +} + +function StageAssertions({ + pipelineId, + stageId, + stageName, +}: { + pipelineId: number; + stageId: number; + stageName: string; +}) { + const qc = useQueryClient(); + const [mining, setMining] = useState(false); + const [mineError, setMineError] = useState(null); + + const assertionsQuery = useQuery({ + queryKey: ["assertions", pipelineId, stageId], + queryFn: () => listAssertions(pipelineId, stageId), + }); + + const approveMut = useMutation({ + mutationFn: (aid: number) => approveAssertion(pipelineId, stageId, aid), + onSuccess: () => qc.invalidateQueries({ queryKey: ["assertions", pipelineId, stageId] }), + }); + + const retireMut = useMutation({ + mutationFn: (aid: number) => retireAssertion(pipelineId, stageId, aid), + onSuccess: () => qc.invalidateQueries({ queryKey: ["assertions", pipelineId, stageId] }), + }); + + const handleMine = async () => { + setMining(true); + setMineError(null); + try { + await mineContract(pipelineId, stageId); + qc.invalidateQueries({ queryKey: ["assertions", pipelineId, stageId] }); + } catch (e) { + setMineError(e instanceof Error ? e.message : "Mining failed"); + } finally { + setMining(false); + } + }; + + const assertions = assertionsQuery.data ?? []; + + return ( +
+
+

{stageName}

+ +
+ + {mineError && ( +

+ {mineError} +

+ )} + + {assertionsQuery.isLoading && ( +

Loading…

+ )} + + {!assertionsQuery.isLoading && assertions.length === 0 && ( +

+ No assertions yet — click "Mine contract" to extract invariants from existing traces. +

+ )} + + {assertions.length > 0 && ( + + + + + Kind + Description + Confidence + Status + + + + + {assertions.map((a) => ( + + {a.kind} + + {a.description || JSON.stringify(a.spec)} + {a.noise_floor != null && ( + + noise {Math.round(a.noise_floor * 100)}% + + )} + + + {a.confidence != null ? `${Math.round(a.confidence * 100)}%` : "—"} + + + {a.status} + + +
+ {a.status !== "approved" && ( + + )} + {a.status !== "retired" && ( + + )} +
+ + + ))} + +
+ {assertions[0]?.entropy != null && ( +
+ Semantic entropy: {assertions[0].entropy.toFixed(3)} nats +
+ )} + + )} +
+ ); +} + +export function ContractReviewPanel({ pipelineId }: { pipelineId: number }) { + const dagQuery = useQuery({ + queryKey: ["pipeline-dag", pipelineId], + queryFn: () => getPipelineDag(pipelineId), + }); + + const stages = dagQuery.data?.stages ?? []; + + if (dagQuery.isLoading) { + return

Loading stages…

; + } + + if (stages.length === 0) { + return ( + + + No stages found — import a pipeline first. + + + ); + } + + return ( +
+

+ Mine contracts from existing traces to extract invariants (required keys, enum values, + regex patterns) that the stage always produces. Approve invariants to promote them to + executable assertions used in Phase 8 validation. +

+ {Object.values(stages).map((stage) => ( + + ))} +
+ ); +} diff --git a/apps/web/src/components/data-table.test.tsx b/apps/web/src/components/data-table.test.tsx index dcfce37..473db12 100644 --- a/apps/web/src/components/data-table.test.tsx +++ b/apps/web/src/components/data-table.test.tsx @@ -50,6 +50,8 @@ function baseDag(): DagResponse { avg_tokens_in: 1, avg_tokens_out: 1, avg_latency_ms: 1, + trace_count: 0, + total_cost_usd: null, }, "20": { id: 20, @@ -58,6 +60,8 @@ function baseDag(): DagResponse { avg_tokens_in: 1, avg_tokens_out: 1, avg_latency_ms: 1, + trace_count: 0, + total_cost_usd: null, }, }, edges: [], diff --git a/apps/web/src/components/migration-success-screen.test.tsx b/apps/web/src/components/migration-success-screen.test.tsx index 0f4c026..507ff99 100644 --- a/apps/web/src/components/migration-success-screen.test.tsx +++ b/apps/web/src/components/migration-success-screen.test.tsx @@ -1,3 +1,4 @@ +import React from "react"; import { describe, it, expect, vi, beforeEach } from "vitest"; import { render, screen, waitFor, within } from "@testing-library/react"; import { fireEvent } from "@testing-library/react"; @@ -30,6 +31,13 @@ vi.mock("@/components/pipeline-canvas", () => ({ ), })); +// Link needs a RouterProvider context that these unit tests don't provide — +// stub it as a passthrough so terminal-state tests don't crash on it. +vi.mock("@tanstack/react-router", async () => { + const actual = await vi.importActual("@tanstack/react-router"); + return { ...actual, Link: ({ children }: { children: React.ReactNode }) => <>{children} }; +}); + import { getMigrationResults, getMigrationStatus, getPipelineDag } from "@/lib/api"; function baseDag(): DagResponse { @@ -37,8 +45,8 @@ function baseDag(): DagResponse { pipeline_id: 1, layers: [{ stage_ids: [10, 20] }], stages: { - "10": { id: 10, name: "Extract", model: "gpt-4o", avg_tokens_in: 100, avg_tokens_out: 50, avg_latency_ms: 500 }, - "20": { id: 20, name: "Summarize", model: "gpt-4o", avg_tokens_in: 80, avg_tokens_out: 40, avg_latency_ms: 400 }, + "10": { id: 10, name: "Extract", model: "gpt-4o", avg_tokens_in: 100, avg_tokens_out: 50, avg_latency_ms: 500, trace_count: 0, total_cost_usd: null }, + "20": { id: 20, name: "Summarize", model: "gpt-4o", avg_tokens_in: 80, avg_tokens_out: 40, avg_latency_ms: 400, trace_count: 0, total_cost_usd: null }, }, edges: [], }; @@ -150,6 +158,7 @@ function baseResults(): StageResultOut[] { winning_prompt: "Extract the total figure", winning_model: "claude-haiku-4-5", score: 0.92, + holdout_score: null, }, ]; } diff --git a/apps/web/src/components/migration-success-screen.tsx b/apps/web/src/components/migration-success-screen.tsx index 8bbb0ec..677ff07 100644 --- a/apps/web/src/components/migration-success-screen.tsx +++ b/apps/web/src/components/migration-success-screen.tsx @@ -1,4 +1,5 @@ import { useEffect, useRef, useState } from "react"; +import { Link } from "@tanstack/react-router"; import { useMutation, useQuery } from "@tanstack/react-query"; import { ApiError, @@ -193,7 +194,15 @@ export function MigrationSuccessScreen({ /> )} -
+
+ {isTerminal && ( + + + + )} diff --git a/apps/web/src/components/new-migration-wizard.test.tsx b/apps/web/src/components/new-migration-wizard.test.tsx index e822763..f3bc8cc 100644 --- a/apps/web/src/components/new-migration-wizard.test.tsx +++ b/apps/web/src/components/new-migration-wizard.test.tsx @@ -44,6 +44,8 @@ function baseDag(): DagResponse { avg_tokens_in: 100, avg_tokens_out: 50, avg_latency_ms: 500, + trace_count: 0, + total_cost_usd: null, }, "11": { id: 11, @@ -52,6 +54,8 @@ function baseDag(): DagResponse { avg_tokens_in: 200, avg_tokens_out: 80, avg_latency_ms: 700, + trace_count: 0, + total_cost_usd: null, }, }, edges: [{ from_stage_id: 10, to_stage_id: 11 }], @@ -70,6 +74,8 @@ function baseModels(): ModelOption[] { supports_json_mode: true, supports_function_calling: true, requires_api_key: true, + family: "openai", + transform_descriptions: ["Strip hedging/filler phrases and fold each paragraph's sentences into terse imperative bullet points."], }, { model: "claude-haiku-4-5", @@ -81,6 +87,8 @@ function baseModels(): ModelOption[] { supports_json_mode: true, supports_function_calling: true, requires_api_key: true, + family: "anthropic", + transform_descriptions: ["Wrap recognized labeled sections in XML-ish tags, per Anthropic's documented prompting guidance."], }, ]; } diff --git a/apps/web/src/components/new-migration-wizard.tsx b/apps/web/src/components/new-migration-wizard.tsx index d61e69e..89a0c21 100644 --- a/apps/web/src/components/new-migration-wizard.tsx +++ b/apps/web/src/components/new-migration-wizard.tsx @@ -19,27 +19,6 @@ import { AddApiKeyDrawer } from "@/components/add-api-key-drawer"; type WizardStep = "target-model" | "budget" | "confirm"; -type ModelFamily = "anthropic" | "gemini" | "openai" | "llama" | "generic"; - -const _OPEN_WEIGHT_MARKERS = ["llama", "mistral", "mixtral", "gemma", "qwen", "deepseek", "phi", "vicuna", "falcon", "starcoder"]; - -function resolveFamily(model: string): ModelFamily { - const lower = model.toLowerCase(); - if (_OPEN_WEIGHT_MARKERS.some((m) => lower.includes(m))) return "llama"; - if (lower.includes("claude")) return "anthropic"; - if (lower.includes("gemini")) return "gemini"; - if (lower.includes("gpt")) return "openai"; - return "generic"; -} - -const FAMILY_TRANSFORM_LABELS: Record = { - anthropic: "XML-tagged sections (Anthropic convention)", - gemini: "Markdown headers (Gemini convention)", - openai: "Compression on mini/small variants only", - llama: "Compression on small variants only", - generic: "Compression on small variants only", -}; - const STEPS: { id: WizardStep; label: string }[] = [ { id: "target-model", label: "Target model" }, { id: "budget", label: "Budget & parity threshold" }, @@ -304,7 +283,6 @@ export function NewMigrationWizard({ {(modelsQuery.data ?? []).map((option) => { const locked = isLocked(option); const checked = !locked && selectedModels.has(option.model); - const family = resolveFamily(option.model); const modelCard = modelCards[option.model]; return (
{modelCard && (
diff --git a/apps/web/src/components/pipeline-graph.tsx b/apps/web/src/components/pipeline-graph.tsx new file mode 100644 index 0000000..c1d1e0e --- /dev/null +++ b/apps/web/src/components/pipeline-graph.tsx @@ -0,0 +1,505 @@ +import { useEffect, useMemo, useState } from "react"; +import { useQuery } from "@tanstack/react-query"; +import { + ReactFlow, + Background, + Controls, + Panel, + useReactFlow, + type Edge, + type Node, + type NodeProps, + Handle, + Position, +} from "@xyflow/react"; +import "@xyflow/react/dist/style.css"; +import { + getPipelineDag, + listStageRecords, + type StageInfo, + type StageRecordOut, +} from "@/lib/api"; +import { Card, CardContent } from "@/components/ui/card"; +import { Badge } from "@/components/ui/badge"; +import { + computeCanvasLayout, + type CanvasOrientation, +} from "@/lib/canvas-layout"; +import { cn } from "@/lib/utils"; + +// ---- Dimensions ---- +// Approximate rendered sizes. Dagre uses STAGE_NODE_H for stage-to-stage +// vertical spacing (via canvas-layout.ts's hardcoded NODE_HEIGHT=150, which +// underestimates our richer node — the 50px difference narrows the gap from +// 96px to ~46px at spacious spacing, still non-overlapping). +// Model and call nodes are positioned manually so their sizes only affect +// visual overlap, not the dagre layout. +const STAGE_NODE_W = 256; +const STAGE_NODE_H = 200; +const MODEL_NODE_H = 72; +const MODEL_V_GAP = 20; +const MODEL_COL_GAP = 100; // horizontal gap: rightmost stage right edge → model column left edge +const CALL_NODE_W = 240; +const CALL_NODE_H = 82; +const CALL_V_GAP = 10; +const CALL_FROM_STAGE = 28; // gap from stage bottom to first call node top + +// ---- Node data types ---- + +type StageGraphNodeData = { + stage: StageInfo; + isExpanded: boolean; + isHighlighted: boolean; +}; +type StageGraphFlowNode = Node; + +type ModelGraphNodeData = { + model: string; + stageCount: number; + isHighlighted: boolean; +}; +type ModelGraphFlowNode = Node; + +type CallGraphNodeData = { + record: StageRecordOut; + index: number; +}; +type CallGraphFlowNode = Node; + +function modelNodeId(model: string) { + // Replace non-word chars so the id is safe as a CSS selector / React key + return `model-${model.replace(/\W/g, "_")}`; +} + +// ---- StageGraphNode ---- +// Richer than the canvas StageNode: trace_count, total_cost_usd, and an +// expand-calls toggle. Clicking the node is handled by ReactFlow's +// onNodeClick in the parent — the node itself is fully passive. + +function StageGraphNode({ data }: NodeProps) { + const { stage, isExpanded, isHighlighted } = data; + const canExpand = (stage.trace_count ?? 0) > 0; + + return ( + + {/* target: left — incoming dep edges */} + + +

+ {stage.name} +

+ + {stage.model} + +
+

+ {canExpand + ? `${stage.trace_count} trace${stage.trace_count === 1 ? "" : "s"}` + : "No traces yet"} +

+ {stage.avg_tokens_in != null && stage.avg_tokens_in > 0 && ( +

+ {Math.round(stage.avg_tokens_in)}→{Math.round(stage.avg_tokens_out ?? 0)}{" "} + tok · {Math.round(stage.avg_latency_ms ?? 0)}ms avg +

+ )} + {stage.total_cost_usd != null && ( +

${stage.total_cost_usd.toFixed(4)} total

+ )} +
+ {canExpand && ( +

+ {isExpanded ? "▼ collapse calls" : "▶ expand calls"} +

+ )} + + {/* source right — outgoing dep edges + model edges */} + + {/* source bottom — call edges (always rendered, hidden when no calls) */} + + + ); +} + +// ---- ModelGraphNode ---- +// Fixed-column node to the right of all stages. Clicking highlights the +// stages that use this model (sets highlightedModel via onNodeClick in +// PipelineGraph, not an internal handler). No source handles — only +// incoming edges from stages. + +function ModelGraphNode({ data }: NodeProps) { + const { model, stageCount, isHighlighted } = data; + return ( +
+ +

+ {model} +

+

+ {stageCount} stage{stageCount !== 1 ? "s" : ""} +

+
+ ); +} + +// ---- CallGraphNode ---- +// Appears below its parent stage when expanded. Compact — just the key +// stats. Full input/output is in the Data tab's record browser. + +function CallGraphNode({ data }: NodeProps) { + const { record, index } = data; + return ( +
+ +
+ Call #{index} +
+ {record.tokens_in != null && ( + + {record.tokens_in}→{record.tokens_out ?? "?"} tok + + )} + {record.latency_ms != null && ( + {Math.round(record.latency_ms)}ms + )} +
+
+ {record.cost != null && ( +

+ ${record.cost.toFixed(4)} +

+ )} +
+ ); +} + +const nodeTypes = { + stageGraph: StageGraphNode, + modelGraph: ModelGraphNode, + callGraph: CallGraphNode, +}; + +// ---- RefitOnChange ---- +// Same double-rAF pattern as pipeline-canvas.tsx: one rAF fires before the +// browser's layout pass, a second one fires after it, so fitView sees the +// real rendered positions of any newly-added call nodes. + +function RefitOnChange({ + nodeCount, + orientation, +}: { + nodeCount: number; + orientation: CanvasOrientation; +}) { + const { fitView } = useReactFlow(); + useEffect(() => { + if (nodeCount === 0) return; + let inner = () => {}; + const frame = requestAnimationFrame(() => { + const frame2 = requestAnimationFrame(() => + fitView({ padding: 0.1, minZoom: 0.25 }), + ); + inner = () => cancelAnimationFrame(frame2); + }); + return () => { + cancelAnimationFrame(frame); + inner(); + }; + }, [nodeCount, orientation, fitView]); + return null; +} + +// ---- Orientation persistence (separate key from Canvas tab) ---- + +function loadGraphOrientation(pipelineId: number): CanvasOrientation { + try { + const raw = localStorage.getItem(`reprompt.graph-orient.${pipelineId}`); + return raw === "vertical" ? "vertical" : "horizontal"; + } catch { + return "horizontal"; + } +} + +function saveGraphOrientation(pipelineId: number, o: CanvasOrientation): void { + try { + localStorage.setItem(`reprompt.graph-orient.${pipelineId}`, o); + } catch {} +} + +// ---- PipelineGraph ---- + +export function PipelineGraph({ pipelineId }: { pipelineId: number }) { + const dagQuery = useQuery({ + queryKey: ["pipeline-dag", pipelineId], + queryFn: () => getPipelineDag(pipelineId), + }); + + const [expandedStageIds, setExpandedStageIds] = useState(new Set()); + // Map — populated lazily on first expand, never evicted + // (re-expanding uses the cached records without a second network request). + const [stageRecordsMap, setStageRecordsMap] = useState( + new Map(), + ); + const [highlightedModel, setHighlightedModel] = useState(null); + const [orientation, setOrientation] = useState(() => + loadGraphOrientation(pipelineId), + ); + + useEffect(() => { + saveGraphOrientation(pipelineId, orientation); + }, [pipelineId, orientation]); + + // Fetch records for any stage that's been expanded but not yet loaded. + // Using an effect (not a query per stage) because we can't call hooks + // conditionally — we don't know stage IDs until the dag loads. + useEffect(() => { + for (const stageId of expandedStageIds) { + if (!stageRecordsMap.has(stageId)) { + listStageRecords(pipelineId, { stageId, limit: 8 }).then((page) => { + setStageRecordsMap((prev) => new Map(prev).set(stageId, page.records)); + }); + } + } + }, [expandedStageIds, stageRecordsMap, pipelineId]); + + const layout = useMemo( + () => ({ orientation, spacing: "spacious" as const }), + [orientation], + ); + + const { nodes, edges } = useMemo(() => { + const dag = dagQuery.data; + if (!dag) + return { + nodes: [] as (StageGraphFlowNode | ModelGraphFlowNode | CallGraphFlowNode)[], + edges: [] as Edge[], + }; + + const stagePositions = computeCanvasLayout(dag.layers, dag.edges, layout); + const stageList = Object.values(dag.stages); + + // ---- Stage nodes ---- + const stageNodes: StageGraphFlowNode[] = stageList.map((stage) => ({ + id: `stage-${stage.id}`, + type: "stageGraph" as const, + position: stagePositions[String(stage.id)] ?? { x: 0, y: 0 }, + data: { + stage, + isExpanded: expandedStageIds.has(stage.id), + isHighlighted: highlightedModel === stage.model, + }, + })); + + // ---- Model nodes (fixed column right of all stages) ---- + const stageRightEdges = stageList.map( + (s) => (stagePositions[String(s.id)]?.x ?? 0) + STAGE_NODE_W, + ); + const maxStageRight = stageRightEdges.length ? Math.max(...stageRightEdges) : 0; + const modelColX = maxStageRight + MODEL_COL_GAP; + + const uniqueModels = [...new Set(stageList.map((s) => s.model))]; + const stageYValues = stageList.map((s) => stagePositions[String(s.id)]?.y ?? 0); + const minStageY = stageYValues.length ? Math.min(...stageYValues) : 0; + const maxStageY = stageYValues.length ? Math.max(...stageYValues) : 0; + const totalModelH = + uniqueModels.length * MODEL_NODE_H + (uniqueModels.length - 1) * MODEL_V_GAP; + const modelStartY = (minStageY + maxStageY) / 2 - totalModelH / 2; + + const modelNodes: ModelGraphFlowNode[] = uniqueModels.map((model, i) => ({ + id: modelNodeId(model), + type: "modelGraph" as const, + position: { + x: modelColX, + y: modelStartY + i * (MODEL_NODE_H + MODEL_V_GAP), + }, + data: { + model, + stageCount: stageList.filter((s) => s.model === model).length, + isHighlighted: highlightedModel === model, + }, + })); + + // ---- Call nodes (fan below their parent stage when expanded) ---- + const callNodes: CallGraphFlowNode[] = []; + for (const stageId of expandedStageIds) { + const records = stageRecordsMap.get(stageId) ?? []; + const pos = stagePositions[String(stageId)]; + if (!pos) continue; + records.forEach((record, i) => { + callNodes.push({ + id: `call-${record.id}`, + type: "callGraph" as const, + position: { + x: pos.x + (STAGE_NODE_W - CALL_NODE_W) / 2, + y: pos.y + STAGE_NODE_H + CALL_FROM_STAGE + i * (CALL_NODE_H + CALL_V_GAP), + }, + data: { record, index: i + 1 }, + }); + }); + } + + // ---- Dependency edges (solid) ---- + const depEdges: Edge[] = dag.edges.map((e) => ({ + id: `dep-${e.from_stage_id}-${e.to_stage_id}`, + source: `stage-${e.from_stage_id}`, + sourceHandle: "right", + target: `stage-${e.to_stage_id}`, + targetHandle: "left", + style: { stroke: "var(--line)", strokeWidth: 2 }, + })); + + // ---- Model edges (dashed, stage → model node) ---- + const modelEdges: Edge[] = stageList.map((stage) => ({ + id: `model-edge-${stage.id}`, + source: `stage-${stage.id}`, + sourceHandle: "right", + target: modelNodeId(stage.model), + targetHandle: "left", + style: { + stroke: "var(--beam)", + strokeWidth: 1.5, + strokeDasharray: "5 3", + opacity: highlightedModel === stage.model ? 0.9 : 0.3, + }, + })); + + // ---- Call edges (straight, stage bottom → call top) ---- + const callEdges: Edge[] = []; + for (const stageId of expandedStageIds) { + const records = stageRecordsMap.get(stageId) ?? []; + records.forEach((record) => { + callEdges.push({ + id: `call-edge-${record.id}`, + source: `stage-${stageId}`, + sourceHandle: "bottom", + target: `call-${record.id}`, + targetHandle: "top", + type: "straight", + style: { + stroke: "var(--ink-soft)", + strokeWidth: 1, + opacity: 0.4, + }, + }); + }); + } + + return { + nodes: [...stageNodes, ...modelNodes, ...callNodes], + edges: [...depEdges, ...modelEdges, ...callEdges], + }; + }, [dagQuery.data, layout, expandedStageIds, stageRecordsMap, highlightedModel]); + + if (dagQuery.isLoading) { + return ( +

+ Loading graph… +

+ ); + } + if (dagQuery.isError) { + return ( +

+ {dagQuery.error instanceof Error + ? dagQuery.error.message + : "Couldn't load pipeline graph"} +

+ ); + } + + return ( +
+ { + if (node.id.startsWith("stage-")) { + const stageId = Number(node.id.slice(6)); + const stage = dagQuery.data?.stages[String(stageId)]; + if (!stage || (stage.trace_count ?? 0) === 0) return; + setExpandedStageIds((prev) => { + const next = new Set(prev); + next.has(stageId) ? next.delete(stageId) : next.add(stageId); + return next; + }); + } else if (node.id.startsWith("model-")) { + const model = (node.data as ModelGraphNodeData).model; + setHighlightedModel((prev) => (prev === model ? null : model)); + } + }} + > + + + +
+ {(["horizontal", "vertical"] as CanvasOrientation[]).map((o) => ( + + ))} +
+ + + +
+ ); +} diff --git a/apps/web/src/components/rubric-review-panel.test.tsx b/apps/web/src/components/rubric-review-panel.test.tsx index 7cb986f..0fc5722 100644 --- a/apps/web/src/components/rubric-review-panel.test.tsx +++ b/apps/web/src/components/rubric-review-panel.test.tsx @@ -194,6 +194,8 @@ describe("RubricReviewPanel", () => { avg_tokens_in: null, avg_tokens_out: null, avg_latency_ms: null, + trace_count: 0, + total_cost_usd: null, }, }, edges: [], diff --git a/apps/web/src/components/stage-node.test.tsx b/apps/web/src/components/stage-node.test.tsx index 58a0aeb..4a44027 100644 --- a/apps/web/src/components/stage-node.test.tsx +++ b/apps/web/src/components/stage-node.test.tsx @@ -14,6 +14,8 @@ function baseStage(overrides: Partial = {}): StageInfo { avg_tokens_in: 120, avg_tokens_out: 45, avg_latency_ms: 812, + trace_count: 0, + total_cost_usd: null, ...overrides, }; } diff --git a/apps/web/src/lib/api.ts b/apps/web/src/lib/api.ts index 89fd264..74eb9a8 100644 --- a/apps/web/src/lib/api.ts +++ b/apps/web/src/lib/api.ts @@ -51,6 +51,8 @@ export interface StageInfo { avg_tokens_in: number | null; avg_tokens_out: number | null; avg_latency_ms: number | null; + trace_count: number; + total_cost_usd: number | null; } export interface DagEdge { @@ -168,6 +170,8 @@ export interface ModelOption { supports_json_mode: boolean; supports_function_calling: boolean; requires_api_key: boolean; + family: string; + transform_descriptions: string[]; } export interface TargetModelConfig { @@ -279,6 +283,7 @@ export interface StageResultOut { winning_prompt: string; winning_model: string; score: number; + holdout_score: number | null; } export function getMigrationResults( @@ -288,6 +293,43 @@ export function getMigrationResults( return request(`/pipelines/${pipelineId}/migrations/${migrationId}/results`); } +export interface SeamCheckResultOut { + upstream_stage_id: number; + upstream_stage_name: string; + downstream_stage_id: number; + downstream_stage_name: string; + parity_score: number | null; + passed: boolean; + substitution_applied: boolean; + reason: string; +} + +export function getSeamResults( + pipelineId: number, + migrationId: number +): Promise { + return request( + `/pipelines/${pipelineId}/migrations/${migrationId}/seam-results` + ); +} + +export async function downloadMigrationExport( + pipelineId: number, + migrationId: number +): Promise { + const response = await fetch( + `${API_BASE_URL}/pipelines/${pipelineId}/migrations/${migrationId}/export` + ); + if (!response.ok) throw new ApiError(response.statusText, response.status); + const blob = await response.blob(); + const url = URL.createObjectURL(blob); + const a = document.createElement("a"); + a.href = url; + a.download = `reprompt-config-${migrationId}.json`; + a.click(); + URL.revokeObjectURL(url); +} + // --------------------------------------------------------------------------- // Model cards (migration wizard): read-only info on model family transforms // --------------------------------------------------------------------------- @@ -566,6 +608,70 @@ export function listSystemModels(): Promise { return request("/settings/system-models", { headers: authHeaders() }); } +// --------------------------------------------------------------------------- +// Phase 5 — Contract Mining (assertions) +// --------------------------------------------------------------------------- + +export interface AssertionOut { + id: number; + stage_id: number; + kind: string; + spec: Record; + description: string; + confidence: number | null; + status: "candidate" | "approved" | "retired"; + source: "mined" | "manual" | "counterexample"; + noise_floor: number | null; + entropy: number | null; + counterexamples: unknown[]; + version: number; + created_at: string; +} + +export function listAssertions(pipelineId: number, stageId: number): Promise { + return request( + `/pipelines/${pipelineId}/stages/${stageId}/assertions`, + { headers: authHeaders() } + ); +} + +export function mineContract( + pipelineId: number, + stageId: number, + opts: { axis_b_repeats?: number; budget?: number } = {} +): Promise { + return request( + `/pipelines/${pipelineId}/stages/${stageId}/mine-contract`, + { + method: "POST", + headers: { ...authHeaders(), "Content-Type": "application/json" }, + body: JSON.stringify({ axis_b_repeats: opts.axis_b_repeats ?? 3, budget: opts.budget ?? 1.0 }), + } + ); +} + +export function approveAssertion( + pipelineId: number, + stageId: number, + assertionId: number +): Promise { + return request( + `/pipelines/${pipelineId}/stages/${stageId}/assertions/${assertionId}/approve`, + { method: "POST", headers: authHeaders() } + ); +} + +export function retireAssertion( + pipelineId: number, + stageId: number, + assertionId: number +): Promise { + return request( + `/pipelines/${pipelineId}/stages/${stageId}/assertions/${assertionId}/retire`, + { method: "POST", headers: authHeaders() } + ); +} + export function addApiKey(provider: string, apiKey: string): Promise { return request("/settings/api-keys", { method: "POST", diff --git a/apps/web/src/router.tsx b/apps/web/src/router.tsx index 9f262c1..b7456e2 100644 --- a/apps/web/src/router.tsx +++ b/apps/web/src/router.tsx @@ -4,6 +4,7 @@ import Home from "./routes/home"; import DevKit from "./routes/dev-kit"; import PipelinesImport from "./routes/pipelines-import"; import PipelineWorkspace, { WORKSPACE_TABS, type WorkspaceTab } from "./routes/pipeline-workspace"; +import MigrationDetail from "./routes/migration-detail"; import Login from "./routes/login"; import AuthVerify from "./routes/auth-verify"; import Settings from "./routes/settings"; @@ -88,6 +89,12 @@ const newMigrationRedirectRoute = createRoute({ }, }); +const migrationDetailRoute = createRoute({ + getParentRoute: () => rootRoute, + path: "/pipelines/$pipelineId/migrations/$migrationId", + component: MigrationDetail, +}); + const loginRoute = createRoute({ getParentRoute: () => rootRoute, path: "/login", @@ -125,6 +132,7 @@ const routeTree = rootRoute.addChildren([ pipelineWorkspaceRoute, rubricReviewRedirectRoute, newMigrationRedirectRoute, + migrationDetailRoute, loginRoute, authVerifyRoute, settingsRoute, diff --git a/apps/web/src/routes/migration-detail.tsx b/apps/web/src/routes/migration-detail.tsx new file mode 100644 index 0000000..30eb398 --- /dev/null +++ b/apps/web/src/routes/migration-detail.tsx @@ -0,0 +1,285 @@ +import { Link, useParams } from "@tanstack/react-router"; +import { useQuery } from "@tanstack/react-query"; +import { downloadMigrationExport, getMigrationResults, getMigrationStatus, getSeamResults, type SeamCheckResultOut, type StageResultOut } from "@/lib/api"; +import { AppShell } from "@/components/app-shell"; +import { Card, CardContent } from "@/components/ui/card"; +import { Badge } from "@/components/ui/badge"; +import { Button } from "@/components/ui/button"; +import { ParityBeam, parityStatus } from "@/components/parity-beam"; +import { diffWords } from "@/lib/text-diff"; +import { + Table, + TableBody, + TableCell, + TableHead, + TableHeader, + TableRow, +} from "@/components/ui/table"; + +const STATUS_LABELS: Record = { + pending: "Pending", + running: "Running", + completed: "Completed", + failed: "Failed", + stopped_early: "Stopped early", +}; + +const STATUS_VARIANTS: Record = { + pending: "outline", + running: "neutral", + completed: "pass", + failed: "fail", + stopped_early: "near", +}; + +function DiffPrompt({ original, winning }: { original: string; winning: string }) { + const ops = diffWords(original, winning); + return ( + 
+      {ops.map((op, i) => {
+        if (op.type === "delete")
+          return {op.text};
+        if (op.type === "insert")
+          return {op.text};
+        return {op.text};
+      })}
+
+ ); +} + +function StageResultRow({ + result, + parityThreshold, +}: { + result: StageResultOut; + parityThreshold: number; +}) { + const pct = Math.round(result.score * 100); + const status = parityStatus(pct, parityThreshold * 100); + + return ( + + {result.stage_name} + + + + + {result.winning_model} + + +
+ + {pct}% +
+ + + {result.holdout_score != null ? ( + {Math.round(result.holdout_score * 100)}% + ) : ( + — + )} + + + ); +} + +export default function MigrationDetail() { + const { pipelineId, migrationId } = useParams({ + from: "/pipelines/$pipelineId/migrations/$migrationId", + }); + const pid = Number(pipelineId); + const mid = Number(migrationId); + + const statusQuery = useQuery({ + queryKey: ["migration-status", pid, mid], + queryFn: () => getMigrationStatus(pid, mid), + }); + + const resultsQuery = useQuery({ + queryKey: ["migration-results", pid, mid], + queryFn: () => getMigrationResults(pid, mid), + }); + + const seamQuery = useQuery({ + queryKey: ["seam-results", pid, mid], + queryFn: () => getSeamResults(pid, mid), + }); + + const migration = statusQuery.data; + const results: StageResultOut[] = resultsQuery.data ?? []; + const seamResults: SeamCheckResultOut[] = seamQuery.data ?? []; + + return ( + +
+ + ← Pipeline canvas + + +
+

+ Migration #{mid} +

+ {migration && ( + + {STATUS_LABELS[migration.status] ?? migration.status} + + )} +
+ + {(statusQuery.isLoading || resultsQuery.isLoading) && ( +

+ Loading results… +

+ )} + + {(statusQuery.isError || resultsQuery.isError) && ( +

+ {(() => { + const err = statusQuery.error ?? resultsQuery.error; + return err instanceof Error ? err.message : "Couldn't load migration results."; + })()} +

+ )} + + {migration && ( +
+ + +

Total spend

+

+ {migration.total_cost_usd != null + ? `$${migration.total_cost_usd.toFixed(4)}` + : "—"} +

+ + + + +

Budget

+

+ ${migration.budget.toFixed(2)} +

+ + + + +

Parity threshold

+

+ {Math.round(migration.parity_threshold * 100)}% +

+ + + + +

Stages optimized

+

+ {results.length} +

+ + +
+ )} + + {migration?.stop_reason && ( +

+ Stop reason: {migration.stop_reason} +

+ )} + +
+

Stage results

+ {results.length > 0 && migration && ["completed", "stopped_early", "failed"].includes(migration.status) && ( + + )} +
+

+ Winning prompt per stage — the best-scoring variant found across all target models. +

+ + {!resultsQuery.isLoading && results.length === 0 ? ( + + + No results yet — the migration may not have run yet. + + + ) : ( + + + + + Stage + Prompt diff (original → winning) + Model + Score + Holdout + + + + {results.map((r) => ( + + ))} + +
+ + )} + {seamResults.length > 0 && ( + <> +

Seam checks

+

+ Downstream stages re-validated against the migrated upstream output. +

+ + + + + Upstream → Downstream + Score + Result + Note + + + + {seamResults.map((r) => ( + + + {r.upstream_stage_name} → {r.downstream_stage_name} + + + {r.parity_score != null ? `${Math.round(r.parity_score * 100)}%` : "—"} + + + + {r.passed ? "Pass" : "Fail"} + + + + {!r.substitution_applied && "Stability check only — upstream key not found in downstream input. "} + {r.reason} + + + ))} + +
+ + + )} +
+ + ); +} diff --git a/apps/web/src/routes/pipeline-workspace.canvas-live.test.tsx b/apps/web/src/routes/pipeline-workspace.canvas-live.test.tsx index 2f0f553..be620e7 100644 --- a/apps/web/src/routes/pipeline-workspace.canvas-live.test.tsx +++ b/apps/web/src/routes/pipeline-workspace.canvas-live.test.tsx @@ -76,6 +76,8 @@ function baseDag(): DagResponse { avg_tokens_in: 100, avg_tokens_out: 50, avg_latency_ms: 500, + trace_count: 0, + total_cost_usd: null, }, }, edges: [], diff --git a/apps/web/src/routes/pipeline-workspace.test.tsx b/apps/web/src/routes/pipeline-workspace.test.tsx index cc99439..2a940fb 100644 --- a/apps/web/src/routes/pipeline-workspace.test.tsx +++ b/apps/web/src/routes/pipeline-workspace.test.tsx @@ -83,6 +83,8 @@ function baseDag(): DagResponse { avg_tokens_in: 100, avg_tokens_out: 50, avg_latency_ms: 500, + trace_count: 0, + total_cost_usd: null, }, }, edges: [], diff --git a/apps/web/src/routes/pipeline-workspace.tsx b/apps/web/src/routes/pipeline-workspace.tsx index 0bb823f..c8a0b69 100644 --- a/apps/web/src/routes/pipeline-workspace.tsx +++ b/apps/web/src/routes/pipeline-workspace.tsx @@ -24,8 +24,10 @@ import { import { AppShell } from "@/components/app-shell"; import { Dropzone } from "@/components/dropzone"; import { PipelineCanvas } from "@/components/pipeline-canvas"; +import { PipelineGraph } from "@/components/pipeline-graph"; import { DataTable } from "@/components/data-table"; import { RubricReviewPanel } from "@/components/rubric-review-panel"; +import { ContractReviewPanel } from "@/components/contract-review-panel"; import { NewMigrationWizard } from "@/components/new-migration-wizard"; import { MigrationSuccessScreen } from "@/components/migration-success-screen"; import { Button } from "@/components/ui/button"; @@ -41,15 +43,17 @@ import { } from "@/components/ui/drawer"; import { cn } from "@/lib/utils"; -export type WorkspaceTab = "canvas" | "data" | "rubrics" | "migrations"; +export type WorkspaceTab = "canvas" | "data" | "rubrics" | "contracts" | "migrations" | "graph"; -export const WORKSPACE_TABS: readonly WorkspaceTab[] = ["canvas", "data", "rubrics", "migrations"]; +export const WORKSPACE_TABS: readonly WorkspaceTab[] = ["canvas", "data", "rubrics", "contracts", "migrations", "graph"]; const TAB_LABELS: Record = { canvas: "Canvas", data: "Data", rubrics: "Rubrics", + contracts: "Contracts", migrations: "Migrations", + graph: "Graph", }; /** @@ -221,11 +225,18 @@ export default function PipelineWorkspace() {
)} + {tab === "contracts" && ( +
+

Contract Mining

+ +
+ )} {tab === "migrations" && (
goToTab("canvas")} />
)} + {tab === "graph" && }
diff --git a/docs/PRISM_PHASES_PLAN.md b/docs/PRISM_PHASES_PLAN.md new file mode 100644 index 0000000..1c8cfc2 --- /dev/null +++ b/docs/PRISM_PHASES_PLAN.md @@ -0,0 +1,484 @@ +# Prism — Remaining Phases Implementation Plan (Phases 3–8 + GEPA) + +Authored 2026-07-22 (Opus) against the PDF `reprompt_prism_documentation.pdf` +and the live codebase. This is the execution spec handed to Sonnet. Each +phase section is self-contained: what to build, where it lives, the data +model, the API surface, the core logic, the tests, and its dependencies. + +Ground truth as of writing: `apps/api` **180 passed**, `packages/core` +**290 passed / 21 skipped**, `apps/web` **153 passed**. Phase 2 (M4 holdout) +is complete. + +--- + +## 0. What already exists (do not rebuild) + +Read this first so no phase re-derives infrastructure that is already here. + +| Capability | Where | Notes | +|---|---|---| +| Stage DAG (`depends_on` / `dependents`) | `models.py:159-173` (self-ref M2M `stage_dependencies`) | **Phase 4 seam regression uses this directly** — the downstream set is `stage.dependents`. | +| `Stage.source_id` (user's own stage id) | `models.py:141` | Comment literally says *"M5's config export needs to write migrated prompts back out keyed by the user's own stage ids."* **Phase 3 keys export on this.** | +| `Rubric.downstream_contract` (`list[str]` of field names) | `models.py:294`, editable in `rubric-review-panel.tsx:325` | The seam definition. Currently plain field names, **not** executable predicates. | +| `Trace.is_holdout` | `models.py:226` | Already consumed by Phase 2. | +| Three-signal scoring (det 25% / embed 30% / judge 45%) | `scoring.py` — `score_candidate`, `compute_composite_score`, hard-gate on `json_schema` | Reuse for all downstream/seam scoring. Do **not** fork a second scoring path. | +| Position-swapped judge + single-pass judge | `judge.py` — `judge_pairwise`, `judge_single_pass` | Cross-family. Reuse for seam validation ("independent validator, isolated context"). | +| Role separation | `model_select.py` — `select_model(purpose, available, explicit=)`, purposes `rubric_generation`/`judge`/`mutator` | Explicit override always wins. Add new purposes here as needed. | +| Optimizer engine | `packages/core/optimizer/loop.py` — `run_optimizer(strategy="simple"\|"prism")`, shared `run_sweep_for_stage`, injected `call` | GEPA slots in as a third strategy branch. | +| Mutate / critique / refine | `optimizer/mutator.py` — `generate_prompt_mutations`, `critique_and_refine`, `select_few_shot_examples` | Lessons file (Phase 6) feeds into these calls. | +| Local, zero-cost model infra pattern | `embedding.py` (bge-m3, lazy-loaded, test-overridable model name) | **The NLI cross-encoder (Phase 5) copies this exact pattern** — local, no injected `call`, no API key. | +| LLM-derived rubric ("contract v0") | `rubric_generator.py` — one strong-model call → `deterministic_checks` + `judge_criteria` + `downstream_contract` | Phase 5 contract mining is the **evolution** of this, feeding the same rubric structures. | +| API-shell wiring pattern | `optimizer_runner.py` (core stays headless; shell reads DB, builds inputs, persists rows, `on_attempt`/`on_phase` closures) | Every new core capability follows this core/shell split. | +| BYOK creds + Fernet | `llm_context.complete_with_workspace_credentials`, `models.WorkspaceApiKey` | All paid calls route through the workspace-credential closure. | + +**Not yet present anywhere:** assertion registry table, feature-flag / +promotion table, lessons store, drift daemon, GEPA backend, NLI model, +any `export` endpoint. + +--- + +## 1. Cross-cutting decisions & conflict flags + +These are the places the PDF's vision collides with, or must be reconciled +against, what's already built. Resolve these once, here, so no phase +silently re-decides them. + +**D1 — Contract mining augments the rubric generator; it does not replace the scoring path.** +`rubric_generator.py` already produces the exact three structures the +scorer consumes. Phase 5 changes *how invariants are derived* (NLI +semantic-entropy clustering over two-axis samples instead of one LLM call) +but the output still lands as `deterministic_checks` + an **assertion +registry** (new). There must remain exactly one composite-scoring path +(`scoring.py`). Mined invariants become executable checks *inside* that +path, not a parallel evaluator. + +**D2 — Assertions are a NEW versioned table, additive to `Rubric`.** +`Rubric.deterministic_checks` is per-stage, unversioned, overwrite-on- +regenerate. The PDF wants "versioned, executable invariants" with stored +counterexamples. That is a new `assertions` table (Phase 5/8), **not** a +mutation of the rubric column. The existing HITL rubric-review flow stays +untouched. + +**D3 — Do not change `downstream_contract`'s type.** +It is `list[str]` (field names) today, edited in the rubric UI and used by +Phase 4. Making it structured/executable would break the rubric-review +panel, its API, and its tests. Phase 8's executable seam predicates go in a +**separate** structure (assertion registry rows scoped to a seam), leaving +`downstream_contract` as the human-readable field list it is now. + +**D4 — The NLI cross-encoder is local infra, NOT an LLM role.** +It sits beside `embedding.py` (lazy-loaded local model, overridable model +name for tests), does **not** go through the injected `call`, and does +**not** get selected via `model_select.py`. It's PDF tier-1 ("highest +volume, effectively zero marginal cost"). Keep `transformers`/`torch` an +optional/lazy import so the test suite runs without downloading it — mirror +how the embedding tests override `DEFAULT_EMBEDDING_MODEL`. + +**D5 — Phase 5 and Phase 8 are two halves of one system.** +Phase 5 = *derive* invariants and store them as assertion specs. Phase 8 = +*run* those specs as predicates inside the optimizer loop with backtracking ++ counterexample capture. Build 5 first (produces data), then 8 (consumes +it). Seam regression (Phase 4) can ship a v1 **before** either, using +embedding+judge parity on the downstream output; it gets sharper once +assertions exist. + +**D6 — GEPA is a third `strategy`, not a rewrite.** +`run_optimizer(strategy=...)` already branches simple/prism. GEPA adds a +third branch reusing `run_sweep_for_stage`, the mutator, and the judge. Add +a routing heuristic (example count) in `optimizer_runner.py`, not in core. + +**D7 — Governance is last and depends on 4 + 5.** +Promotion gate needs seam + end-to-end regression (Phase 4). Drift daemon +needs contract entropy (Phase 5's NLI clustering). Feature-flag/rollback is +a self-contained table + endpoints that can be built any time but is only +*useful* once regression gates exist. + +--- + +## 2. Phase 3 — Config Export *(independent, ship first)* + +**Goal:** one-click download of the winning migrated config — the migrated +prompt + params per stage, keyed by `Stage.source_id`, so a user can apply +the result to their own pipeline. + +**Data model:** none. Pure read over existing `Candidate` rows (same +winner-selection logic as `get_migration_results`). + +**API (`apps/api/src/reprompt_api/migrations.py`):** +- `GET /pipelines/{pid}/migrations/{mid}/export` → JSON body: + ```json + { + "migration_id": 12, + "pipeline_id": 3, + "generated_at": "2026-07-22T...", + "stages": [ + { + "stage_source_id": "classify_risk_flag", + "stage_name": "Risk Flag Classification", + "winning_model": "gemini/gemini-2.0-flash-lite", + "winning_prompt": "…", + "params": {"temperature": 0.2, "format_mode": "plain"}, + "training_score": 0.91, + "holdout_score": 0.72 + } + ] + } + ``` +- Reuse the exact winner query from `get_migration_results` (`max(candidates, + key=final)` per `(migration_id, stage_id)`); join `Stage.source_id`. +- Return `Content-Disposition: attachment; filename="reprompt-config-{mid}.json"` + so the browser downloads it. (FastAPI `Response`/`JSONResponse` with the + header — no new dependency.) + +**Frontend (`apps/web/src/routes/migration-detail.tsx`):** +- "Download winning config" button in the results header. On click, fetch + the export endpoint and trigger a client-side blob download (no new lib — + `URL.createObjectURL`). Disable/hide until the migration is terminal and + has ≥1 result. Follow `saas-product-design` (no dead button on empty + state). + +**Tests:** +- API: export shape, keyed by `source_id`, winner selection matches + `/results`, 404s for unknown pipeline/migration, empty `stages` when no + candidates. +- Web: button renders only in terminal state; click calls the endpoint. + +**Depends on:** nothing. **Unblocks:** nothing (leaf). + +--- + +## 3. Phase 4 — Seam-Level Regression *(build v1 before assertions)* + +**Goal (PDF §2 Phase 4, §3, §7.4):** after a stage's prompt is migrated, +re-validate every **downstream** stage against the migrated stage's *new* +output — not just re-confirm the upstream stage in isolation. Catches the +"call 19 → call 20 dropped an implicit pre-tax convention" failure class. + +**Core (new `packages/core/optimizer/seam.py`, headless):** +- `evaluate_seam(upstream_result, downstream_stage_input, *, call, judge_model, budget) -> SeamResult` + 1. Take the winning upstream prompt's produced output on a benchmark + trace (re-render + call, or reuse the attempt output if available). + 2. Substitute that new upstream output into the downstream stage's input + (the downstream stage consumed the *original* upstream output during + baseline capture; now feed it the *migrated* one). + 3. Run the downstream stage on that seam input, score its output against + the downstream stage's own baseline output using the **existing** + `score_candidate` (det + embedding + judge). This is the "independent + validator with isolated context" — the judge here shares no context + with the upstream producing call. + 4. Return `SeamResult(upstream_stage_id, downstream_stage_id, + parity_score, passed, reason)`. +- `SeamResult` / a `run_seam_regression(...)` that walks the dependency + order (`stage.dependents`) and returns `list[SeamResult]`. +- **v1 metric:** composite parity ≥ `parity_threshold` on the downstream + output. **v2 (after Phase 8):** also run the seam's executable assertions. + +**Wiring: how downstream input gets the upstream output.** +The tricky part. During baseline capture, a `StageRecord` for the +downstream stage has an `input` that already embedded the upstream output. +For v1, the pragmatic approach: identify which `downstream_contract` fields +the downstream stage reads from the upstream stage, and swap those fields' +values in the downstream `input` with the migrated upstream output (or the +relevant extracted field). Document this as the known-approximate seam +model; the exact field-mapping precision improves with Phase 5's mined +field-level contracts. Keep v1 honest about the approximation in the result +`reason`. + +**API (`migrations.py`):** +- Seam regression runs as part of the migration (in `optimizer_runner.py` + after all stages optimize) and its results persist. **New table** + `seam_results` (migration_id, upstream_stage_id, downstream_stage_id, + parity_score, passed, reason, created_at) — or persist onto a JSON field + on `Migration`. Recommend a small table for queryability. +- `GET /pipelines/{pid}/migrations/{mid}/seam-results` → `list[SeamResultOut]`. + +**Frontend:** a "Seam checks" section on `migration-detail.tsx` — a small +matrix/list of upstream→downstream pairs with pass/fail + parity, so a +caught seam (à la PDF §7.4) is visible. No blank state: show "No downstream +dependencies for this pipeline" when the DAG is flat. + +**Tests:** +- Core: seam pass when downstream output stays in-parity; seam fail when the + migrated upstream output degrades the downstream output; flat DAG → empty + result; budget exhaustion mid-walk handled. +- API: seam results persisted + returned; dependency-order traversal. + +**Depends on:** DAG (exists), scoring/judge (exists). **Unblocks:** Phase 7 +promotion gate (which requires seam pass). + +--- + +## 4. Phase 5 — Contract Mining (NLI + semantic entropy + two-axis) *(the big one)* + +**Goal (PDF §2 Phase 1, §4.1, §4.4):** replace the single-LLM-call rubric +derivation with a sampled, clustered, noise-aware miner that extracts what a +call *invariantly* does, and encode those invariants as executable +assertions. + +**5a — Local NLI cross-encoder (`packages/core/nli.py`, new, mirrors `embedding.py`):** +- `entails(premise: str, hypothesis: str) -> float` and/or + `nli_label(premise, hypothesis) -> Literal["entailment","neutral","contradiction"]` + using a DeBERTa-class MNLI cross-encoder (e.g. + `cross-encoder/nli-deberta-v3-base` via `sentence-transformers` + `CrossEncoder`, or `transformers` pipeline). +- Lazy-load; module-level `DEFAULT_NLI_MODEL`, overridable for tests (copy + `embedding.py`'s override seam exactly). Optional heavy deps — guard the + import so the suite runs without the model present (tests inject a fake + entailment fn, same as embedding tests use a tiny model / monkeypatch). +- Zero FastAPI imports. Does **not** use the injected `call` (D4). + +**5b — Semantic-entropy clustering (`packages/core/contract/cluster.py`, new):** +- `cluster_by_meaning(outputs: list[str], *, entails) -> list[Cluster]`: + bidirectional-entailment clustering (two outputs are in the same cluster + iff each entails the other, per Kuhn/Farquhar semantic entropy). Pure + function taking the entailment callable → fully unit-testable with a fake. +- `semantic_entropy(clusters) -> float`: entropy over cluster probabilities + — the drift signal Phase 7's daemon reads. + +**5c — Two-axis sampling (`packages/core/contract/mine.py`, new):** +- `mine_contract(stage_input, *, call, entails, budget) -> MinedContract`: + - **Axis A (vary context):** run the stage on N different real inputs → + reveals what *should* vary (signal). + - **Axis B (repeat identical context):** run the stage K times on the + *same* input → establishes the noise floor (how much the model + disagrees with itself when nothing changed). + - Cluster each axis. An output property that stays invariant across Axis A + clusters **and** is stable within Axis B (above the noise floor) becomes + a **contract invariant**. A split that only appears in Axis B is noise, + not a real branch — do not encode it. + - Emit invariants as **assertion specs** (structured, executable — + reusing `deterministic.py` check types where possible: `required_keys`, + `enum_values`, `regex`, `no_hallucinated_ids`, plus a new + `cited_figure`-style check if needed for the PDF's "always cites a + number" example). +- `MinedContract`: `{invariants: list[AssertionSpec], noise_floor: float, + entropy: float, samples_used: int}`. + +**5d — Assertion registry (new table `assertions`, per D2):** +- Columns: `id, stage_id, kind` (deterministic check type or seam), + `spec` (JSON predicate), `version` (int), `status` + (`candidate`/`approved`/`retired`), `source` + (`mined`/`manual`/`counterexample`), `counterexamples` (JSON list), + `created_at`. Alembic migration. +- Mining writes `status="candidate"` rows; a human approves them (reuse the + rubric-approval HITL pattern from `rubrics.py`). + +**API + shell:** +- `packages/core` stays pure (mining functions take injected `call` + + `entails`). New shell module `contract_miner.py` (analog of + `rubric_generator` shell wiring): reads stage records, runs + `mine_contract` with the workspace-credential closure, persists + `assertions` rows. +- `POST /pipelines/{pid}/stages/{sid}/mine-contract` → runs mining, returns + candidate assertions for review. +- `GET /pipelines/{pid}/stages/{sid}/assertions`, approve endpoints + (mirror rubric approve/approve-all). + +**Frontend:** a contract-review surface (extend the existing rubric-review +panel or a sibling): show mined invariants, the noise floor, per-invariant +"approve", so the HITL gate the PDF requires exists. + +**Tests:** +- NLI module with a fake entailment fn (deterministic clustering assertions). +- `cluster_by_meaning`: equivalent-meaning outputs cluster together; + contradictory ones split; entropy computed correctly. +- Two-axis: an Axis-B-only split is correctly classified as noise (not + encoded); an Axis-A invariant is encoded. +- Assertion persistence + approval flow. + +**Depends on:** scoring/deterministic (exists). **Unblocks:** Phase 8 +(executable assertions consume these specs), Phase 7 drift daemon (reads +entropy). + +--- + +## 5. Phase 6 — Lessons File *(independent; wires into mutator)* + +**Goal (PDF §5.6):** persist judge critiques as per-model-family "lessons" +and feed them to the mutator on future jobs, so the optimizer stops +rediscovering the same model-family quirks each run (GEPA's reflective- +evolution mechanic). + +**Data model (new table `lessons`):** +- `id, model_family` (e.g. `"claude"`, `"gpt"`, `"gemini"` — derive from the + model string, same family logic `model_card.py` already uses), `lesson` + (text), `source_migration_id`, `stage_id` (nullable), `created_at`, + `weight`/`hits` (optional, for ranking). Alembic migration. + +**Core:** +- `optimizer/mutator.py`: `generate_prompt_mutations` and + `critique_and_refine` gain an optional `lessons: list[str] = []` param, + injected into the mutator system prompt ("Known lessons for this model + family: …"). Keep it optional so existing callers/tests are unaffected. +- A pure `extract_lesson(judge_result, critique) -> str | None` helper that + distills a durable, model-family-level rule from a critique (one strong- + model call, or a heuristic condensation — start heuristic to avoid cost). + +**Shell (`optimizer_runner.py`):** +- Before optimizing a stage, load lessons for the target model's family and + thread them into the mutator calls. +- After a migration, extract lessons from the critiques produced during + Prism rounds and persist them. + +**API:** `GET /settings/lessons` (view/manage), optional delete — this is +memory the user may want to inspect/prune. + +**Tests:** +- `extract_lesson` produces/omits sensibly. +- Mutator prompt includes lessons when supplied (assert on the rendered + mutator prompt). +- Family derivation from model strings. +- Round-trip: a migration writes lessons; a later run reads them. + +**Depends on:** mutator (exists). **Unblocks:** nothing hard; improves all +future optimizer runs. + +--- + +## 6. Phase 7 — Governance Plane *(last; depends on 4 + 5)* + +**Goal (PDF §4.1 governance plane, §2 Phase 5):** three-level regression +gate → promotion behind a feature flag → drift daemon that re-triggers the +loop. + +**7a — End-to-end regression (core `optimizer/e2e.py`):** +- After call-level (exists) and seam-level (Phase 4), run the *whole* + pipeline on golden traces with the winning prompts swapped in, score the + final output parity. `run_e2e_regression(...) -> E2EResult`. + +**7b — Promotion gate + feature flags (new table `prompt_deployments`):** +- Columns: `id, pipeline_id, stage_id, migration_id, prompt, model, params, + state` (`staged`/`live`/`rolled_back`), `flag_key`, `promoted_at`, + `promoted_by`, `previous_deployment_id` (for rollback). Alembic migration. +- Gate logic: a migration result may be promoted **only** if call-level, + seam-level, and e2e regression all pass (compliance-critical/pinned + stages additionally require explicit human sign-off — mirror the rubric + `approved` gate; PDF §7.1). +- API: `POST …/migrations/{mid}/promote` (stages behind a flag), + `POST …/deployments/{id}/rollback` (one-click undo → restores + `previous_deployment_id`), `GET …/deployments`. + +**7c — Drift daemon (shell `drift_daemon.py`):** +- Samples live traffic (new `StageRecord`s / traces since last check), + re-runs Phase 5 clustering against the mined contract, and re-triggers a + migration job when semantic entropy rises above the mined noise floor. +- Runtime: a scheduled job. Given the current stack has no worker queue, + start with an on-demand `POST …/check-drift` endpoint + a simple + interval trigger (document the cron/worker as the productionization step); + keep the detection logic pure and testable in core. + +**Frontend:** deployments view (staged vs live, promote/rollback buttons), +drift status badge. Follow `saas-product-design` for the promotion +confirmation (irreversible-ish action → confirm). + +**Tests:** +- E2E regression pass/fail. +- Promotion blocked when any regression level fails; allowed when all pass; + pinned stage requires sign-off. +- Rollback restores the previous deployment. +- Drift: entropy above noise floor triggers; below does not. + +**Depends on:** Phase 4 (seam), Phase 5 (entropy/contract). **Unblocks:** +closes the loop (PDF Phase 5). + +--- + +## 7. Phase 8 — Executable Assertions + Backtracking *(pairs with Phase 5)* + +**Goal (PDF §2 Phase 1 "DSPy Assertions", §4.3):** the mined assertion +specs from Phase 5 become **runnable predicates** enforced inside the +optimizer loop: a failed predicate triggers backtracking (re-mutate with the +failure as context), and the failing case is stored as a counterexample for +future rounds. + +**Core (`optimizer/assertions.py`):** +- `run_assertions(output, assertions, *, input) -> AssertionRunResult`: + evaluate each `AssertionSpec` predicate against a candidate output (reuse + `deterministic.evaluate_deterministic_checks` for the check-typed ones; + add the new predicate kinds mined in Phase 5). +- Wire into `run_sweep_for_stage` / the Prism loop: on assertion failure, + feed the failure into `critique_and_refine` (backtracking = an extra + targeted refine round driven by the failed predicate), and record the + failing `(input, output, assertion)` as a counterexample on the + `assertions` row. +- Counterexamples become future few-shot / mutation context (bootstrapping, + per PDF). + +**API/DB:** append to `assertions.counterexamples`; expose counterexamples +in the contract-review UI. + +**Tests:** +- Predicate pass/fail per kind. +- A failing assertion triggers exactly one backtrack refine round (bounded). +- Counterexample persisted and surfaced. +- Assertion enforcement integrates with the existing composite score + without creating a second scoring path (D1). + +**Depends on:** Phase 5 (produces the specs). **Unblocks:** upgrades Phase 4 +seam v1 → v2 (seam assertions) and strengthens Phase 7's gate. + +--- + +## 8. GEPA backend *(optional third optimizer; flagged in PDF §4.3, not in the numbered phases)* + +**Goal:** add GEPA (reflective, Pareto-frontier) as a third +`strategy="gepa"` for the example-scarce case (2–10 samples), where the PDF +says it beats MIPROv2/PromptWizard. + +**Core (`optimizer/loop.py` + `optimizer/gepa.py`):** +- `_optimize_stage_gepa(...)`: sample trajectories → reflect in natural + language on failures (reuse the judge for the eval signal, the mutator for + the reflective update) → maintain a Pareto frontier of candidate prompts + (score vs cost, or score vs a second objective) instead of a single best → + final sweep via the shared `run_sweep_for_stage`. +- Add `"gepa"` to the `strategy` Literal and the `run_optimizer` branch. + +**Routing (`optimizer_runner.py`):** pick the backend by example count — +GEPA when `len(examples) <= ~10`, Prism/simple otherwise (PDF §4.3 +heuristic). Overridable by `OPTIMIZER_STRATEGY`. + +**Tests:** frontier maintenance (non-dominated set kept); reflective update +uses the judge signal; example-count routing. + +**Depends on:** mutator/judge/sweep (exist). Independent of 4–8; can land +whenever. Recommend **after** Phase 6 (lessons) since GEPA's reflection +benefits from the lessons file. + +--- + +## 9. Recommended build order + +1. **Phase 3 — Config export.** Trivial, independent, immediately useful. +2. **Phase 4 — Seam regression v1** (embedding+judge parity). Unblocks + governance; visible product value. +3. **Phase 5 — Contract mining** (NLI + two-axis + assertion registry). The + research core; everything downstream sharpens once this exists. +4. **Phase 8 — Executable assertions + backtracking.** Consumes Phase 5; + upgrades seam v1→v2. +5. **Phase 6 — Lessons file.** Independent; improves every subsequent run + (do before GEPA). +6. **GEPA backend.** Optional; benefits from lessons. +7. **Phase 7 — Governance plane** (e2e regression → promotion/flags → + drift daemon). Last; depends on 4 + 5. + +## 10. Invariants every phase must honor + +- **Headless core:** `packages/core` never imports FastAPI/SQLAlchemy. New + capability = pure core fn (injected `call`/`entails`) + a thin `apps/api` + shell that persists. (`optimizer_runner.py` / `rubric_generator.py` + pattern.) +- **One scoring path:** everything routes through `scoring.py`. No parallel + evaluator (D1). +- **Additive schema:** new tables, not type changes to `downstream_contract` + or `deterministic_checks` (D2, D3). One Alembic migration per phase, + single head. +- **Test side-by-side:** each phase ships with core unit tests (fake + `call`/`entails`, zero API cost) + API tests + web tests, all three suites + green before moving on. Use the `.claude/skills` (`ponytail` for laziest + correct solution, `spec-driven-planning` to map each claim to a real file, + `saas-product-design` for no-blank-state UX, `webapp-testing` for + Playwright). +- **Budget honored:** every new paid call records against `BudgetTracker` + and checks `is_exhausted` (mirror `_evaluate_holdout`). +- **BYOK:** all paid calls go through + `complete_with_workspace_credentials`. diff --git a/docs/TESTING.md b/docs/TESTING.md index eeeee3a..12a56ca 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -642,6 +642,48 @@ with `MigrationSuccessScreen`'s own run view). stops entirely while it isn't the active tab — both polls are scoped to the Canvas tab actually being mounted, not global background polling. +### 3.3d Graph tab — model/call drill-down visualization + +**What it is**: A new "Graph" tab in the pipeline workspace, complementary to +the live-run-focused Canvas tab — this one is a static analytics/drill-down +view. All UI is rendered as React Flow nodes directly inside the graph +canvas itself (no floating panels, no slide-in drawers — everything you see +below is an inline node type: `StageGraphNode`, `ModelGraphNode`, +`CallGraphNode` in `pipeline-graph.tsx`). + +**Walkthrough**: +1. Open any pipeline with at least one stage → click the **Graph** tab. +2. All stages render as nodes connected by dependency edges. Node shows: + - Stage name (truncated, full name on hover) + - Model badge + - Trace count + avg token/latency stats (shows "No traces yet" if none) + - Total accumulated cost (only when StageRecord.cost data exists) + - "View inference calls →" affordance +3. **Model nodes**: a fixed column of nodes to the right of the stages, one + per unique model used in the pipeline, connected by dashed edges from + every stage that uses it. Click a model node → its edges and every + connected stage node highlight (beam-accent border/glow). Click again to + clear the highlight. +4. **Call nodes**: click a stage node's "View inference calls" affordance → + up to 20 compact `CallGraphNode`s appear inline in the graph, positioned + below their parent stage (not a drawer/panel) — each shows tokens in/out, + latency, and cost for that individual call. Full input/output text isn't + shown here; use the Data tab's record browser for that. +5. **Orientation toggle (top-left)**: `→` = horizontal (left-to-right ranks), + `↓` = vertical (top-to-bottom). Choice persists in `localStorage` per + pipeline (separate key from Canvas tab's layout preference). +6. If a pipeline has no stages, the graph renders empty (no error). +7. Layout uses `spacious` dagre spacing — same underlying `computeCanvasLayout` + function as the Canvas tab, reusing the shared `["pipeline-dag", id]` query + cache, so switching Canvas→Graph doesn't trigger a second network request. + +**Potential regressions to watch for**: +- Adding the "graph" tab to `WORKSPACE_TABS` extends the tab bar — verify the + other four tabs still render correctly and `?tab=canvas` still works. +- `StageInfo` now has `trace_count` and `total_cost_usd` — if the backend is + not running the latest code, the graph still renders (fields default to 0 / + null via Pydantic defaults), but stats will be blank. + ### 3.4 Auth + Settings (M5) 1. Go to `/login`, enter any email, submit. diff --git a/docs/examples/trace_recorder.py b/docs/examples/trace_recorder.py index def2570..98f74b2 100644 --- a/docs/examples/trace_recorder.py +++ b/docs/examples/trace_recorder.py @@ -38,7 +38,7 @@ class _StageRecord: def to_dict(self) -> dict[str, Any]: record: dict[str, Any] = { "stage_id": self.stage_id, - "input": self.input, + "input": self.input, "rendered_prompt": self.rendered_prompt, "output": self.output, "documents": self.documents, diff --git a/packages/core/src/reprompt_core/contract/__init__.py b/packages/core/src/reprompt_core/contract/__init__.py new file mode 100644 index 0000000..58711f8 --- /dev/null +++ b/packages/core/src/reprompt_core/contract/__init__.py @@ -0,0 +1 @@ +"""Contract mining — NLI-based semantic clustering and two-axis invariant extraction.""" diff --git a/packages/core/src/reprompt_core/contract/cluster.py b/packages/core/src/reprompt_core/contract/cluster.py new file mode 100644 index 0000000..fbd5f40 --- /dev/null +++ b/packages/core/src/reprompt_core/contract/cluster.py @@ -0,0 +1,79 @@ +"""Semantic-entropy clustering over LLM outputs (PDF §4.1, §4.4). + +Pure functions — no LLM calls, no DB imports. The ``entails`` callable is +injected by the caller so tests can pass a deterministic fake without loading +any NLI model. + +Bidirectional-entailment clustering (Kuhn / Farquhar semantic entropy): +two outputs A and B belong to the same cluster iff A entails B AND B entails +A, i.e. they are semantically equivalent under the entailment model. The +cluster representative is the first member added, used for pairwise +comparisons against new candidates. + +Semantic entropy = Shannon entropy over cluster-size probabilities. A value +near zero means the model always says the same thing (one big cluster). A +high value means the model produces many semantically distinct outputs — a +signal of either genuine multi-modal behaviour or noise. +""" + +from __future__ import annotations + +import math +from collections.abc import Callable +from dataclasses import dataclass, field + +__all__ = ["Cluster", "cluster_by_meaning", "semantic_entropy"] + + +@dataclass +class Cluster: + members: list[str] = field(default_factory=list) + + @property + def representative(self) -> str: + return self.members[0] + + def __len__(self) -> int: + return len(self.members) + + +def cluster_by_meaning( + outputs: list[str], + *, + entails: Callable[[str, str], bool], +) -> list[Cluster]: + """Partition *outputs* into semantic-equivalence clusters. + + Two outputs end up in the same cluster iff each entails the other + (bidirectional test against the cluster's representative). Greedy, + O(n × k) where k is the number of clusters found so far. For typical + stage-level sample sizes (5–50 outputs) this is fast enough. + """ + clusters: list[Cluster] = [] + for output in outputs: + placed = False + for cluster in clusters: + rep = cluster.representative + if entails(rep, output) and entails(output, rep): + cluster.members.append(output) + placed = True + break + if not placed: + clusters.append(Cluster(members=[output])) + return clusters + + +def semantic_entropy(clusters: list[Cluster]) -> float: + """Shannon entropy (nats) over cluster-size probabilities. + + Returns 0.0 for an empty cluster list or a single cluster (no uncertainty). + """ + total = sum(len(c) for c in clusters) + if total == 0 or len(clusters) <= 1: + return 0.0 + entropy = 0.0 + for cluster in clusters: + p = len(cluster) / total + if p > 0: + entropy -= p * math.log(p) + return entropy diff --git a/packages/core/src/reprompt_core/contract/mine.py b/packages/core/src/reprompt_core/contract/mine.py new file mode 100644 index 0000000..6a66778 --- /dev/null +++ b/packages/core/src/reprompt_core/contract/mine.py @@ -0,0 +1,248 @@ +"""Two-axis contract mining — structural invariant extraction (PDF §4.1, §4.4). + +Axis A (vary context): run the stage on N different real inputs from existing +traces. Properties that are the same across all outputs are signals (invariants). + +Axis B (repeat identical context): run the stage K times on the *same* input +at temperature > 0. Properties that vary here are noise, not invariants. + +An output property is a CONTRACT INVARIANT iff it appears in all Axis A +outputs AND is not merely an Axis-B noise artefact. + +This module is headless: it takes ``call`` (injected LLM callable) and +``entails`` (injected NLI boolean callable) so every function here is fully +unit-testable with fakes. +""" + +from __future__ import annotations + +import json +import logging +import re +from collections.abc import Callable +from typing import Any + +from pydantic import BaseModel, ConfigDict, Field + +from reprompt_core.budget import BudgetTracker +from reprompt_core.contract.cluster import cluster_by_meaning, semantic_entropy +from reprompt_core.llm.client import LLMResponse + +__all__ = ["AssertionSpec", "MineExample", "MineInput", "MinedContract", "mine_contract"] + +logger = logging.getLogger(__name__) + +_TEMPLATE_VAR = re.compile(r"\{\{\s*(\w+)\s*\}\}") + + +def _render(template: str, data: dict[str, Any] | str) -> str: + if not isinstance(data, dict): + return template + + def _sub(m: re.Match[str]) -> str: + k = m.group(1) + v = data.get(k) + if v is None: + return m.group(0) + return v if isinstance(v, str) else json.dumps(v, ensure_ascii=False) + + return _TEMPLATE_VAR.sub(_sub, template) + + +class AssertionSpec(BaseModel): + """One mined or manual invariant — portable across the assertion registry.""" + + model_config = ConfigDict(extra="forbid") + + kind: str = Field(description="deterministic check type: required_keys, regex, enum_values, …") + spec: dict[str, Any] = Field(description="predicate parameters matching the check type") + description: str = "" + confidence: float = Field(default=1.0, ge=0.0, le=1.0) + + +class MineExample(BaseModel): + """One existing trace record used as an Axis-A sample.""" + + model_config = ConfigDict(extra="forbid") + + input: dict[str, Any] + rendered_prompt: str + output: str + + +class MineInput(BaseModel): + """Everything needed to mine the contract for one stage.""" + + model_config = ConfigDict(extra="forbid") + + stage_id: int + prompt_template: str + target_model: str + params: dict[str, Any] = Field(default_factory=dict) + examples: list[MineExample] = Field(min_length=1) + axis_b_repeats: int = Field(default=3, ge=0, le=10) + + +class MinedContract(BaseModel): + """Result of a two-axis mining run.""" + + model_config = ConfigDict(extra="forbid") + + invariants: list[AssertionSpec] + noise_floor: float = Field( + ge=0.0, + le=1.0, + description="Proportion of Axis-B samples that fell outside the dominant cluster.", + ) + entropy: float = Field( + ge=0.0, + description="Semantic entropy (nats) over Axis-A clusters — drift signal for Phase 7.", + ) + samples_used: int + axis_a_count: int + axis_b_count: int + + +# --------------------------------------------------------------------------- +# Structural invariant extraction (heuristic, no NLI needed) +# --------------------------------------------------------------------------- + + +def _try_parse_json_objects(outputs: list[str]) -> list[dict[str, Any]] | None: + """Return parsed dicts if every output is a JSON object, else None.""" + parsed = [] + for o in outputs: + stripped = o.strip() + try: + val = json.loads(stripped) + if not isinstance(val, dict): + return None + parsed.append(val) + except json.JSONDecodeError: + return None + return parsed or None + + +def _longest_common_start(strings: list[str], min_length: int = 3) -> str: + if not strings: + return "" + ref = strings[0] + for i, ch in enumerate(ref): + if not all(s[i : i + 1] == ch for s in strings[1:]): + common = ref[:i].rstrip() + return common if len(common) >= min_length else "" + common = ref.rstrip() + return common if len(common) >= min_length else "" + + +def _extract_invariants(outputs: list[str], noise_floor: float) -> list[AssertionSpec]: + """Heuristic structural invariant extraction from a set of outputs.""" + if not outputs: + return [] + confidence = max(0.0, 1.0 - noise_floor) + invariants: list[AssertionSpec] = [] + + json_objects = _try_parse_json_objects(outputs) + if json_objects: + common_keys = set(json_objects[0].keys()) + for obj in json_objects[1:]: + common_keys &= set(obj.keys()) + if common_keys: + invariants.append( + AssertionSpec( + kind="required_keys", + spec={"keys": sorted(common_keys)}, + description=f"Output always contains keys: {sorted(common_keys)}", + confidence=confidence, + ) + ) + for key in sorted(common_keys): + values = [str(obj[key]) for obj in json_objects if key in obj] + unique = list(dict.fromkeys(values)) + # Only emit enum_values when the cardinality is small and sample is meaningful + if 1 < len(unique) <= 5 and len(values) >= 3: + invariants.append( + AssertionSpec( + kind="enum_values", + spec={"field": key, "values": unique}, + description=f"Field '{key}' always has one of: {unique}", + confidence=confidence, + ) + ) + else: + stripped = [o.strip() for o in outputs] + common_prefix = _longest_common_start(stripped) + if common_prefix: + pattern = "^" + re.escape(common_prefix) + invariants.append( + AssertionSpec( + kind="regex", + spec={"pattern": pattern}, + description=f"Output always starts with: '{common_prefix}'", + confidence=confidence, + ) + ) + + return invariants + + +# --------------------------------------------------------------------------- +# Two-axis mining +# --------------------------------------------------------------------------- + + +def mine_contract( + mine_input: MineInput, + *, + call: Callable[..., LLMResponse], + entails: Callable[[str, str], bool], + budget: BudgetTracker, +) -> MinedContract: + """Run the two-axis sampling protocol and return mined invariants. + + Axis A: existing outputs from ``mine_input.examples`` (no new LLM calls). + Axis B: ``axis_b_repeats`` calls on the first example's rendered prompt + at temperature > 0 to measure self-noise. + """ + axis_a_outputs = [ex.output for ex in mine_input.examples] + axis_b_outputs: list[str] = [] + + # Axis B — run the first example's already-rendered prompt K times + if mine_input.axis_b_repeats > 0 and mine_input.examples: + rendered_b = mine_input.examples[0].rendered_prompt + for _ in range(mine_input.axis_b_repeats): + if budget.is_exhausted: + break + try: + resp = call( + mine_input.target_model, + [{"role": "user", "content": rendered_b}], + temperature=0.7, + ) + budget.record_spend(resp.cost_usd or 0.0, candidate_id="mine-axis-b") + axis_b_outputs.append(resp.content) + except Exception as exc: # noqa: BLE001 + logger.warning("Axis-B call failed: %s", exc) + + # Cluster Axis A → semantic entropy + axis_a_clusters = cluster_by_meaning(axis_a_outputs, entails=entails) + entropy = semantic_entropy(axis_a_clusters) + + # Noise floor from Axis B + if axis_b_outputs: + axis_b_clusters = cluster_by_meaning(axis_b_outputs, entails=entails) + dominant = max(len(c) for c in axis_b_clusters) if axis_b_clusters else 0 + noise_floor = 1.0 - (dominant / len(axis_b_outputs)) + else: + noise_floor = 0.0 + + invariants = _extract_invariants(axis_a_outputs, noise_floor=noise_floor) + + return MinedContract( + invariants=invariants, + noise_floor=noise_floor, + entropy=entropy, + samples_used=len(axis_a_outputs) + len(axis_b_outputs), + axis_a_count=len(axis_a_outputs), + axis_b_count=len(axis_b_outputs), + ) diff --git a/packages/core/src/reprompt_core/nli.py b/packages/core/src/reprompt_core/nli.py new file mode 100644 index 0000000..ffec262 --- /dev/null +++ b/packages/core/src/reprompt_core/nli.py @@ -0,0 +1,127 @@ +"""Local NLI cross-encoder — entailment scoring between two text strings. + +Mirrors ``embedding.py``'s architecture exactly (D4 from the conflict flags): +- Zero FastAPI imports, zero ``call``/LLM dependency. +- Lazy-load via ``lru_cache`` so the first call pays the model-download cost, + subsequent calls are free. +- Module-level ``DEFAULT_NLI_MODEL`` constant, freely overridable per-call so + tests can inject a fake callable instead of loading any model at all. +- ``sentence_transformers`` / ``torch`` are optional heavy imports — guarded + so the full suite runs without them (unit tests inject a fake ``entails`` + callable; live tests are skipped when the package is absent). + +Usage in production: + from reprompt_core.nli import entails, entailment_score + score = entailment_score("Paris is in France", "Paris is a European city") + # → float in [0, 1] + +Usage in tests (no model, no download): + # Pass a fake entails callable directly to cluster_by_meaning / mine_contract + def exact_match(a, b): return a.strip() == b.strip() +""" + +from __future__ import annotations + +from functools import lru_cache +from typing import TYPE_CHECKING, Final, Literal + +if TYPE_CHECKING: + from sentence_transformers import CrossEncoder + +__all__ = [ + "DEFAULT_NLI_MODEL", + "NLI_AVAILABLE", + "entailment_score", + "nli_label", + "entails", +] + +DEFAULT_NLI_MODEL: Final[str] = "cross-encoder/nli-deberta-v3-base" +"""Default local NLI cross-encoder model. + +Overridable per-call: pass ``model_name=`` to any function in this module. +Do not change this default for tests — pass a fake callable instead. +""" + +try: + from sentence_transformers import CrossEncoder as _CrossEncoder # noqa: F401 + NLI_AVAILABLE: bool = True +except ImportError: + NLI_AVAILABLE = False + + +@lru_cache(maxsize=None) +def _load_model(model_name: str) -> "CrossEncoder": + from sentence_transformers import CrossEncoder + + return CrossEncoder(model_name) + + +def _entailment_idx(model: "CrossEncoder") -> int: + """Return the output-array index that corresponds to 'entailment'.""" + try: + for idx, label in model.config.id2label.items(): + if "entail" in str(label).lower(): + return int(idx) + except AttributeError: + pass + return 1 # NLI convention: most models use index 1 for entailment + + +def entailment_score( + premise: str, + hypothesis: str, + model_name: str = DEFAULT_NLI_MODEL, +) -> float: + """Return the probability (0–1) that *premise* entails *hypothesis*. + + Raises ``ImportError`` if ``sentence_transformers`` is not installed. + Raises ``ValueError`` if either input is empty. + """ + if not premise.strip(): + raise ValueError("nli: premise is empty") + if not hypothesis.strip(): + raise ValueError("nli: hypothesis is empty") + + model = _load_model(model_name) + scores = model.predict([(premise, hypothesis)], apply_softmax=True) + ent_idx = _entailment_idx(model) + raw = float(scores[0][ent_idx]) + return max(0.0, min(1.0, raw)) + + +def nli_label( + premise: str, + hypothesis: str, + model_name: str = DEFAULT_NLI_MODEL, +) -> Literal["entailment", "neutral", "contradiction"]: + """Return the most likely NLI relation (highest-probability label).""" + if not premise.strip(): + raise ValueError("nli: premise is empty") + if not hypothesis.strip(): + raise ValueError("nli: hypothesis is empty") + + model = _load_model(model_name) + scores = model.predict([(premise, hypothesis)], apply_softmax=True) + idx = int(scores[0].argmax()) + + try: + raw_label = str(model.config.id2label[idx]).lower() + except (AttributeError, KeyError): + raw_label = ["contradiction", "entailment", "neutral"][idx] + + if "entail" in raw_label: + return "entailment" + if "contradict" in raw_label: + return "contradiction" + return "neutral" + + +def entails( + premise: str, + hypothesis: str, + threshold: float = 0.5, + model_name: str = DEFAULT_NLI_MODEL, +) -> bool: + """Return True if the NLI entailment probability ≥ *threshold*.""" + return entailment_score(premise, hypothesis, model_name) >= threshold diff --git a/packages/core/src/reprompt_core/optimizer/loop.py b/packages/core/src/reprompt_core/optimizer/loop.py index 32b2f16..eb56a87 100644 --- a/packages/core/src/reprompt_core/optimizer/loop.py +++ b/packages/core/src/reprompt_core/optimizer/loop.py @@ -193,6 +193,12 @@ class StageOptimizationInput(BaseModel): description="Real benchmark input/output pairs for this stage. The first is used as the " "representative example this stage's attempts are scored against — see module docstring.", ) + holdout_examples: list[MutationExample | dict[str, Any]] = Field( + default_factory=list, + description="Examples withheld from optimization, used only for final unbiased evaluation " + "of the winning prompt after selection. Empty means no holdout pass. Scored with " + "deterministic + embedding only (no judge) to keep holdout evaluation cheap.", + ) class StageAttempt(BaseModel): @@ -202,6 +208,7 @@ class StageAttempt(BaseModel): model_config = ConfigDict(extra="forbid") stage_id: int + target_model: str = Field(description="LiteLLM model string this attempt was run against.") prompt_variant: str params: dict[str, Any] format_mode: str @@ -269,6 +276,12 @@ class StageResult(BaseModel): met_threshold: bool selection_reason: str error: str | None = Field(default=None, description="Set if this stage failed unexpectedly (caught, not propagated).") + holdout_score: float | None = Field( + default=None, + description="Mean composite score (deterministic + embedding, no judge) of the winning " + "prompt across holdout_examples. None when no holdout examples were provided or all " + "holdout calls failed. Lower than the training score by design (judge weight excluded).", + ) class OptimizationResult(BaseModel): @@ -512,6 +525,7 @@ def run_sweep_for_stage( scored_candidate = ScoredSweepCandidate(candidate=sweep_candidate, score=composite) attempt = StageAttempt( stage_id=stage_input.stage_id, + target_model=stage_input.target_model, prompt_variant=prompt_text, params={ "temperature": sweep_candidate.temperature, @@ -874,6 +888,61 @@ def _optimize_stage_prism( return result +# --------------------------------------------------------------------------- +# Holdout evaluation — M4 +# --------------------------------------------------------------------------- + + +def _evaluate_holdout( + best: StageAttempt, + stage_input: StageOptimizationInput, + call: Callable[..., LLMResponse], + budget: BudgetTracker, +) -> float | None: + """Score the winning prompt against held-out examples it was never optimized on. + + Runs deterministic + embedding scoring only — no judge calls — to keep the + holdout pass cheap. Returns the mean final_score across all holdout examples + that completed, or None if every call failed or the budget was exhausted + before any ran. The resulting score is inherently lower than the training + score (judge weight is absent), so compare holdout vs holdout, not + holdout vs training. + """ + deterministic_checks = parse_deterministic_checks( + stage_input.rubric.get("deterministic_checks") or [] + ) + transformed_prompt = apply_model_card_transform(best.prompt_variant, stage_input.target_model) + scores: list[float] = [] + + for raw_example in stage_input.holdout_examples: + if budget.is_exhausted: + break + example = _example_dict(raw_example) + example_input = example.get("input", {}) + benchmark_output = example["output"] + rendered = _render_template(transformed_prompt, example_input) + try: + response = call( + stage_input.target_model, + [{"role": "user", "content": rendered}], + temperature=best.params.get("temperature", 0.0), + ) + except Exception as exc: # noqa: BLE001 - one holdout failure must not abort the whole pass + logger.warning("Holdout call failed for stage %s: %s", stage_input.stage_id, exc) + continue + budget.record_spend(response.cost_usd or 0.0, candidate_id=f"holdout-{stage_input.stage_id}") + composite = score_candidate( + benchmark_output=benchmark_output, + candidate_output=response.content, + deterministic_checks=deterministic_checks, + input=example_input, + judge_score=None, # holdout is cheap — no judge, only det + embedding + ) + scores.append(composite.final_score) + + return sum(scores) / len(scores) if scores else None + + # --------------------------------------------------------------------------- # Entry point # --------------------------------------------------------------------------- @@ -995,6 +1064,14 @@ def run_optimizer( selection_reason="", error=str(exc), ) + # M4 holdout: score the winner on examples it was never optimized on. + # Skipped silently if no holdout examples were provided or the budget + # is already exhausted — never fails the stage. + if result.best is not None and stage_input.holdout_examples and not budget.is_exhausted: + try: + result.holdout_score = _evaluate_holdout(result.best, stage_input, call, budget) + except Exception as exc: # noqa: BLE001 + logger.warning("Holdout evaluation failed for stage %s: %s", stage_input.stage_id, exc) stage_results.append(result) stopped_early = stop_reason is not None or budget.is_exhausted diff --git a/packages/core/src/reprompt_core/optimizer/seam.py b/packages/core/src/reprompt_core/optimizer/seam.py new file mode 100644 index 0000000..ede7112 --- /dev/null +++ b/packages/core/src/reprompt_core/optimizer/seam.py @@ -0,0 +1,228 @@ +"""Seam-level regression — re-validate downstream stages against a migrated +upstream stage's new output (PDF §2 Phase 4, §3, §7.4). + +Per-call contract satisfaction is necessary but not sufficient for a pipeline. +A downstream stage's contract was validated against the *original* upstream +output distribution; swapping the upstream model shifts that distribution even +when the upstream still satisfies its own contract. This module checks the +seam: does the downstream stage, running its **original** prompt on the +**original** model, still produce correct output when the upstream changes? + +Headless convention +------------------- +Zero FastAPI/DB imports. The caller (optimizer_runner.py) provides: +- ``call`` — the injected LLM-calling function +- ``budget`` — BudgetTracker to record / gate spend +- ``judge_model`` — for the downstream scoring (cross-family, as always) + +Seam input substitution (v1 approximation) +------------------------------------------ +The downstream stage's input dict typically contains the upstream stage's +output under the key matching the upstream ``source_id`` (e.g. a stage +``"root"`` puts its output into the downstream's ``{"root": "..."}`` input). +v1 substitutes that key. If the key is absent (the seam mapping can't be +inferred from source_id alone), the downstream is called with its original +inputs unchanged and ``SeamResult.substitution_applied`` is ``False`` — the +score then measures downstream stability rather than true seam impact. +Scoring uses det+embedding only (no judge) to keep the seam pass cheap, +mirroring the holdout pass rationale. +""" + +from __future__ import annotations + +import json +import logging +import re +from collections.abc import Callable +from typing import Any + +from pydantic import BaseModel, ConfigDict, Field + +from reprompt_core.budget import BudgetTracker +from reprompt_core.deterministic import parse_deterministic_checks +from reprompt_core.llm.client import LLMResponse +from reprompt_core.llm.model_card import apply_model_card_transform +from reprompt_core.scoring import score_candidate + +__all__ = ["SeamExample", "SeamInput", "SeamResult", "evaluate_seam"] + +_TEMPLATE_VAR = re.compile(r"\{\{\s*(\w+)\s*\}\}") + + +def _render(template: str, data: dict[str, Any] | str) -> str: + if not isinstance(data, dict): + return template + def _sub(m: re.Match[str]) -> str: + k = m.group(1) + v = data.get(k) + if v is None: + return m.group(0) + return v if isinstance(v, str) else json.dumps(v, ensure_ascii=False) + return _TEMPLATE_VAR.sub(_sub, template) + +logger = logging.getLogger(__name__) + + +class SeamExample(BaseModel): + """One benchmark trace pair for a seam check.""" + + model_config = ConfigDict(extra="forbid") + + upstream_input: dict[str, Any] + upstream_baseline_output: str + downstream_input: dict[str, Any] + downstream_baseline_output: str + + +class SeamInput(BaseModel): + """Everything needed to check one (upstream, downstream) stage seam.""" + + model_config = ConfigDict(extra="forbid") + + upstream_stage_id: int + upstream_source_id: str + upstream_winning_prompt: str + upstream_target_model: str + upstream_params: dict[str, Any] = Field(default_factory=dict) + + downstream_stage_id: int + downstream_original_prompt: str + downstream_original_model: str + downstream_rubric: dict[str, Any] = Field(default_factory=dict) + + examples: list[SeamExample] = Field(min_length=1) + parity_threshold: float = 0.95 + + +class SeamResult(BaseModel): + model_config = ConfigDict(extra="forbid") + + upstream_stage_id: int + downstream_stage_id: int + parity_score: float | None = Field( + default=None, + description="Mean composite score (det + embedding) of the downstream stage's output " + "when given the migrated upstream output. None when every call failed or budget ran out.", + ) + passed: bool + substitution_applied: bool = Field( + description="True when the upstream output was successfully substituted into the " + "downstream input (keyed by upstream_source_id). False when the key was absent — " + "score then reflects downstream stability, not true seam impact.", + ) + reason: str + + +def evaluate_seam( + seam_input: SeamInput, + *, + call: Callable[..., LLMResponse], + budget: BudgetTracker, +) -> SeamResult: + """Run one seam check: migrated upstream → original downstream → score. + + Steps per example: + 1. Run the winning upstream prompt on the upstream input → migrated output. + 2. Substitute migrated output into the downstream input under ``upstream_source_id``. + 3. Run the original downstream prompt on the (possibly modified) downstream input. + 4. Score downstream output vs baseline (det + embedding, no judge). + Return mean score; passed = mean ≥ parity_threshold. + """ + deterministic_checks = parse_deterministic_checks( + seam_input.downstream_rubric.get("deterministic_checks") or [] + ) + up_prompt_transformed = apply_model_card_transform( + seam_input.upstream_winning_prompt, seam_input.upstream_target_model + ) + down_prompt_transformed = apply_model_card_transform( + seam_input.downstream_original_prompt, seam_input.downstream_original_model + ) + + scores: list[float] = [] + any_substitution = False + + for ex in seam_input.examples: + if budget.is_exhausted: + break + + # Step 1 — run migrated upstream prompt. + up_rendered = _render(up_prompt_transformed, ex.upstream_input) + try: + up_response = call( + seam_input.upstream_target_model, + [{"role": "user", "content": up_rendered}], + temperature=seam_input.upstream_params.get("temperature", 0.0), + ) + except Exception as exc: # noqa: BLE001 + logger.warning("Seam upstream call failed for stage %s: %s", seam_input.upstream_stage_id, exc) + continue + budget.record_spend( + up_response.cost_usd or 0.0, + candidate_id=f"seam-up-{seam_input.upstream_stage_id}", + ) + migrated_up_output = up_response.content + + if budget.is_exhausted: + break + + # Step 2 — substitute migrated upstream output into downstream input. + down_input = dict(ex.downstream_input) + substituted = seam_input.upstream_source_id in down_input + if substituted: + down_input[seam_input.upstream_source_id] = migrated_up_output + any_substitution = True + + # Step 3 — run original downstream prompt on seam input. + down_rendered = _render(down_prompt_transformed, down_input) + try: + down_response = call( + seam_input.downstream_original_model, + [{"role": "user", "content": down_rendered}], + temperature=0.0, + ) + except Exception as exc: # noqa: BLE001 + logger.warning("Seam downstream call failed for stage %s: %s", seam_input.downstream_stage_id, exc) + continue + budget.record_spend( + down_response.cost_usd or 0.0, + candidate_id=f"seam-down-{seam_input.downstream_stage_id}", + ) + + # Step 4 — score downstream output vs its baseline (det + embedding, no judge). + try: + composite = score_candidate( + benchmark_output=ex.downstream_baseline_output, + candidate_output=down_response.content, + deterministic_checks=deterministic_checks, + input=down_input, + judge_score=None, + ) + scores.append(composite.final_score) + except ValueError as exc: + logger.warning("Seam scoring failed for stage %s: %s", seam_input.downstream_stage_id, exc) + + if not scores: + return SeamResult( + upstream_stage_id=seam_input.upstream_stage_id, + downstream_stage_id=seam_input.downstream_stage_id, + parity_score=None, + passed=False, + substitution_applied=any_substitution, + reason="No examples could be evaluated (all calls failed or budget exhausted).", + ) + + mean_score = sum(scores) / len(scores) + passed = mean_score >= seam_input.parity_threshold + sub_note = "" if any_substitution else " (upstream output key absent — downstream stability check only)" + return SeamResult( + upstream_stage_id=seam_input.upstream_stage_id, + downstream_stage_id=seam_input.downstream_stage_id, + parity_score=mean_score, + passed=passed, + substitution_applied=any_substitution, + reason=( + f"Downstream stage {seam_input.downstream_stage_id} scored " + f"{mean_score:.3f} (threshold {seam_input.parity_threshold:.2f}) — " + f"{'PASS' if passed else 'FAIL'}{sub_note}." + ), + ) diff --git a/packages/core/tests/test_contract_cluster.py b/packages/core/tests/test_contract_cluster.py new file mode 100644 index 0000000..af15f65 --- /dev/null +++ b/packages/core/tests/test_contract_cluster.py @@ -0,0 +1,109 @@ +"""Tests for reprompt_core.contract.cluster — semantic entropy clustering. + +All tests use a fake entails callable: exact string match. This makes the +clustering deterministic without loading any NLI model. +""" + +from __future__ import annotations + +import math + +from reprompt_core.contract.cluster import Cluster, cluster_by_meaning, semantic_entropy + + +def exact_match(a: str, b: str) -> bool: + return a.strip() == b.strip() + + +def always_entails(a: str, b: str) -> bool: + return True + + +def never_entails(a: str, b: str) -> bool: + return False + + +def test_identical_outputs_cluster_together() -> None: + outputs = ["hello world", "hello world", "hello world"] + clusters = cluster_by_meaning(outputs, entails=exact_match) + assert len(clusters) == 1 + assert len(clusters[0]) == 3 + + +def test_distinct_outputs_each_get_own_cluster() -> None: + outputs = ["apple", "banana", "cherry"] + clusters = cluster_by_meaning(outputs, entails=exact_match) + assert len(clusters) == 3 + assert all(len(c) == 1 for c in clusters) + + +def test_partial_duplicates_cluster_correctly() -> None: + outputs = ["foo", "bar", "foo", "baz", "bar"] + clusters = cluster_by_meaning(outputs, entails=exact_match) + assert len(clusters) == 3 + sizes = sorted(len(c) for c in clusters) + assert sizes == [1, 2, 2] + + +def test_always_entails_gives_one_cluster() -> None: + outputs = ["a", "b", "c", "d"] + clusters = cluster_by_meaning(outputs, entails=always_entails) + assert len(clusters) == 1 + assert len(clusters[0]) == 4 + + +def test_never_entails_gives_one_cluster_per_output() -> None: + outputs = ["x", "y", "z"] + # even with never_entails, the first output creates a cluster and the + # second call entails(rep, output) is False → new cluster each time. + clusters = cluster_by_meaning(outputs, entails=never_entails) + assert len(clusters) == 3 + + +def test_empty_outputs_gives_empty_clusters() -> None: + clusters = cluster_by_meaning([], entails=exact_match) + assert clusters == [] + + +def test_single_output_gives_single_cluster() -> None: + clusters = cluster_by_meaning(["only"], entails=exact_match) + assert len(clusters) == 1 + assert clusters[0].representative == "only" + + +def test_cluster_representative_is_first_member() -> None: + clusters = cluster_by_meaning(["alpha", "beta", "alpha"], entails=exact_match) + alpha_cluster = next(c for c in clusters if c.representative == "alpha") + assert alpha_cluster.representative == "alpha" + + +# --------------------------------------------------------------------------- +# semantic_entropy +# --------------------------------------------------------------------------- + + +def test_entropy_zero_for_single_cluster() -> None: + clusters = [Cluster(members=["a", "b", "c"])] + assert semantic_entropy(clusters) == 0.0 + + +def test_entropy_zero_for_empty_list() -> None: + assert semantic_entropy([]) == 0.0 + + +def test_entropy_log2_for_two_equal_clusters() -> None: + clusters = [Cluster(members=["a", "b"]), Cluster(members=["c", "d"])] + # p = 0.5 each → entropy = -2 * (0.5 * ln(0.5)) = ln(2) ≈ 0.693 + entropy = semantic_entropy(clusters) + assert abs(entropy - math.log(2)) < 1e-9 + + +def test_entropy_higher_for_more_clusters() -> None: + two_clusters = [Cluster(members=["a", "b"]), Cluster(members=["c", "d"])] + four_clusters = [Cluster(members=["a"]), Cluster(members=["b"]), Cluster(members=["c"]), Cluster(members=["d"])] + assert semantic_entropy(four_clusters) > semantic_entropy(two_clusters) + + +def test_entropy_non_negative() -> None: + clusters = [Cluster(members=["x"]), Cluster(members=["y", "z"])] + assert semantic_entropy(clusters) >= 0.0 diff --git a/packages/core/tests/test_contract_mine.py b/packages/core/tests/test_contract_mine.py new file mode 100644 index 0000000..70a952c --- /dev/null +++ b/packages/core/tests/test_contract_mine.py @@ -0,0 +1,220 @@ +"""Tests for reprompt_core.contract.mine — two-axis contract mining. + +All tests use fake call/entails callables (no model, no network). +""" + +from __future__ import annotations + +import json + +from reprompt_core.budget import BudgetTracker +from reprompt_core.contract.mine import AssertionSpec, MineExample, MineInput, MinedContract, mine_contract +from reprompt_core.llm.client import LLMResponse +from reprompt_core.trace import TokenUsage + + +def _fake_usage() -> TokenUsage: + return TokenUsage(input=5, output=5, thinking=None) + + +def _make_call(output: str = "response", cost: float = 0.001): + def call(model: str, messages: list, **kw) -> LLMResponse: + return LLMResponse( + content=output, + model=model, + provider="fake", + usage=_fake_usage(), + cost_usd=cost, + latency_ms=1.0, + finish_reason="stop", + ) + return call + + +def _exact_match(a: str, b: str) -> bool: + return a.strip() == b.strip() + + +def _json_example(keys: list[str], output: dict) -> MineExample: + return MineExample( + input={"q": "test"}, + rendered_prompt="Answer: {{q}}", + output=json.dumps(output), + ) + + +def _mine_input(examples: list[MineExample], axis_b_repeats: int = 0) -> MineInput: + return MineInput( + stage_id=1, + prompt_template="{{q}}", + target_model="gpt-4o-mini", + examples=examples, + axis_b_repeats=axis_b_repeats, + ) + + +# --------------------------------------------------------------------------- +# Structural invariant extraction +# --------------------------------------------------------------------------- + + +def test_required_keys_mined_from_uniform_json_outputs() -> None: + examples = [ + MineExample(input={}, rendered_prompt="p", output=json.dumps({"currency": "USD", "value": 100})), + MineExample(input={}, rendered_prompt="p", output=json.dumps({"currency": "EUR", "value": 200})), + MineExample(input={}, rendered_prompt="p", output=json.dumps({"currency": "GBP", "value": 50})), + ] + result = mine_contract( + _mine_input(examples, axis_b_repeats=0), + call=_make_call(), + entails=_exact_match, + budget=BudgetTracker(budget_usd=10.0), + ) + kinds = {inv.kind for inv in result.invariants} + assert "required_keys" in kinds + req_keys_spec = next(inv for inv in result.invariants if inv.kind == "required_keys") + assert set(req_keys_spec.spec["keys"]) == {"currency", "value"} + + +def test_enum_values_mined_from_repeated_field_value() -> None: + examples = [ + MineExample(input={}, rendered_prompt="p", output=json.dumps({"status": "ok", "score": 1})), + MineExample(input={}, rendered_prompt="p", output=json.dumps({"status": "fail", "score": 0})), + MineExample(input={}, rendered_prompt="p", output=json.dumps({"status": "ok", "score": 1})), + ] + result = mine_contract( + _mine_input(examples, axis_b_repeats=0), + call=_make_call(), + entails=_exact_match, + budget=BudgetTracker(budget_usd=10.0), + ) + enum_specs = [inv for inv in result.invariants if inv.kind == "enum_values" and inv.spec.get("field") == "status"] + assert enum_specs, "expected enum_values assertion for 'status' field" + assert set(enum_specs[0].spec["values"]) == {"ok", "fail"} + + +def test_regex_prefix_mined_from_text_outputs() -> None: + examples = [ + MineExample(input={}, rendered_prompt="p", output="Answer: yes this works"), + MineExample(input={}, rendered_prompt="p", output="Answer: no it does not"), + MineExample(input={}, rendered_prompt="p", output="Answer: maybe"), + ] + result = mine_contract( + _mine_input(examples, axis_b_repeats=0), + call=_make_call(), + entails=_exact_match, + budget=BudgetTracker(budget_usd=10.0), + ) + regex_specs = [inv for inv in result.invariants if inv.kind == "regex"] + assert regex_specs, "expected regex assertion for common prefix" + assert "Answer" in regex_specs[0].spec["pattern"] + + +def test_no_invariants_for_heterogeneous_json_keys() -> None: + examples = [ + MineExample(input={}, rendered_prompt="p", output=json.dumps({"a": 1})), + MineExample(input={}, rendered_prompt="p", output=json.dumps({"b": 2})), + ] + result = mine_contract( + _mine_input(examples, axis_b_repeats=0), + call=_make_call(), + entails=_exact_match, + budget=BudgetTracker(budget_usd=10.0), + ) + req_keys = [inv for inv in result.invariants if inv.kind == "required_keys"] + assert not req_keys, "no required_keys when no common keys across outputs" + + +# --------------------------------------------------------------------------- +# Axis B noise floor +# --------------------------------------------------------------------------- + + +def test_axis_b_stable_output_gives_low_noise_floor() -> None: + examples = [MineExample(input={}, rendered_prompt="p", output="same output")] + result = mine_contract( + _mine_input(examples, axis_b_repeats=3), + call=_make_call(output="same output"), # Axis B always returns same string + entails=_exact_match, + budget=BudgetTracker(budget_usd=10.0), + ) + assert result.noise_floor < 0.5 # all Axis B outputs cluster together + + +def test_axis_b_varying_output_gives_high_noise_floor() -> None: + examples = [MineExample(input={}, rendered_prompt="p", output="baseline")] + call_count = [0] + + def varying_call(model, messages, **kw): + call_count[0] += 1 + return LLMResponse( + content=f"unique output {call_count[0]}", + model=model, + provider="fake", + usage=_fake_usage(), + cost_usd=0.001, + latency_ms=1.0, + finish_reason="stop", + ) + + result = mine_contract( + _mine_input(examples, axis_b_repeats=3), + call=varying_call, + entails=_exact_match, # exact match → each unique output is its own cluster + budget=BudgetTracker(budget_usd=10.0), + ) + assert result.noise_floor > 0.0 + + +# --------------------------------------------------------------------------- +# Axis B only noise (not an invariant when absent from Axis A) +# --------------------------------------------------------------------------- + + +def test_axis_b_only_variance_does_not_create_invariant() -> None: + """When Axis A outputs are heterogeneous, no invariant should be emitted + even if Axis B is stable — the invariant must appear in Axis A.""" + examples = [ + MineExample(input={}, rendered_prompt="p", output=json.dumps({"x": 1})), + MineExample(input={}, rendered_prompt="p", output=json.dumps({"y": 2})), + ] + result = mine_contract( + _mine_input(examples, axis_b_repeats=0), + call=_make_call(output=json.dumps({"x": 1})), + entails=_exact_match, + budget=BudgetTracker(budget_usd=10.0), + ) + req_keys = [inv for inv in result.invariants if inv.kind == "required_keys"] + assert not req_keys + + +# --------------------------------------------------------------------------- +# MinedContract fields +# --------------------------------------------------------------------------- + + +def test_samples_used_counts_both_axes() -> None: + examples = [MineExample(input={}, rendered_prompt="p", output="out")] + result = mine_contract( + _mine_input(examples, axis_b_repeats=2), + call=_make_call(), + entails=_exact_match, + budget=BudgetTracker(budget_usd=10.0), + ) + assert result.axis_a_count == 1 + assert result.axis_b_count == 2 + assert result.samples_used == 3 + + +def test_budget_exhausted_skips_axis_b() -> None: + examples = [MineExample(input={}, rendered_prompt="p", output="out")] + budget = BudgetTracker(budget_usd=0.001) + budget.record_spend(0.001) # pre-exhaust + result = mine_contract( + _mine_input(examples, axis_b_repeats=3), + call=_make_call(), + entails=_exact_match, + budget=budget, + ) + assert result.axis_b_count == 0 + assert result.axis_a_count == 1 diff --git a/packages/core/tests/test_nli.py b/packages/core/tests/test_nli.py new file mode 100644 index 0000000..0169aa5 --- /dev/null +++ b/packages/core/tests/test_nli.py @@ -0,0 +1,54 @@ +"""Tests for reprompt_core.nli — NLI cross-encoder entailment module. + +All unit tests use a fake callable instead of loading any model, so the +suite runs without sentence_transformers installed. Live model tests are +skipped when the package is absent. +""" + +from __future__ import annotations + +import pytest + +from reprompt_core.nli import NLI_AVAILABLE, entailment_score, entails, nli_label + + +def test_nli_available_is_bool() -> None: + assert isinstance(NLI_AVAILABLE, bool) + + +@pytest.mark.skipif(not NLI_AVAILABLE, reason="sentence_transformers not installed") +def test_entailment_score_identical_strings_is_high() -> None: + score = entailment_score("The sky is blue.", "The sky is blue.") + assert score >= 0.5 + + +@pytest.mark.skipif(not NLI_AVAILABLE, reason="sentence_transformers not installed") +def test_entailment_score_range() -> None: + score = entailment_score("Paris is in France.", "Paris is a European city.") + assert 0.0 <= score <= 1.0 + + +@pytest.mark.skipif(not NLI_AVAILABLE, reason="sentence_transformers not installed") +def test_nli_label_returns_valid_label() -> None: + label = nli_label("All birds can fly.", "Penguins can fly.") + assert label in ("entailment", "neutral", "contradiction") + + +@pytest.mark.skipif(not NLI_AVAILABLE, reason="sentence_transformers not installed") +def test_entails_returns_bool() -> None: + result = entails("The cat sat on the mat.", "There is a cat.") + assert isinstance(result, bool) + + +def test_entailment_score_raises_on_empty_premise() -> None: + if not NLI_AVAILABLE: + pytest.skip("sentence_transformers not installed") + with pytest.raises(ValueError, match="premise"): + entailment_score("", "something") + + +def test_entailment_score_raises_on_empty_hypothesis() -> None: + if not NLI_AVAILABLE: + pytest.skip("sentence_transformers not installed") + with pytest.raises(ValueError, match="hypothesis"): + entailment_score("something", " ") diff --git a/packages/core/tests/test_optimizer_loop.py b/packages/core/tests/test_optimizer_loop.py index bb79888..fc5fa9b 100644 --- a/packages/core/tests/test_optimizer_loop.py +++ b/packages/core/tests/test_optimizer_loop.py @@ -407,6 +407,27 @@ def fake_similarity(benchmark_or_existing, candidate, **kw): assert "near-duplicate variant" not in attempted_variants +def test_stage_attempt_carries_target_model() -> None: + """Every StageAttempt passed to on_attempt must include the target_model + string from StageOptimizationInput — so callers can persist which model + produced each Candidate without reading it from context.""" + call, _captured = _make_call(mutation_variants=["variant A"]) + budget = BudgetTracker(budget_usd=10.0) + observed_target_models: list[str] = [] + + def on_attempt(attempt): + observed_target_models.append(attempt.target_model) + + run_optimizer( + [_stage()], call=call, budget=budget, judge_model=JUDGE_MODEL, + strategy="simple", max_sweep_candidates_per_prompt=1, parity_threshold=0.0, + on_attempt=on_attempt, + ) + + assert len(observed_target_models) > 0 + assert all(m == TARGET_MODEL for m in observed_target_models) + + def test_prism_one_stage_failure_does_not_abort_run(monkeypatch: pytest.MonkeyPatch) -> None: def broken_embedding_similarity(*args, **kwargs): raise RuntimeError("simulated unexpected failure") @@ -525,3 +546,74 @@ def test_on_phase_is_optional_and_defaults_to_no_op() -> None: ) assert result.stage_results[0].error is None + + +# --------------------------------------------------------------------------- +# M4 — Holdout evaluation +# --------------------------------------------------------------------------- + + +def test_holdout_score_is_set_when_holdout_examples_are_provided() -> None: + """After optimization, the winner is scored on holdout examples it + was never trained on. holdout_score must be a float in [0, 1].""" + call, _ = _make_call(mutation_variants=["variant A"]) + holdout = [{"input": {"document": "Q2 revenue was $2.1M EUR."}, "output": '{"currency": "EUR", "revenue": 2100000}'}] + stage = StageOptimizationInput( + stage_id=1, + stage_name="Extract", + original_prompt_template="Extract data from: {{document}}", + target_model=TARGET_MODEL, + rubric=EMPTY_RUBRIC, + examples=EXAMPLES, + holdout_examples=holdout, + ) + budget = BudgetTracker(budget_usd=10.0) + result = run_optimizer( + [stage], call=call, budget=budget, judge_model=JUDGE_MODEL, + strategy="simple", max_sweep_candidates_per_prompt=1, parity_threshold=0.0, + num_prompt_variants=1, + ) + + sr = result.stage_results[0] + assert sr.error is None + assert sr.best is not None + assert sr.holdout_score is not None + assert 0.0 <= sr.holdout_score <= 1.0 + + +def test_holdout_score_is_none_when_no_holdout_examples_given() -> None: + """No holdout_examples → holdout_score stays None.""" + call, _ = _make_call(mutation_variants=["variant A"]) + budget = BudgetTracker(budget_usd=10.0) + result = run_optimizer( + [_stage()], call=call, budget=budget, judge_model=JUDGE_MODEL, + strategy="simple", max_sweep_candidates_per_prompt=1, parity_threshold=0.0, + num_prompt_variants=1, + ) + assert result.stage_results[0].holdout_score is None + + +def test_holdout_score_is_none_when_stage_produces_no_winner() -> None: + """If optimization fails (no winner), holdout evaluation is skipped.""" + def always_fail(model, messages, **kwargs): + raise RuntimeError("network down") + + holdout = [{"input": {"document": "Q3 data."}, "output": '{"currency": "USD", "revenue": 0}'}] + stage = StageOptimizationInput( + stage_id=1, + stage_name="Extract", + original_prompt_template="Extract: {{document}}", + target_model=TARGET_MODEL, + rubric=EMPTY_RUBRIC, + examples=EXAMPLES, + holdout_examples=holdout, + ) + budget = BudgetTracker(budget_usd=10.0) + result = run_optimizer( + [stage], call=always_fail, budget=budget, judge_model=JUDGE_MODEL, + strategy="simple", max_sweep_candidates_per_prompt=1, parity_threshold=0.0, + num_prompt_variants=1, + ) + sr = result.stage_results[0] + assert sr.best is None + assert sr.holdout_score is None diff --git a/packages/core/tests/test_seam.py b/packages/core/tests/test_seam.py new file mode 100644 index 0000000..ac28140 --- /dev/null +++ b/packages/core/tests/test_seam.py @@ -0,0 +1,134 @@ +"""Unit tests for reprompt_core.optimizer.seam — Phase 4 seam regression.""" + +from __future__ import annotations + +from dataclasses import dataclass + +from reprompt_core.budget import BudgetTracker +from reprompt_core.llm.client import LLMResponse +from reprompt_core.optimizer.seam import SeamExample, SeamInput, evaluate_seam +from reprompt_core.trace import TokenUsage + + +def _fake_usage() -> TokenUsage: + return TokenUsage(input=10, output=10, thinking=None) + + +def _make_call(output: str = "downstream output", cost: float = 0.001): + """Fake call: returns a fixed LLMResponse regardless of model/messages.""" + def call(model: str, messages: list, **kw) -> LLMResponse: + return LLMResponse( + content=output, + model=model, + provider="fake", + usage=_fake_usage(), + cost_usd=cost, + latency_ms=1.0, + finish_reason="stop", + ) + return call + + +def _example( + up_input: dict | None = None, + up_output: str = "upstream output", + down_input: dict | None = None, + down_output: str = "downstream output", +) -> SeamExample: + return SeamExample( + upstream_input=up_input or {"q": "hello"}, + upstream_baseline_output=up_output, + downstream_input=down_input or {"root": "upstream output"}, + downstream_baseline_output=down_output, + ) + + +def _seam_input(examples: list[SeamExample], threshold: float = 0.5) -> SeamInput: + return SeamInput( + upstream_stage_id=1, + upstream_source_id="root", + upstream_winning_prompt="{{q}}", + upstream_target_model="gpt-4o-mini", + upstream_params={"temperature": 0.0}, + downstream_stage_id=2, + downstream_original_prompt="{{root}}", + downstream_original_model="gpt-4o-mini", + downstream_rubric={}, + examples=examples, + parity_threshold=threshold, + ) + + +def test_seam_pass_when_downstream_output_matches_baseline(): + """Identical output → score > threshold (0.5) → passed. + + Seam scoring uses det+embedding only (no judge), so the max composite is + ~0.55 (det×0.25 + embedding×0.30). We check > 0.5, not > 0.9. + """ + call = _make_call(output="downstream output") + budget = BudgetTracker(budget_usd=10.0) + result = evaluate_seam(_seam_input([_example()]), call=call, budget=budget) + assert result.passed + assert result.parity_score is not None + assert result.parity_score > 0.5 + + +def test_seam_fail_when_downstream_output_diverges(): + """Completely different output should fail at a high threshold.""" + call = _make_call(output="COMPLETELY UNRELATED GIBBERISH XYZ 12345") + budget = BudgetTracker(budget_usd=10.0) + result = evaluate_seam(_seam_input([_example()], threshold=0.99), call=call, budget=budget) + assert not result.passed + + +def test_seam_substitution_applied_when_upstream_source_id_in_downstream_input(): + """upstream_source_id key present in downstream_input → substitution_applied=True.""" + call = _make_call(output="downstream output") + budget = BudgetTracker(budget_usd=10.0) + ex = _example(down_input={"root": "old upstream output"}) + result = evaluate_seam(_seam_input([ex]), call=call, budget=budget) + assert result.substitution_applied + + +def test_seam_no_substitution_when_key_absent(): + """Key absent from downstream_input → substitution_applied=False (stability check only).""" + call = _make_call(output="downstream output") + budget = BudgetTracker(budget_usd=10.0) + ex = _example(down_input={"some_other_key": "value"}) + result = evaluate_seam(_seam_input([ex]), call=call, budget=budget) + assert not result.substitution_applied + + +def test_seam_none_score_when_all_calls_fail(): + """All-failing call → parity_score=None, passed=False.""" + def failing_call(model, messages, **kw): + raise RuntimeError("network error") + + budget = BudgetTracker(budget_usd=10.0) + result = evaluate_seam(_seam_input([_example()]), call=failing_call, budget=budget) + assert result.parity_score is None + assert not result.passed + + +def test_seam_budget_exhausted_before_first_example(): + """Pre-exhausted budget → parity_score=None.""" + call = _make_call() + budget = BudgetTracker(budget_usd=0.001) + budget.record_spend(0.001) # pre-exhaust + assert budget.is_exhausted + result = evaluate_seam(_seam_input([_example(), _example()]), call=call, budget=budget) + assert result.parity_score is None + + +def test_seam_multiple_examples_mean_score(): + """Mean of per-example scores is returned; budget is consumed.""" + call = _make_call(output="downstream output") + budget = BudgetTracker(budget_usd=10.0) + result = evaluate_seam( + _seam_input([_example(), _example(), _example()]), + call=call, + budget=budget, + ) + assert result.parity_score is not None + assert 0.0 <= result.parity_score <= 1.0 + assert budget.spent_usd > 0.0