feat: add reflection probing and steering pipelines

[xzAscC]

Summary

Adds end-to-end inference, LLM-as-judge evaluation, reflection diagnosis, steering-vector extraction/inference, linear probing, and ROSCOE reasoning-quality analysis. Shared model, batching, prompt, and judge utilities reduce duplication; deterministic tests mock external datasets and cover inference limits, token-only decoding, and OOM recovery. Repository ignore rules keep local research data and workflow state off GitHub while .ignore preserves OpenCode/ripgrep searchability.

Linked issue

N/A — no repository issue is associated with this work.

Verification

• uv sync — passed (176 packages resolved)
• uv run ruff check src/ tests/ scripts/ — passed, 0 violations
• uv run ruff format --check src/ tests/ scripts/ — passed, 30 files formatted
• uv run mypy src/ — passed, 0 errors in 15 source files
• uv run pytest — passed, 159 tests
• bash -n scripts/probing_reflection/*.sh — passed
• Second-round review: goal/constraints PASS, QA PASS, code quality PASS, security PASS, PR context PASS

Self-review checklist

• Changes are split into focused, reviewable commits
• Tests cover happy paths, edge cases, and failure paths
• External dataset/model boundaries are mocked in tests
• No Any or type-error suppressions remain in src/
• Local data, generated figures, notebooks, and agent state remain untracked
• README citation updated to arXiv:2506.12217

Risks and considerations

• Real experiment commands still require access to the configured Hugging Face models and sufficient GPU resources.
• Cross-family judge scripts expect local inputs under the gitignored data/hard_negatives/ directory.
• The PR removes the previously tracked .sisyphus draft and keeps .sisyphus/ local-only by design.

---

Summary by cubic

Adds an end-to-end pipeline for reflection probing and activation steering, including batch inference, LLM-as-judge evaluation, reflection diagnosis, steering-vector extraction/application, linear probing, and ROSCOE reasoning-quality scoring. Updates the README with pipeline overviews and example commands for running experiments on MATH-500, AIME, and GPQA.

• New Features

  • CLI probing-reflection with commands: inference, evaluate, reflection-diagnose, extract-vectors.
  • Steering vectors: extraction from R/N sets, 4-bit steering inference with hooks, and multi-dataset adapters (MATH-500, AIME, GPQA).
  • Linear probe training/eval with metrics and plot exports; weights saved to .npz.
  • LLM-as-judge utilities for answer comparison and reflection detection; cross-family judge scripts and result merger.
  • ROSCOE-based reasoning quality scoring and report generator.
  • Documentation and shared utils for model loading, batching, prompts; extensive tests and sample fixtures.
  • Repo hygiene: expanded .gitignore and .ignore to keep local research data untracked but searchable.

• Dependencies

  • Adds transformers, torch, datasets, tqdm, accelerate, bitsandbytes, scikit-learn, matplotlib, and runtime jupyter.

^{Written for commit 37b4170. Summary will update on new commits.}