LLMCodeProbe

Find bugs in LLM-generated code that test suites miss.

Every major LLM code benchmark (HumanEval, MBPP, SWE-Bench) evaluates correctness by running test cases. Pass the tests = correct. But test suites are finite — they can't cover all inputs. LLMCodeProbe uses symbolic execution, property-based testing, and boundary analysis to find bugs in code that passes all tests.

The correctness gap

correctness_gap = solutions_with_hidden_bugs / solutions_passing_tests

A correctness gap of 0.15 means 15% of "passing" LLM solutions contain bugs that no test caught — but symbolic execution or property-based testing did.

How it works

LLM-generated solution (passes all tests)
                │
    ┌───────────┼───────────┐
    ▼           ▼           ▼
CrossHair   Hypothesis   Boundary
(symbolic    (property-   (edge case
execution)   based test)  analysis)
    │           │           │
    └───────────┼───────────┘
                ▼
        ProbeResult {
          passes_tests: true,
          bugs: [
            {counterexample: "x=-2147483648", ...},
            {counterexample: "lst=[]", ...},
          ]
        }

Analyzers

Analyzer	Technique	Finds
CrossHair	Symbolic execution (Z3)	Postcondition violations, type errors for all possible inputs
Hypothesis	Property-based testing	Crashes on random inputs, shrunk to minimal counterexamples
Boundary	Edge case injection	Off-by-one, empty input, overflow, recursion depth issues

Each bug comes with a concrete counterexample — a specific input that triggers the bug.

Quick start

pip install -e .

# Probe HumanEval solutions
python -m llmcodeprobe.run \
    --problems datasets/humaneval/HumanEval.jsonl \
    --solutions datasets/humaneval/solutions-claude.jsonl \
    --model claude-sonnet-4-6 \
    --output results/claude/

# Probe a directory of solution files
python -m llmcodeprobe.run \
    --solutions-dir my_solutions/ \
    --model gpt-4o \
    --output results/gpt4o/

Output

============================================================
  LLMCodeProbe Correctness Gap Report — claude-sonnet-4-6
============================================================
  Solutions analyzed:    164
  Pass test suite:       158
  Have hidden bugs:      23
  CORRECTNESS GAP:       14.6%
============================================================

  Bugs by category:
    boundary_crash: 12
    symbolic_violation: 8
    property_violation: 3

  Bugs found by:
    boundary: 12
    crosshair: 8
    hypothesis: 3

Project structure

llmcodeprobe/
├── llmcodeprobe/
│   ├── analyzers/
│   │   ├── base.py                 # Analyzer ABC + Bug dataclass
│   │   ├── crosshair_analyzer.py   # Symbolic execution via CrossHair/Z3
│   │   ├── hypothesis_analyzer.py  # Property-based testing
│   │   └── boundary.py             # Boundary value edge cases
│   ├── benchmarks/
│   │   └── loader.py               # HumanEval/MBPP solution loader
│   ├── reports/
│   │   └── gap.py                  # Correctness gap computation
│   ├── probe.py                    # Core probing pipeline
│   └── run.py                      # CLI entry point
├── datasets/
│   ├── humaneval/
│   └── mbpp/
└── tests/