chore(evaluator): vendor code-evaluator + float-tol + scicode support#41
chore(evaluator): vendor code-evaluator + float-tol + scicode support#41jack-scitix-ai wants to merge 5 commits into
Conversation
…p submodule) Convert submodules/code-evaluator (git submodule -> scitix/code-evaluator) into a self-contained in-tree copy under vendor/code-evaluator, byte-identical to the previously pinned commit e4802268. Removes .gitmodules (was the only submodule) so a plain clone of sieval is self-contained. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…olerance Port fix/checker-messages-float-tol (cfc47d8) onto the vendored evaluator: LiveCodeBench output comparison now reports got-vs-expected on mismatch, and an opt-in CODE_EVAL_FLOAT_TOL env var enables tolerant float compare (default off = exact Decimal ==, matching official LiveCodeBench). Tolerance applies only to fractional expected values; integer expectations stay exact. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Route source="scicode" through the direct-execution path (scicode sends a self-contained program with inlined test cases, so a clean run == pass, same as human-eval), add requirements/scicode.txt with the scientific stack the problems import (numpy / scipy / sympy / matplotlib) pinned to the env used to validate SciCode on Kimi-K2.6, and docker/Dockerfile.scicode (python:3.11-slim, since scipy 1.16.x needs Python >= 3.11). venv install remains supported per README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Vendored code lives in-tree now, so ruff/ty/mypy would lint the third-party source. Point the existing "submodules" excludes at "vendor" (the old submodules/ path no longer exists after the vendoring). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pre-commit hooks (end-of-file-fixer, trailing-whitespace, ruff, markdownlint) run on explicit file lists and don't honor the pyproject ruff/ty excludes, so they were reformatting the vendored third-party code in vendor/. Add vendor/ to the pre-commit global exclude (alongside sieval/community/) to keep vendored files byte-identical to upstream. Also wrap a bare URL in VENDORED.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1c3f387 to
c1c9990
Compare
ethan-scitix
left a comment
There was a problem hiding this comment.
Bug (blocking): Dockerfile:21 — COPY submodules /app/submodules is now dangling.
This PR removes submodules/ (gitlink + .gitmodules) but doesn't touch the Dockerfile. A fresh clone has no submodules/, so docker build fails on this line — defeating the PR's "self-contained clone" goal. It only builds locally because of an untracked leftover. The image bundles the eval service (L9 installs fastapi[standard]+psutil, which sieval itself doesn't use), so fix to COPY vendor /app/vendor (and update the launch path). Please pull the Dockerfile into this PR.
Question: float-tol paths disagree on "integer expected value" (vendor/code-evaluator/app/exec_py_test.py). _decimals_match (stdio) is value-based (b.to_integral_value() → 1.0 treated as integer, exact), while _value_close (fn-call, L54) is type-based (isinstance(b, int) → float 1.0 gets tolerance). Same logical 1.0 compares exact in one path, tolerant in the other. Opt-in only (default off) and byte-identical to upstream cfc47d8, so not blocking — worth aligning upstream or noting in _value_close's docstring that it keys off type.
Nit: VENDORED.md is bloated + partly wrong — slim it to provenance or drop it. Keep only what git can't: upstream URL + pinned SHA + patch list with source commits. Current issues: "internal scitix repo" is wrong (scitix/code-evaluator is public); the modifications list omits README.md (also patched); the service description and scipy/Python-3.11 note duplicate the vendored README and the requirements/scicode.txt + Dockerfile.scicode comments.
Type
Summary
code-evaluatorfrom a git submodule (submodules/code-evaluator→scitix/code-evaluator) to a self-contained vendored copy undervendor/code-evaluator, byte-identical to the previously pinned commite4802268. A plaingit cloneis now self-contained (nogit submodule update). Requested in review.float-tol(fix/checker-messages-float-tol,cfc47d8): clearer LiveCodeBench output-comparison messages + opt-in float tolerance viaCODE_EVAL_FLOAT_TOL(default off = exact==, matching official LiveCodeBench).source="scicode"through the direct-run path, addrequirements/scicode.txt(numpy/scipy/sympy/matplotlib, pinned) anddocker/Dockerfile.scicode(python:3.11-slim, since scipy 1.16.x needs Python ≥ 3.11). venv install remains supported.excludefromsubmodules/tovendor/so the vendored third-party code isn't linted/type-checked.vendor/code-evaluator/VENDORED.md;code-evaluatoris an internal scitix repo with no separate LICENSE. No AI-generated Python modules in this PR (vendored third-party + config/docs only).Test Plan
Automated
ruff check && ruff format --check)ty checkormypy --strict)pdm run pytest)Manual
ty check sievalclean;ruff/formatclean withvendor/excluded;pdm run pytest tests/unit→ 2379 passed. Vendored tree verified byte-identical toe4802268(+ documented float-tol/scicode patches);app/*.pycompile;source="scicode"alias present.Checklist
Required (all PRs)
type(scope): description)AI-Generated Code - <model> (<provider>)in module docstringcore/