Skip to content

chore(evaluator): vendor code-evaluator + float-tol + scicode support#41

Open
jack-scitix-ai wants to merge 5 commits into
scitix:mainfrom
jack-scitix-ai:evaluator
Open

chore(evaluator): vendor code-evaluator + float-tol + scicode support#41
jack-scitix-ai wants to merge 5 commits into
scitix:mainfrom
jack-scitix-ai:evaluator

Conversation

@jack-scitix-ai

Copy link
Copy Markdown
Contributor

Type

  • chore — CI, tooling, dependencies, config

Summary

  • Convert code-evaluator from a git submodule (submodules/code-evaluatorscitix/code-evaluator) to a self-contained vendored copy under vendor/code-evaluator, byte-identical to the previously pinned commit e4802268. A plain git clone is now self-contained (no git submodule update). Requested in review.
  • Port float-tol (fix/checker-messages-float-tol, cfc47d8): clearer LiveCodeBench output-comparison messages + opt-in float tolerance via CODE_EVAL_FLOAT_TOL (default off = exact ==, matching official LiveCodeBench).
  • Add SciCode execution support: route source="scicode" through the direct-run path, add requirements/scicode.txt (numpy/scipy/sympy/matplotlib, pinned) and docker/Dockerfile.scicode (python:3.11-slim, since scipy 1.16.x needs Python ≥ 3.11). venv install remains supported.
  • Retarget ruff/ty/mypy exclude from submodules/ to vendor/ so the vendored third-party code isn't linted/type-checked.
  • Provenance recorded in vendor/code-evaluator/VENDORED.md; code-evaluator is an internal scitix repo with no separate LICENSE. No AI-generated Python modules in this PR (vendored third-party + config/docs only).
  • The sieval-side SciCode dataset/task lands in a follow-up PR that depends on this one.

Test Plan

Automated

  • Lint/format clean (ruff check && ruff format --check)
  • Type check clean (ty check or mypy --strict)
  • Unit tests pass (pdm run pytest)

Manual

  • ty check sieval clean; ruff/format clean with vendor/ excluded; pdm run pytest tests/unit → 2379 passed. Vendored tree verified byte-identical to e4802268 (+ documented float-tol/scicode patches); app/*.py compile; source="scicode" alias present.

Checklist

Required (all PRs)

  • PR title follows conventional format (type(scope): description)
  • No internal paths, credentials, or personal info in committed files
  • AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
  • No new upper-layer dependencies added to core/
  • Deleted code verified — no remaining call sites depend on it

jack-scitix-ai and others added 5 commits July 21, 2026 17:43
…p submodule)

Convert submodules/code-evaluator (git submodule -> scitix/code-evaluator) into
a self-contained in-tree copy under vendor/code-evaluator, byte-identical to the
previously pinned commit e4802268. Removes .gitmodules (was the only submodule)
so a plain clone of sieval is self-contained.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…olerance

Port fix/checker-messages-float-tol (cfc47d8) onto the vendored evaluator:
LiveCodeBench output comparison now reports got-vs-expected on mismatch, and an
opt-in CODE_EVAL_FLOAT_TOL env var enables tolerant float compare (default off =
exact Decimal ==, matching official LiveCodeBench). Tolerance applies only to
fractional expected values; integer expectations stay exact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Route source="scicode" through the direct-execution path (scicode sends a
self-contained program with inlined test cases, so a clean run == pass, same as
human-eval), add requirements/scicode.txt with the scientific stack the problems
import (numpy / scipy / sympy / matplotlib) pinned to the env used to validate
SciCode on Kimi-K2.6, and docker/Dockerfile.scicode (python:3.11-slim, since
scipy 1.16.x needs Python >= 3.11). venv install remains supported per README.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Vendored code lives in-tree now, so ruff/ty/mypy would lint the third-party
source. Point the existing "submodules" excludes at "vendor" (the old submodules/
path no longer exists after the vendoring).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pre-commit hooks (end-of-file-fixer, trailing-whitespace, ruff, markdownlint)
run on explicit file lists and don't honor the pyproject ruff/ty excludes, so they
were reformatting the vendored third-party code in vendor/. Add vendor/ to the
pre-commit global exclude (alongside sieval/community/) to keep vendored files
byte-identical to upstream. Also wrap a bare URL in VENDORED.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@ethan-scitix ethan-scitix left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug (blocking): Dockerfile:21COPY submodules /app/submodules is now dangling.
This PR removes submodules/ (gitlink + .gitmodules) but doesn't touch the Dockerfile. A fresh clone has no submodules/, so docker build fails on this line — defeating the PR's "self-contained clone" goal. It only builds locally because of an untracked leftover. The image bundles the eval service (L9 installs fastapi[standard]+psutil, which sieval itself doesn't use), so fix to COPY vendor /app/vendor (and update the launch path). Please pull the Dockerfile into this PR.


Question: float-tol paths disagree on "integer expected value" (vendor/code-evaluator/app/exec_py_test.py). _decimals_match (stdio) is value-based (b.to_integral_value()1.0 treated as integer, exact), while _value_close (fn-call, L54) is type-based (isinstance(b, int) → float 1.0 gets tolerance). Same logical 1.0 compares exact in one path, tolerant in the other. Opt-in only (default off) and byte-identical to upstream cfc47d8, so not blocking — worth aligning upstream or noting in _value_close's docstring that it keys off type.


Nit: VENDORED.md is bloated + partly wrong — slim it to provenance or drop it. Keep only what git can't: upstream URL + pinned SHA + patch list with source commits. Current issues: "internal scitix repo" is wrong (scitix/code-evaluator is public); the modifications list omits README.md (also patched); the service description and scipy/Python-3.11 note duplicate the vendored README and the requirements/scicode.txt + Dockerfile.scicode comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants