Six reproducible defects in the DE-Arch evaluation suite (English tasks always score 0; 7/30 English gold rubrics truncated)

I opened #7 earlier today for the first of these and closed it myself — this issue supersedes it and reports everything I found in one place.

Not duplicating #2 (result confusion), #3 (RPM), #5 (gold self-consistency) or #6 (time drift).

**Every finding below is verifiable offline.** No API key, no network, no third-party packages, no LLM call — each one reads only files that ship with the repository, so two people on the same commit get byte-identical output. A script that checks all of them is in the appendix.

Context: I reproduced DE-Arch end to end (5 models × 60 tasks = 300 evaluations, English + Chinese) and ran the DE-Impl / DE-Evol and DA pipelines. The six items below are what had to be changed before any of it would run.

---

## How to verify

Save the appendix script as `verify_findings.py` and run it against a clean checkout:

```bash
python verify_findings.py /path/to/DAComp
```

Expected output on the current `main`:

```
  [x] 1   DE-Arch scorer only reads the Chinese total key
  [x] 1b  English and Chinese prompts request different keys
  [x] 2   English gold rubrics are truncated
  [x] 3   Default gold path misses one directory level
  [x] 4   Azure client class is hardcoded in both entry points
  [x] 5   Truncated judge replies are silently recorded as 0
  [x] 6   DE judge omits temperature while DA judge sets it to 0

  7/7 findings reproduce on this checkout
```

Exit code is 0 when everything reproduces. Six defects, seven checks — 1 and 1b are two halves of the same mismatch.

---

## 1. `extract_actual_score()` reads only the Chinese total key, so every English DE-Arch task is recorded as 0

`dacomp-de/evaluation_suite_arch/evaluate.py:280-290`

```python
def extract_actual_score(self, evaluation_result: Dict[str, Any]) -> int:
    """Extract actual score from evaluation result (retains CN keys)."""
    if "总得分" in evaluation_result:
        return evaluation_result["总得分"]
    elif "parse_error" not in evaluation_result:
        total_score = 0
        for value in evaluation_result.values():
            if isinstance(value, dict) and "总得分" in value:
                total_score += value["总得分"]
        return total_score
    return 0
```

Both branches test for `总得分` only. The docstring says as much: *"retains CN keys"*.

**1b.** The two prompt templates ask the judge for **different** key names:

| template | line | requested key |
|---|---|---|
| `eval_prompt` (English, `utils/eval_prompt.py:1`) | 65, 83 | `"Total Score": int` |
| `eval_prompt_zh` (Chinese, `utils/eval_prompt.py:89`) | 153, 171 | `"总得分": int` |

So an English judge response has no `总得分` anywhere, falls through both branches, and returns `0`.

**Verify** — call the function with each shape:

```python
zh = {"Requirement 1": {"总得分": 7},      "总得分": 12}
en = {"Requirement 1": {"Total Score": 7}, "Total Score": 12}
extract_actual_score(zh)  #  12
extract_actual_score(en)  #   0   <- both keys present, both ignored
```

**Impact.** With the released code the English DE-Arch column cannot produce a non-zero score for any model, regardless of what the judge actually decided.

**Suggested fix.** Accept both names in both branches:

```python
TOTAL_KEYS = ("总得分", "Total Score", "total_score")
```

`extract_max_score_from_rubric()` in the same file already handles both `总分` and `Total Score`, so the bilingual intent is present — this looks like one function that was missed.

---

## 2. 7 of 30 English gold rubrics are truncated

`dacomp-de/evaluation_suite_arch/gold/dacomp-arch-gold.jsonl`

Each rubric opens with a list of its three fixed requirements and a total score, then gives the criteria under `## Requirement I/II/III` headings. In seven files the declared requirements are not all present:

| id | declared | missing |
|---|---|---|
| dacomp-de-arch-002 | I, II, III | **II, III** |
| dacomp-de-arch-006 | I, II, III | **II, III** |
| dacomp-de-arch-007 | I, II, III | **II, III** |
| dacomp-de-arch-012 | I, II, III | **II, III** |
| dacomp-de-arch-016 | I, II, III | **II, III** |
| dacomp-de-arch-017 | I, II, III | **II, III** |
| dacomp-de-arch-019 | I, II, III | **III** |

The last 46 characters of each rubric, verbatim (`rubric.rstrip()[-46:]`):

```
dacomp-de-arch-002   nt metrics【1 Point｜Criteria】Presence of `power
dacomp-de-arch-006   teger, non-null, unique, and joinable with the
dacomp-de-arch-007   e name fields + `platform`, with platform enum
dacomp-de-arch-012   : [1 point | Supplemental] `total_open_actions
dacomp-de-arch-016   rview`, `workday__position_overview`).

                     - Path
dacomp-de-arch-019    have an unknown type.

                     ### Criterion 2.2 (Max
```

None of these end at a section boundary. `002` stops inside a backtick-quoted
identifier, partway through `power`; `007` stops mid-sentence after "platform
enum"; `012` stops partway through `total_open_actions`. `016` and `019` are
the clearest:
`016` ends just after a `- Path` bullet begins, and `019` ends in the middle of
the heading `### Criterion 2.2 (Max` — a markdown heading cut in half. These are
mid-document cuts that leave the opening fragment of the next section behind,
not sections that were cleanly omitted.

**Verify:**

```python
import json, re
for line in open("dacomp-arch-gold.jsonl", encoding="utf-8"):
    d = json.loads(line)
    declared = re.findall(r'^-\s*Requirement\s+(I{1,3})\s*[:：]', d["rubric"], re.M)
    present  = re.findall(r'^##\s*Requirement\s+(I{1,3})\s*[:：]', d["rubric"], re.M)
    missing  = [r for r in dict.fromkeys(declared) if r not in set(present)]
    if missing:
        print(d["id"], "missing", missing)
```

**Impact.** `extract_max_score_from_rubric()` takes the denominator from the header, which counts points for requirements that have no criteria in the file. On these seven tasks the score rate is divided by a total the judge was never given a way to award.

**Note.** `dacomp-arch-zh-gold.jsonl` is complete for all 30, and structurally parallel on these seven — same declared points per requirement, same number of 1-point criteria per section. That suggests the English rubrics were complete upstream and the loss happened when the data was packaged. Consistent with this, the reported English (Table 3) and Chinese (Table 4) figures are close to each other, which would be hard to obtain if 23% of English tasks had an inflated denominator.

I reconstructed the missing English sections from the Chinese parallel text so I could run the suite locally, and can open a PR with them — though publishing the originals would obviously be better.

---

## 3. Default gold path is missing one directory level

`dacomp-de/evaluation_suite_arch/utils/config.py:5-8`

```python
BASE_DIR = Path(__file__).resolve().parent          # -> evaluation_suite_arch/utils
DEFAULT_RESULTS_DIR   = BASE_DIR / "results"
DEFAULT_GOLD_EN_JSONL = BASE_DIR / "gold" / "dacomp-arch-gold.jsonl"
DEFAULT_GOLD_ZH_JSONL = BASE_DIR / "gold" / "dacomp-arch-zh-gold.jsonl"
```

`config.py` lives in `utils/`, so `BASE_DIR` is `utils/` — but `gold/` sits one level up:

```
evaluation_suite_arch/utils/gold/dacomp-arch-gold.jsonl   exists=False   <- default
evaluation_suite_arch/gold/dacomp-arch-gold.jsonl         exists=True    <- actual
```

The defaults can never resolve, so `--gold-en-jsonl` / `--gold-zh-jsonl` must always be passed explicitly. `DEFAULT_RESULTS_DIR` has the same offset.

**Suggested fix.** `SUITE_DIR = BASE_DIR.parent`, and build the gold and results paths from that.

---

## 4. Both entry points hardcode `openai.AzureOpenAI`

- `dacomp-de/evaluation_suite_arch/evaluate.py:153`
- `methods/de-agent/evaluation/benchmarks/dacomp/run_infer_de_arch.py:80`

```python
client = openai.AzureOpenAI(
    azure_endpoint=config["base_url"],
    api_version=config["api_version"],
    api_key=config["api_key"],
    timeout=300
)
```

`AzureOpenAI` builds `/openai/deployments/<model>/chat/completions?api-version=...`. Any other OpenAI-compatible endpoint returns 404, so neither script can reach a non-Azure provider without editing the source.

This matters because the shipped config has no endpoint to begin with — `config.py:11` is `DEFAULT_BASE_URL = ""`, and all 17 model entries point at it. Anyone running the suite has to supply their own provider, and the pinned client class is what stops them.

`evaluate.py:174` also still carries `"extra_headers": {"X-TT-LOGID": ""}` — harmless, but the same root cause: the file was exported from an internal environment without an adaptation pass for external use.

**Suggested fix.**

```python
client = openai.OpenAI(base_url=config["base_url"], api_key=config["api_key"], timeout=300)
```

---

## 5. A truncated judge reply is silently recorded as 0, indistinguishable from a real score of 0

`dacomp-de/evaluation_suite_arch/evaluate.py:184-185`

```python
completion = client.chat.completions.create(**api_params)
return completion.choices[0].message.content
```

The response body is returned as-is; `finish_reason` does not appear anywhere in the file. If a reply is cut short, the JSON parse fails and control reaches `evaluate.py:276-278`:

```python
except json.JSONDecodeError as e:
    logger.warning(f"Could not parse JSON response: {e}")
    return {"parse_error": str(e), "raw_content": response_content}
```

`extract_actual_score()` then hits its final `return 0`.

**Verify** — this is a property of the code, not a matter of luck with a provider:

```python
truncated = '{"Requirement 1": {"Criterion 1.1": {"analysis": "The submis'
# json.JSONDecodeError: Unterminated string starting at: line 1 column 50
parsed = {"parse_error": "...", "raw_content": truncated}
extract_actual_score(parsed)   # 0
```

**Impact.** The saved result records `0` with no field distinguishing "the transport dropped the reply" from "the judge awarded nothing". Nothing downstream can tell the two apart, so this class of loss cannot be detected by inspecting the results — only by re-running and comparing.

For what it is worth, I did hit this in practice: my provider intermittently returned bodies of 4 / 19 / 55 / 196 / 238 / 1591 / 3483 / 4589 characters where a complete reply is ~12,000. Adding a completeness check and retry recovered every affected task. But the point above stands without that anecdote.

**Suggested fix.**

```python
choice  = completion.choices[0]
content = choice.message.content or ""
if getattr(choice, "finish_reason", None) not in ("stop", None) or len(content) < 200:
    raise RuntimeError(f"incomplete response: finish_reason={choice.finish_reason}, len={len(content)}")
```

so the existing retry path handles it instead of a partial body reaching the parser. Recording `finish_reason` alongside the score would also make the failure visible after the fact.

Related, in `run_infer_de_arch.py`: `call_gpt_api()` uses `retries=5, backoff_base=1.0` (~31 s of total backoff) and builds its client without a `timeout`, while `evaluate.py` passes `timeout=300`.

---

## 6. The DE judge omits `temperature`; the DA judge in the same repository sets it to 0

`dacomp-de/evaluation_suite_arch/evaluate.py:160-175`

```python
api_params = {
    "model": model_name,
    "messages": [...],
    "max_tokens": config["max_tokens"],
    "extra_headers": {"X-TT-LOGID": ""},
}
```

No `temperature`, so OpenAI-compatible endpoints apply their default of 1.0 and grading is sampled rather than deterministic.

The DA judge does pin it — `dacomp-da/evaluation_suite/core/config.py`, lines 12, 19 and 27:

```python
"generate_kwargs": {"max_tokens": 16384, "temperature": 0},
```

Same repository, two judges, only one is deterministic. Since grading variance is what §3.6 / Table 10 quantifies, the DE side seems like it was meant to be pinned too.

**Suggested fix.** Add `"temperature": 0` to `api_params`, and consider `"response_format": {"type": "json_object"}` on providers that support it — the prompts already require strict JSON.

---

## Summary

| # | Defect | Effect |
|---|---|---|
| 1 | Score extraction is Chinese-key-only | English DE-Arch always 0 |
| 2 | 7/30 English gold rubrics truncated | Denominator counts unawardable points |
| 3 | Default gold path off by one directory | Defaults never resolve |
| 4 | `AzureOpenAI` hardcoded in both entry points | Non-Azure providers unreachable |
| 5 | No `finish_reason` check | Dropped replies indistinguishable from 0 |
| 6 | No `temperature` on the DE judge | Grading is nondeterministic |

Happy to open PRs for any of the code fixes, and to share the reconstructed English rubric sections from item 2.

Thanks for releasing this — the CS / CFS / SR decomposition was genuinely useful to work through, and the cascading-failure framing is the part I got the most out of.

---

<details>
<summary><b>Appendix — <code>verify_findings.py</code></b> (stdlib only, no network, no API key)</summary>

```python
#!/usr/bin/env python3
"""Verify every finding in this report — offline, no API key, stdlib only.

    python verify_findings.py /path/to/DAComp

Every check reads only files that ship with the repository. Nothing calls an LLM,
so two people running this on the same commit get byte-identical output.
Exit code is 0 if all findings reproduce, 1 otherwise.
"""
import ast
import json
import re
import sys
import textwrap
from pathlib import Path

PASS, FAIL = "REPRODUCED", "not reproduced"
results = []


def report(n, title, ok, detail):
    results.append((n, title, ok))
    print(f"\n[{n}] {title}")
    print(f"    {PASS if ok else FAIL}")
    for line in textwrap.dedent(detail).strip().splitlines():
        print(f"    {line}")


def load_function(src_path: Path, name: str):
    """Pull one function out of a module and exec it in isolation.

    Avoids importing the module (which needs openai, pandas, ...) so this script
    stays dependency-free. Works because the target function only uses builtins.
    """
    tree = ast.parse(src_path.read_text(encoding="utf-8"))
    for node in ast.walk(tree):
        if isinstance(node, ast.FunctionDef) and node.name == name:
            mod = ast.Module(body=[node], type_ignores=[])
            ns = {"Dict": dict, "Any": object}
            exec(compile(mod, str(src_path), "exec"), ns)
            return ns[name], ast.get_source_segment(
                src_path.read_text(encoding="utf-8"), node)
    raise LookupError(f"{name} not found in {src_path}")


def f1_english_always_zero(root: Path):
    """The scorer only recognises the Chinese total key."""
    ev = root / "dacomp-de/evaluation_suite_arch/evaluate.py"
    fn, src = load_function(ev, "extract_actual_score")

    zh_style = {"Requirement 1": {"总得分": 7}, "总得分": 12}
    en_style = {"Requirement 1": {"Total Score": 7}, "Total Score": 12}

    got_zh = fn(None, zh_style)
    got_en = fn(None, en_style)

    detail = f"""
        {ev.relative_to(root)}
        docstring: {ast.get_docstring(ast.parse(src).body[0])!r}
        judge returns Chinese shape -> extract_actual_score = {got_zh}   (expected 12)
        judge returns English shape -> extract_actual_score = {got_en}   (expected 12)
    """
    return got_zh == 12 and got_en == 0, detail


def f1b_prompt_keys_differ(root: Path):
    """The English and Chinese prompts request different key names."""
    p = root / "dacomp-de/evaluation_suite_arch/utils/eval_prompt.py"
    txt = p.read_text(encoding="utf-8")
    tree = ast.parse(txt)
    keys = {}
    for node in tree.body:
        if isinstance(node, ast.Assign) and isinstance(node.value, ast.Constant) \
                and isinstance(node.value.value, str):
            name = node.targets[0].id
            if name.startswith("eval_prompt"):
                body = node.value.value
                keys[name] = sorted({m for m in re.findall(
                    r'"(Total Score|总得分|total_score)"', body)})
    en = keys.get("eval_prompt", [])
    zh = keys.get("eval_prompt_zh", [])
    detail = f"""
        {p.relative_to(root)}
        eval_prompt     (English) asks for: {en}
        eval_prompt_zh  (Chinese) asks for: {zh}
        -> the scorer above only handles the Chinese one
    """
    return "Total Score" in en and "总得分" in zh and "总得分" not in en, detail


def f2_truncated_gold(root: Path):
    """English gold rubrics declare requirements whose bodies are absent."""
    out, tot = [], 0
    for tag, fname in (("en", "dacomp-arch-gold.jsonl"),
                       ("zh", "dacomp-arch-zh-gold.jsonl")):
        p = root / "dacomp-de/evaluation_suite_arch/gold" / fname
        if not p.exists():
            out.append(f"{fname}: MISSING")
            continue
        bad, n = [], 0
        for line in p.read_text(encoding="utf-8").splitlines():
            if not line.strip():
                continue
            n += 1
            d = json.loads(line)
            r = d.get("rubric", "")
            if tag == "en":
                declared = re.findall(r"^-\s*Requirement\s+(I{1,3})\s*[:：]", r, re.M)
                present = re.findall(r"^##\s*Requirement\s+(I{1,3})\s*[:：]", r, re.M)
            else:
                declared = re.findall(r"^-\s*需求\s*(I{1,3})\s*[:：]", r, re.M)
                present = re.findall(r"^##\s*需求\s*(I{1,3})\s*[:：]", r, re.M)
            miss = [x for x in dict.fromkeys(declared) if x not in set(present)]
            if miss:
                bad.append((d.get("id", "?"), miss, r.rstrip()[-46:]))
        out.append(f"{tag}: {len(bad)}/{n} rubrics incomplete")
        for i, m, tail in bad:
            out.append(f"     {i}  missing {','.join(m)}  ends: ...{tail}")
        if tag == "en":
            tot = len(bad)
    return tot > 0, "\n".join(out)


def f3_gold_path(root: Path):
    """Default gold path is one directory level too deep."""
    p = root / "dacomp-de/evaluation_suite_arch/utils/config.py"
    txt = p.read_text(encoding="utf-8")
    base = p.parent
    m = re.search(r"DEFAULT_GOLD_EN_JSONL\s*=\s*(.+)", txt)
    guess = base / "gold" / "dacomp-arch-gold.jsonl"
    actual = root / "dacomp-de/evaluation_suite_arch/gold/dacomp-arch-gold.jsonl"
    detail = f"""
        {p.relative_to(root)}
        {m.group(0).strip() if m else '?'}
        BASE_DIR resolves to : {base.relative_to(root)}
        default path         : {guess.relative_to(root)}   exists={guess.exists()}
        file actually lives  : {actual.relative_to(root)}   exists={actual.exists()}
    """
    return (not guess.exists()) and actual.exists(), detail


def f4_hardcoded_azure(root: Path):
    """Both entry points pin the Azure client class."""
    targets = ["dacomp-de/evaluation_suite_arch/evaluate.py",
               "methods/de-agent/evaluation/benchmarks/dacomp/run_infer_de_arch.py"]
    lines, hits = [], 0
    for t in targets:
        p = root / t
        if not p.exists():
            lines.append(f"{t}: MISSING")
            continue
        for i, ln in enumerate(p.read_text(encoding="utf-8").splitlines(), 1):
            if "AzureOpenAI" in ln:
                lines.append(f"{t}:{i}  {ln.strip()[:74]}")
                hits += 1
    return hits >= 2, "\n".join(lines)


def f5_truncation_scores_zero(root: Path):
    """An incomplete judge reply is recorded as 0, indistinguishable from a bad answer."""
    ev = root / "dacomp-de/evaluation_suite_arch/evaluate.py"
    fn, _ = load_function(ev, "extract_actual_score")
    txt = ev.read_text(encoding="utf-8")

    truncated = '{"Requirement 1": {"Criterion 1.1": {"analysis": "The submis'
    try:
        json.loads(truncated)
        parsed, err = None, "unexpectedly parsed"
    except json.JSONDecodeError as e:
        parsed, err = {"parse_error": str(e), "raw_content": truncated}, str(e)
    score = fn(None, parsed)

    checks = re.search(r"finish_reason", txt)
    detail = f"""
        {ev.relative_to(root)}
        truncated body ({len(truncated)} chars) -> json.JSONDecodeError: {err}
        parse_evaluation_response returns   : {{'parse_error': ..., 'raw_content': ...}}
        extract_actual_score of that        : {score}      <- same value as a genuinely empty answer
        'finish_reason' appears in the file : {bool(checks)}
        -> a transport-level truncation and a real score of 0 are indistinguishable
           in the saved results, so silent data loss cannot be detected downstream
    """
    return score == 0 and not checks, detail


def f6_temperature_inconsistent(root: Path):
    """The DE judge omits temperature; the DA judge in the same repo sets it to 0."""
    de = root / "dacomp-de/evaluation_suite_arch/evaluate.py"
    da = root / "dacomp-da/evaluation_suite/core/config.py"
    de_txt = de.read_text(encoding="utf-8")
    da_txt = da.read_text(encoding="utf-8") if da.exists() else ""

    de_has = re.search(r'["\']?temperature["\']?\s*[:=]', de_txt)
    da_hits = [f"{i}: {ln.strip()[:66]}" for i, ln in
               enumerate(da_txt.splitlines(), 1) if "temperature" in ln]
    api_params = re.search(r"api_params\s*=\s*\{(.{0,400}?)\}", de_txt, re.S)
    detail = f"""
        {de.relative_to(root)}  api_params:
        {(api_params.group(1).strip().replace(chr(10), ' ')[:150] if api_params else '?')}
        temperature set on the DE judge : {bool(de_has)}
        {da.relative_to(root)}:
        {chr(10).join('        ' + h for h in da_hits[:3]) if da_hits else '        (none)'}
        -> same repository, two judges, only one is pinned to a deterministic setting
    """
    return (not de_has) and bool(da_hits), detail


CHECKS = [
    ("1",  "DE-Arch scorer only reads the Chinese total key", f1_english_always_zero),
    ("1b", "English and Chinese prompts request different keys", f1b_prompt_keys_differ),
    ("2",  "English gold rubrics are truncated", f2_truncated_gold),
    ("3",  "Default gold path misses one directory level", f3_gold_path),
    ("4",  "Azure client class is hardcoded in both entry points", f4_hardcoded_azure),
    ("5",  "Truncated judge replies are silently recorded as 0", f5_truncation_scores_zero),
    ("6",  "DE judge omits temperature while DA judge sets it to 0", f6_temperature_inconsistent),
]


def main():
    root = Path(sys.argv[1] if len(sys.argv) > 1 else ".").resolve()
    print("=" * 74)
    print("DAComp findings - offline verification")
    print(f"repository: {root}")
    print("no network, no API key, standard library only")
    print("=" * 74)

    for n, title, fn in CHECKS:
        try:
            ok, detail = fn(root)
        except Exception as e:
            ok, detail = False, f"check raised {type(e).__name__}: {e}"
        report(n, title, ok, detail)

    print()
    print("=" * 74)
    good = sum(1 for _, _, ok in results if ok)
    for n, title, ok in results:
        print(f"  [{'x' if ok else ' '}] {n:<3} {title}")
    print(f"\n  {good}/{len(results)} findings reproduce on this checkout")
    print("=" * 74)
    return 0 if good == len(results) else 1


if __name__ == "__main__":
    sys.exit(main())
```

</details>

Navigation Menu

Uh oh!

Six reproducible defects in the DE-Arch evaluation suite (English tasks always score 0; 7/30 English gold rubrics truncated) #8

How to verify

1. `extract_actual_score()` reads only the Chinese total key, so every English DE-Arch task is recorded as 0

2. 7 of 30 English gold rubrics are truncated

3. Default gold path is missing one directory level

4. Both entry points hardcode `openai.AzureOpenAI`

5. A truncated judge reply is silently recorded as 0, indistinguishable from a real score of 0

6. The DE judge omits `temperature`; the DA judge in the same repository sets it to 0

Summary

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

template	line	requested key
`eval_prompt` (English, `utils/eval_prompt.py:1`)	65, 83	`"Total Score": int`
`eval_prompt_zh` (Chinese, `utils/eval_prompt.py:89`)	153, 171	`"总得分": int`

id	declared	missing
dacomp-de-arch-002	I, II, III	II, III
dacomp-de-arch-006	I, II, III	II, III
dacomp-de-arch-007	I, II, III	II, III
dacomp-de-arch-012	I, II, III	II, III
dacomp-de-arch-016	I, II, III	II, III
dacomp-de-arch-017	I, II, III	II, III
dacomp-de-arch-019	I, II, III	III

#	Defect	Effect
1	Score extraction is Chinese-key-only	English DE-Arch always 0
2	7/30 English gold rubrics truncated	Denominator counts unawardable points
3	Default gold path off by one directory	Defaults never resolve
4	`AzureOpenAI` hardcoded in both entry points	Non-Azure providers unreachable
5	No `finish_reason` check	Dropped replies indistinguishable from 0
6	No `temperature` on the DE judge	Grading is nondeterministic

Uh oh!

Six reproducible defects in the DE-Arch evaluation suite (English tasks always score 0; 7/30 English gold rubrics truncated) #8

Description

How to verify

1. extract_actual_score() reads only the Chinese total key, so every English DE-Arch task is recorded as 0

2. 7 of 30 English gold rubrics are truncated

3. Default gold path is missing one directory level

4. Both entry points hardcode openai.AzureOpenAI

5. A truncated judge reply is silently recorded as 0, indistinguishable from a real score of 0

6. The DE judge omits temperature; the DA judge in the same repository sets it to 0

Summary

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

1. `extract_actual_score()` reads only the Chinese total key, so every English DE-Arch task is recorded as 0

4. Both entry points hardcode `openai.AzureOpenAI`

6. The DE judge omits `temperature`; the DA judge in the same repository sets it to 0