Skip to content

About

A reproducible, self-hosted benchmark for comparing JEV and LLM decision-making on your own datasets, models, and evaluation policies.

Topics

Resources

Stars

3 stars

Watchers

0 watching

Forks

Latest commit

 

History

32 Commits

Folders and files

Repository files navigation

jev-decision-bench

jev-decision-bench evaluates conventional language models on tasks designed around TypeSafe JEV's bounded-decision interface. It is not a general LLM leaderboard and it does not ask JEV to take benchmarks designed for free-form text generation.

The central question is:

Given the same state, decision rubric, and permitted answers, how accurately, reliably, quickly, and cheaply can JEV and conventional LLMs make a code-consumable decision?

Benchmark at a glance

These are reproducible experiment snapshots, not a single cross-task leaderboard. Each cell shows accuracy followed by provider-reported token usage per attempted decision; Score uses nearest-tier accuracy from the reported continuous score.

Model / Accuracy (token usage) Choice Noul Score
JEV 81.0% (2,867) 98.2% (382) 47.4% (484)
DeepSeek V3.2 79.5% (1,297) 97.2% (195) 42.9% (364)
Claude Haiku 4.5 77.9% (2,326) 95.1% (367) 44.6% (649)
Claude Sonnet 5 80.5% (2,038) 97.4% (285) 47.4% (499)
GPT-5.6 Luna 85.6% (1,608) 100.0% (196) 45.2% (516)
GPT-5.6 Sol 86.1% (1,148) 99.8% (208) 45.8% (426)

See published results for output reliability, structured output mode, latency, token use, and detailed reports.

A Benchmark You Run Yourself

This project is not a generic online benchmark suite or a hosted leaderboard. It is an easy-to-run, reproducible toolkit for evaluating your own model configurations and versioned datasets against the same bounded-decision contracts used for JEV, so you can make an evidence-based comparison with JEV in your environment.

See the Architecture for how experiment packages, runs, scores, and comparisons fit together.

Project structure

configs/       Tracked, editable model-route configurations; API keys stay in the environment.
experiments/   Versioned experiment specifications, rubrics, and source-data pins.
src/           Benchmark preparation, adapters, execution, scoring, and comparison code.
docs/          Methodology, experiment index, run guide, and provider setup guides.
artifacts/     Generated experiment packages, runs, scores, comparisons, and logs.
tests/         Contract and regression tests for the benchmark implementation.

Experiment overview

The benchmark is organized by decision task family. Dataset identifies the source where one has been selected; TBD means the family is part of the benchmark design but no dataset has been adopted yet. See the experiment index for implementation and source details.

Task family JEV task type Dataset Description
Intent / queue routing Choice / Noul BANKING77 Route a request to the appropriate queue or determine whether it belongs in a candidate queue.
Semantic attribute extraction Multiple Noul / Choice TBD Extract controlled attributes, such as topic, affected product, stated deadline, or request type.
Policy & guardrail decisions Noul HateCheck Decide whether content or an action satisfies an explicit policy or guardrail.
Tool / action approval Choice / Noul TBD Choose or approve the appropriate next action from a bounded set; actions are never executed.
Severity / priority assessment Score TBD Evaluate ordered judgments such as urgency, risk, or business impact.
Search relevance / ranking Noul / Score TREC Deep Learning Determine query-document relevance and rank candidates.
Generic requirement verification Noul / Choice TBD Determine whether an artifact or state satisfies a named, explicit requirement.
OOD uncertainty & abstention All three TBD Measure whether confidence falls and review is selected when evidence or policy is missing.

What Is a Decision?

A decision is a bounded, code-consumable answer selected from an explicit answer space—not a request to generate free-form text. Each benchmark item contains the state, question, permitted answers, and gold label used for evaluation. Its answer-space geometry determines the native JEV interface:

Decision geometry JEV interface Example
Two mutually exclusive outcomes Noul Is this query about a card that has not arrived?
Unordered, closed-set alternatives Choice Which banking intent best matches this query?
Ordered rubric levels Score How urgent is this request: low, medium, high, or critical?

Note

The benchmark treats all three as bounded decisions, but preserves their semantics. A Score is an ordered rubric position, not an arbitrary number; using a Score for categories with no meaningful order is invalid. A two-level Score is equivalent to a Noul and is not scored as a separate capability.

Methodology

Most LLM evaluations begin with a benchmark designed for general-purpose language models and then ask every system, including JEV, to take it. This project deliberately reverses that direction.

We first design the workload around the kinds of decisions JEV is intended to make: a bounded state, an explicit rubric, and a finite answer space. We then ask conventional LLMs to take the same decision tasks through a normalized, schema-constrained adapter. The purpose is not to test whether JEV can imitate free-form generation. It is to measure how well LLMs can perform the work JEV is designed to do.

This method does not presume anything about JEV's private implementation or that JEV will win. It makes the comparison operationally fair: all systems see the same evidence, labels, and decision criteria; only their native interfaces and reported uncertainty differ.

See the Methodology for more information.

Documentation

The repository follows a small documentation set: this README explains the project, while the docs/ pages describe the portfolio, method, execution, and provider-specific setup.

  • Experiment index — registered experiments, task types, datasets, and status.
  • Methodology — experiment packages, benchmark items, model runs, and result contracts.
  • Architecture — code boundaries, immutable artifacts, task contracts, adapters, and re-evaluation.
  • Running experiments — environment setup, package preparation, and model execution.
  • Analyzing and sharing results — offline scoring, comparison, reproducible exports, and publication artifacts.
  • AWS / Amazon Bedrock — self-service AWS credentials and Bedrock setup.

About

A reproducible, self-hosted benchmark for comparing JEV and LLM decision-making on your own datasets, models, and evaluation policies.

Topics

Resources

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages