Cache design: content-addressed local SQLite store

[ErlisLushtaku]

Context

Follow-up to #36. The langchain cache does not work with vLLM and gives little control, and the earlier drafts mixed identity keys with content keys in different ways. There is now a lean draft stack on top of #118 (#121–#124, #127) that tries to settle the local cache design. I would like to walk through the current approach and get feedback.

What we do

We cache both completions and judgements in separate SQLite trees under an optional --run.store_root. The cache boundary is do_inference. The model is prepared lazily, so a full hit does not load the backend.

Rows are keyed by a hash of the rendered input, and live under a folder keyed by the validated model descriptor (sampling, max tokens, thinking settings, chat template, output-affecting engine settings). Completions and judgements stay in separate role roots so the same model string as battle model and as judge cannot collide and thus they have separate metadata files. Typed columns (benchmark, instruction_id, model / model_a/model_b, orientation, …) are kept for inspection and targeted deletion, but we don't use them as key anymore. Prompt-construction options whose effect already appears in the rendered messages are captured by the input hash. Parsing and aggregation stay outside the raw-output cache.

We additionally add a Hugging Face fetch/push as a separate CLI in #127.

Layout

{store_root}/completions/{task}/{provider}/{model}/{descriptor_hash}/
├── metadata.json
└── completions.db

{store_root}/judgements/{task}/{provider}/{model}/{descriptor_hash}/
├── metadata.json
└── judgements.db

metadata.json stores the descriptor.

Schema

completions(
  input_hash TEXT PRIMARY KEY,
  input_text TEXT NOT NULL,
  completion TEXT NOT NULL,
  benchmark TEXT NOT NULL,
  instruction_id TEXT NOT NULL,
  model TEXT NOT NULL,
  pushed_by TEXT NOT NULL,
  pushed_at TEXT NOT NULL,
  run_id TEXT NOT NULL
)

judgements(
  input_hash TEXT PRIMARY KEY,
  judge_input TEXT NOT NULL,
  judge_completion TEXT NOT NULL,
  benchmark TEXT NOT NULL,
  instruction_id TEXT NOT NULL,
  model_a TEXT NOT NULL,
  model_b TEXT NOT NULL,
  judge TEXT NOT NULL,
  top_logprobs TEXT,
  orientation TEXT,
  pushed_by TEXT NOT NULL,
  pushed_at TEXT NOT NULL,
  run_id TEXT NOT NULL
)

top_logprobs is there so cached judgements can restore the first-token logprobs used by the #118 parsers. orientation is direct/reversed relative to the source model order when that applies. model_a/model_b match the positional slots in the rendered judge prompt.

Why this shape

Identity keys alone miss cases where the same battle is judged with a different rendered prompt or different model settings. Content keys alone don't provide easy filtering of the rows. The current shape keeps content-addressed hits, and the other fields for filtering and targeted deletion of the cache. VLLM execution-only settings (tensor_parallel_size, gpu_memory_utilization, enforce_eager) stay out of the descriptor.

Any thoughts on this? cc: @kargibora

[kargibora]

Thanks for the issue! I think this is a nice and reasonable fit.

Does input_hash hashed from the entire input text to the judge system prompt + user prompt + two annotations? Also since annotations and completions are, on a high level, quite similar perhaps we can do something like

 CREATE TABLE responses (
       input_hash   TEXT PRIMARY KEY,
       input_text   TEXT NOT NULL,
       completion   TEXT NOT NULL,
       top_logprobs TEXT, -- JSON
       created_at   TEXT NOT NULL,
       created_by   TEXT NOT NULL
   );

   CREATE TABLE completion_metadata (
       input_hash     TEXT PRIMARY KEY REFERENCES responses(input_hash),
       benchmark      TEXT NOT NULL,
       instruction_id TEXT NOT NULL,
       model          TEXT NOT NULL,
       run_id         TEXT NOT NULL
   );

   CREATE TABLE judgement_metadata (
       input_hash     TEXT PRIMARY KEY REFERENCES responses(input_hash),
       benchmark      TEXT NOT NULL,
       instruction_id TEXT NOT NULL,
       model_a        TEXT NOT NULL,
       model_b        TEXT NOT NULL,
       judge          TEXT NOT NULL,
       orientation    TEXT,
       run_id         TEXT NOT NULL
   );

but we should also be sure that our judgement is not specific to pairwise (in my opinion). It should include both multi-turn conversation, should be suitable for pointwise judgements (not necessary but may be, do we really need model_b not the be NULL.

[ErlisLushtaku]

@kargibora Thanks!

Does input_hash hashed from the entire input text to the judge system prompt + user prompt + two annotations?

Yes, input_hash is the SHA-256 of the rendered input we send to the backend. For chat judges that is the message list (system + user), so the instruction and both completions are included. For text backends it is the flattened prompt.

I think we should keep completions and judgements in separate folders, so we can inspect, delete, or sync one role without touching the other. If we do that, a shared responses table would not actually be shared, each folder would still have its own db. Splitting responses from the metadata tables would help if one cached output could have several associations (e.g. several run_ids), however I don't think that is realistic to happen often in practice. So I'm not sure if separting the metadata table and sharing the responses table would buy us much.

On pairwise, I agree making model_b nullable is a small change, we can do that now so pointwise can use model_a only. Multi-turn should already work, since we hash the full rendered conversation.

What do you think?