Most MCP tools can't distinguish "found nothing" from "couldn't look". I measured it, starting with my own

[ogasurfproject-jpg]

Pre-submission Checklist

• This post follows the MCP community guidelines

What would you like to share?

I built a read-only probe that asks every tool one question from its declared outputSchema alone: can a consumer separate a failed read from an empty one? It calls tools/list and nothing else. No tools/call, no installs, no account.

Why it matters. If a tool returns the same shape for "the query ran and matched nothing" and "the query threw," an agent cannot tell your outage from your silence. It will confidently report absence during a failure. Federico Blanco Sanchez-Llanos named this "The Mould, Not the Letter."

What I found. Across a spread sample of the 12,429 active public endpoints (18 spoke MCP, 335 tools): 96.7% (a strict structural ruler) to 99.1% (a lexical one) of tools declare no way to hold the difference. I report a band because the rulers err in opposite directions, and I publish the field names because the structural ruler's passes are mostly incidental booleans (one endpoint passed on remote: boolean from a job board, not a read-state).

I measured myself first. 85.7% of my own tools failed it, the gate and ledger included. Better than the field, still most of mine. Fixed to 2.9%; the whole diff, the four runtime bugs, and the reproduce commands are frozen in a public append-only ledger (mould-gh-12).

Check yours (read only, executes nothing):

curl -s -X POST https://gate.horizonshield.dev/check \
  -H 'content-type: application/json' \
  -d '{"endpoint":"https://YOUR-SERVER/mcp"}' | jq .absence_vs_failure

The honest limit: a schema is a declaration, not behaviour. This says what your contract can express, not what your handler does at runtime; that needs tools/call, which I won't do to servers I don't own. Method, both rulers, and the population hash are all public.

Relevant Links

Ledger record (recompute the hash yourself): https://gate.horizonshield.dev/mould/mould-gh-12
Method and per-condition guide: https://shield.the-horizons-innovation.com/verify-directory/operators/
Survey data, CC0: https://github.com/ogasurfproject-jpg/mcp-registry-survey

[okgrit]

The distinction between a declaration and actual handler behavior is the part I would keep prominent. A useful control pair might expose the same outputSchema: one handler legitimately returns an empty collection, while the other reports an MCP tool error through isError. A schema-only probe cannot tell those runtime behaviors apart, even though their consumers can.

Do you have a small published control corpus for that pair, including malformed schema and unavailable introspection? I would be interested in contributing a fixture that expects 'not declared/unknown' separately from 'runtime error handling absent'. That would help keep the detector from making a stronger claim than its inputs support.

Disclosure: I maintain done-is-a-claim, and Codex helped prepare this comment. Its evidence tools distinguish historical command results from current recheck states; they do not audit MCP handlers. I have not independently reproduced your survey figures.

[ogasurfproject-jpg]

@Grit-77 thank you. No, we do not have a published control corpus for that pair yet.

What exists today is narrower. The probe (mould_contract.mjs) reads tools/list only. Separately, the tests of our own gate's tools check that each answer matches its declared outputSchema, and that preflight_agent on an unreachable agent card returns an isError tool error with no structured object (workers/hs-verify-gate/test/output_schema.test.mjs). Those tests cover our handlers, not a detector, so they are not a corpus someone else can run a probe against.

A fixture with the pair you describe would be welcome: the same outputSchema, one handler returning an empty collection and the other an isError result, plus a malformed schema and a server whose tools/list is unavailable. A schema-only probe should give both handlers the same label, because it reads only the declaration. That is the limit the post states, and your fixture would make it checkable. We agree that "not declared or unknown" and "runtime error handling absent" should be separate labels, and that only a probe that calls tools/call could emit the second.

If you open an issue on ogasurfproject-jpg/horizon-shield with the layout you have in mind, we can agree the file names and expected labels there before you write it, and then run the probe against it in public.

[okgrit]

I opened the proposed fixture layout here: ogasurfproject-jpg/horizon-shield#62 . It keeps empty-success and tool-error declarations identical, adds malformed-schema and unavailable-listing controls, and leaves runtime observations in a separate tools/call layer. The issue asks for the actual label vocabulary and preferred location before code is written.

Thanks for making the schema-only boundary explicit; the proposal treats an unavailable declaration as unknown rather than evidence that runtime error handling is absent.

Prepared with Codex on behalf of Grit-77.