Align skills with the 500-line guidance and add evals - #84
Open
kaankacar wants to merge 5 commits into
Open
Conversation
SKILL.md keeps the SEP/CAP routing map (129 lines) with a task-to-file table; Part 2 moves verbatim to ecosystem.md, Part 3 to resources.md. Cross-part references now link the companion files.
SKILL.md keeps SDK setup, wallet connection, and tx build/submit (467 lines). React + Next.js patterns, client-side reads, and passkeys/fee sponsorship move to companion files; Quick Navigation becomes the task-to-file routing table.
SKILL.md keeps the decision table, the shared testnet account setup (keypairs, funding, trustlines, Circle faucet), and the two-USDC-address reference (116 lines). Part 1 becomes x402.md with the OZ-specific runbook steps renumbered; Part 2 becomes mpp.md.
SKILL.md drops to 407 lines and gains the task-to-file routing table; Horizon endpoints/operations/streaming/pagination move verbatim.
Three scenarios per skill plus cross-skill routing checks and an
off-topic negative control, in the {skills, query, expected_behavior}
format with optional machine_checkable assertions. evals/README.md
documents the tiers (compile checks, LLM-judged behavior, trigger
checks), the baseline process, and how to run a scenario; the root
README links it and documents the 500-line router convention.
Scenario expectations reflect current protocol reality rather than the
July proposal where they diverged: Noir/UltraHonk verifies on-chain
since Protocol 26 (#72), MPP Channel mode is now Session (#71), and
getLedgers depth is provider-retention dependent (#73).
11 tasks
|
Contributor
There was a problem hiding this comment.
Pull request overview
Splits oversized Stellar skills into progressive-disclosure routers and adds an evaluation framework.
Changes:
- Split four skills into sub-500-line routers plus companion guides.
- Added 24 scenarios across seven skills and cross-skill routing.
- Documented evaluation and contribution workflows.
Reviewed changes
Copilot reviewed 38 out of 38 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
README.md |
Documents routers and evaluations. |
skills/standards/SKILL.md |
Routes standards-related tasks. |
skills/standards/ecosystem.md |
Houses ecosystem references. |
skills/standards/resources.md |
Houses curated resources. |
skills/data/SKILL.md |
Routes RPC and Horizon tasks. |
skills/data/horizon.md |
Houses legacy Horizon guidance. |
skills/dapp/SKILL.md |
Routes dapp development tasks. |
skills/dapp/react.md |
Houses React and Next.js examples. |
skills/dapp/data-fetching.md |
Houses client data-access examples. |
skills/dapp/smart-accounts.md |
Houses passkey and sponsorship guidance. |
skills/agentic-payments/SKILL.md |
Routes payment protocol tasks. |
skills/agentic-payments/x402.md |
Houses the x402 playbook. |
skills/agentic-payments/mpp.md |
Houses Charge and Session guidance. |
evals/README.md |
Defines evaluation format and workflow. |
evals/scenarios/assets/01-freezable-stablecoin.json |
Evaluates controlled asset issuance. |
evals/scenarios/assets/02-op-no-trust.json |
Evaluates trustline troubleshooting. |
evals/scenarios/assets/03-usdc-in-contract.json |
Evaluates USDC SAC usage. |
evals/scenarios/agentic-payments/01-monetize-express.json |
Evaluates paid Express APIs. |
evals/scenarios/agentic-payments/02-high-frequency-session.json |
Evaluates MPP Session selection. |
evals/scenarios/agentic-payments/03-signer-throws.json |
Evaluates signer troubleshooting. |
evals/scenarios/dapp/01-freighter-payment.json |
Evaluates Freighter payments. |
evals/scenarios/dapp/02-contract-invoke.json |
Evaluates browser contract invocation. |
evals/scenarios/dapp/03-network-config.json |
Evaluates network configuration. |
evals/scenarios/data/01-historical-transactions.json |
Evaluates historical-data routing. |
evals/scenarios/data/02-live-payments.json |
Evaluates payment streaming. |
evals/scenarios/data/03-contract-storage-read.json |
Evaluates contract storage reads. |
evals/scenarios/routing/01-dapp-plus-payments.json |
Evaluates multi-skill payment routing. |
evals/scenarios/routing/02-rwa-compliance.json |
Evaluates asset/standards routing. |
evals/scenarios/routing/03-negative-control-pdf.json |
Adds an off-topic control. |
evals/scenarios/smart-contracts/01-token-admin-mint.json |
Evaluates secure token contracts. |
evals/scenarios/smart-contracts/02-auth-tests.json |
Evaluates authorization tests. |
evals/scenarios/smart-contracts/03-ttl-archival.json |
Evaluates TTL troubleshooting. |
evals/scenarios/standards/01-fiat-onramp-kyc.json |
Evaluates SEP selection. |
evals/scenarios/standards/02-nft-standard-status.json |
Evaluates SEP-50 status handling. |
evals/scenarios/standards/03-contract-event-indexers.json |
Evaluates indexer discovery. |
evals/scenarios/zk-proofs/01-circom-groth16.json |
Evaluates Groth16 guidance. |
evals/scenarios/zk-proofs/02-noir-onchain.json |
Evaluates UltraHonk guidance. |
evals/scenarios/zk-proofs/03-private-airdrop.json |
Evaluates private-airdrop design. |
Suppressed comments (1)
evals/README.md:67
- Tier 1 is only described here: no workflow or runner in the repository reads
machine_checkable, extracts generated code, or executes these checks. The linked proposal requires Tier 1 in CI on every PR, so the claimed full eval set is not operational. Add an executable eval runner plus a workflow triggered by skill changes.
- Tier 1 (compile checks) is cheap and deterministic — run on every PR that touches `skills/`.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| await kit.connectWallet({ prompt: true }); | ||
|
|
||
| // Sign and submit transactions | ||
| const result = await kit.signAndSubmit(transaction); |
| - `"$0.001"` — human-readable, auto-converts to 7-decimal USDC units | ||
| - `{ amount: "1000", asset: "ASSET_SAC_CONTRACT_ID" }` — explicit base units for non-USDC assets | ||
|
|
||
| **`payTo` is the recipient's classic Stellar account (`G...`), not the USDC SAC contract address.** Sending USDC lands in the classic balance of the `payTo` account, which is why that account also needs a USDC trustline. The SAC contract address is what the protocol invokes `transfer` on; see "Two USDC addresses" below. |
|
|
||
| ## Baselines: prove each eval discriminates | ||
|
|
||
| Before trusting a scenario, run it **without** the skills installed and keep the failing transcript under `evals/baseline/<skill>/<scenario>.md`. That proves the eval discriminates (an unskilled model fails it), and tells us which evals to retire as base models improve — an eval every unskilled model passes measures nothing. |
|
|
||
| # 2. Run one scenario headlessly and capture the transcript | ||
| q=$(python3 -c "import json;print(json.load(open('evals/scenarios/dapp/01-freighter-payment.json'))['query'])") | ||
| claude -p "$q" > /tmp/eval-transcript.txt |
|
|
||
| ## Evaluations | ||
|
|
||
| [`evals/`](evals/README.md) holds ~3 task scenarios per skill (plus cross-skill routing checks and a negative control), each encoding a mistake agents actually make without the skill. Three grading tiers: machine-checkable compile checks, LLM-judged behavior assertions, and skill-trigger checks. See [evals/README.md](evals/README.md) for the format and how to run them. |
| @@ -0,0 +1,73 @@ | |||
| # Skill Evaluations | |||
|
|
|||
| Representative task scenarios for every skill in this repo, following [Anthropic's evaluation-driven skill authoring guidance](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices#evaluation-and-iteration). Each scenario encodes a mistake an agent actually makes *without* the skill — several come from real failure modes (the #41 compile bugs, documented pitfalls in agentic-payments, the ZK curve trap), not imagined ones. Run them before publishing skill changes so regressions get caught here instead of by users. | |||
| const closeHandler = server | ||
| .transactions() | ||
| .forAccount(publicKey) | ||
| .cursor("now") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #42.
Both gaps, in one PR: all four oversized skills split to Anthropic's sub-500-line router + companions layout (the #32 / smart-contracts pattern), and the full proposed eval set added.
Gap 1 — the splits
ecosystem.md(462),resources.md(265)react.md(147),data-fetching.md(85),smart-accounts.md(83)x402.md(266),mpp.md(282)horizon.md(151)Per the issue's checklist:
smart-contracts/SKILL.mdstyle; agentic-payments' router additionally keeps the shared testnet setup (keypairs → friendbot → trustlines → Circle faucet) and the two-USDC-addresses reference that both protocols need.skills/with zero broken.site/src/data/skills.tsneeded no source changes — cards keep pointing at eachSKILL.md. Confirmed the scripts handle the new files:copy-skills.mjscopies whole skill directories (8/8 sources copied, companions present underpublic/skills/), andgenerate-llms-txt.mjsauto-indexes all 8 new companions as nested entries with their H1 titles.pnpm lint,lint:ts, andbuildgreen.Gap 2 — evals
evals/scenarios/carries the full proposed set: 3 scenarios × 7 skills + 3 cross-skill routing checks (including the off-topic negative control) = 24 files, in the proposal's{skills, query, expected_behavior}format plus an optionalmachine_checkablefield so CI can find the tier-1 assertions.evals/README.mddocuments the format, the three grading tiers, how to run a scenario against an agent with the skills installed, the baseline process (run without skills, keep the failing transcript, retire evals that stop discriminating), and the CI guidance (tier 1 on PRs, tiers 2–3 manual). The root README links it, documents the 500-line router convention, and asks contributors to update the matching scenario when a skill's facts change.One deliberate deviation from the July proposal, flagged rather than silently applied: three scenario expectations were written before #71/#72/#73 merged and would now punish the correct answer. They're updated to current reality — the Noir/UltraHonk scenario expects the on-chain path (Protocol 26+,
rs-soroban-ultrahonk, maturity caveat) instead of "not possible today"; the high-frequency-agent scenario expects Session mode (Channel kept as the historical synonym); the historical-data scenario expects the provider-retention check (getHealth().oldestLedger) instead of unconditionalgetLedgersdepth. SEP statuses cited in scenarios were re-verified againststellar-protocolpreambles today (SEP-6/12/24 Active, SEP-50 Draft, SEP-57 Draft).Not included, as a natural follow-up: scenarios for
cross-chain, which landed after the issue was filed.