bittensor-core: one Rust core (keys, keyfiles, timelock, codec, RFC-0078 digest, Ledger) under a thin Python SDK

[unarbos]

Summary

Implements the full sdk/bittensor-core-spec.md plan (phases 0–5). One workspace crate, bittensor-core, now owns everything whose right answer is chain-defined — sp-core key primitives, keyfiles, drand timelock, ML-KEM, the SCALE codec + runtime-metadata engine, extrinsic assembly, multisig derivation, and the RFC-0078 merkleized-metadata digest — exposed to Python through one PyO3 binding crate (bittensor-core-py, PyPI: bittensor-core). The Python public API is frozen throughout; each phase landed as its own commit and is independently revertible.

By phase:

• Phase 0 — decoded-value shape corpus recorded from cyscale as the codec's compatibility contract (sdk/python/tests/fixtures/shape_corpus/), crate skeleton, build-core-wheels.yml.
• Phase 1 — py-sp-core + bittensor-drand sources consolidated into the core; old crates, workflows, and the drand C ABI deleted; release train stamps one version.
• Phase 2 — RFC-0078 digest module (merkleized-metadata 0.5.1, vectors pinned against polkadot-js merkleizeMetadata), LedgerSigner clear-signing via the Polkadot generic app (feature ledger), --ledger CLI flag, docs guide.
• Phase 3 — Rust Runtime + codec become the production codec.py seam (CallBytes replaces GenericCall); cyscale, xxhash, TYPE_REGISTRY, and the cyscale-era quirks are deleted. Decode hot paths cross the FFI once per page and run off the GIL (rayon), with a limb-based base58 for ss58 rendering (~6x, byte-identical to sp-core).
• Phase 4 — signature payload, extrinsic assembly, and multisig derivation in core, byte-equal to the golden signing vectors (landed inside phases 2–3).
• Phase 5 — docs updated (repo layout, wallets, release process, sdk-tests dev loop), spec status + acceptance benchmark recorded.

Acceptance benchmark (finney, 2026-07-10 — spec §10)

Metric	Baseline	Target	Accepted
Codec build from metadata bytes	337 ms	≤15 ms	4 ms
Metagraph SCALE decode	3–6 MB/s	≥50 MB/s	60–77 MB/s
compose_call	1.1 ms	≤50 µs	<50 µs
Sign + assemble	0.1 ms	no regression	0.03 ms
query_map decode	92k entries/s	≥500k entries/s	599k entries/s
Warm connect	~1.1 s	~0.75 s	~0.6 s

Review focus

• The codec.py seam rewrite and its consumers (storage.py, extrinsics.py, runtime.py, runtime_api.py, interface.py)
• Shape-corpus + golden coverage: both suites run natively in Rust (cargo test -p bittensor-core) and through the Python seam
• The Ledger APDU/proof path (bittensor-core/src/signers/, bittensor/ledger.py) — on-device verification still to be done by someone with hardware

Test plan

• cargo test -p bittensor-core — 44 tests green (corpus, goldens, digest vectors, keys, timelock)
• cd sdk/python && uv run pytest — 874 unit tests green (one pre-existing EVM ABI-drift failure from 08c1a21, unrelated)
• ruff / ty / codegen static gates green
• Acceptance benchmark recorded (table above)
• e2e suite against a localnet / mainnet clone
• On-device Ledger verification (generic app, real hardware)

Made with Cursor

Update 2026-07-10: runtime changes merged in

This branch now also includes the feat/squashed-2026-07-06 runtime work (previously PR #2847 → devnet-ready), merged in 91b42d5152:

• Coinbase dust-removal rework: DustRemoval hook mirrors balances dust burns into subtensor TotalIssuance; leasing/order-swap transfers consolidated onto transfer_tao
• Reservoir accounting for protocol liquidity (BalancerAlphaReservoir / BalancerTaoReservoir) with dissolution folding reservoirs into reserves before payout
• Migrations: migrate_fix_subnet_hotkey_lock_swaps (index-based rewrite), migrate_fix_total_issuance_evm_fees (dust-collection reset entry)
• decrease_take/increase_take flip Pays::No → Pays::Yes (anti-spam; metadata-only, no transaction_version bump needed)
• New coverage: eco-tests/src/tests_mentat_indexer.rs (indexer storage contract), runtime/tests/balances_dust.rs
• spec_version 428 (already bumped on this branch); benchmark weights regenerated for the drifted pallets