A web index for AI, built through open competition on Bittensor.
AI applications need relevant, current sources and text they can use, soon enough to finish their task. Desearch collects and refreshes useful pages from the public web, prepares them for retrieval and develops the models that help AI find the right sources. Search is the first application: fast search for AI agents, and deeper research for questions that take longer.
The work happens in the open on Bittensor Subnet 22. Miners build and refresh the collection, validators check their work under published rules, and the Desearch team integrates accepted work into a tested index and the search API.
⛏️ Mine · 🛡️ Validate · ⚙️ How it works · 🧭 Desearch 2.0 · 🔑 Search API
In Desearch 1.0, miners answered search requests and validators judged the answers. It worked, but good answers could arrive too late, some depended on outside search providers, and little of the work left anything that could be kept and improved.
Desearch 2.0 prepares information before a query arrives. Pages are collected, cleaned, deduplicated, organized and prepared for retrieval in the background, so a query searches a maintained collection instead of starting from scratch. That gives:
- Faster retrieval: expensive preparation moves out of the query path
- More control over quality: coverage, freshness and supporting text can be inspected, and models improved against the same collection
- Work that lasts: every accepted page serves many later searches, not one answer
The collection covers the public web: company websites, news, articles, blogs and documentation. It starts with selected sources and grows as results prove useful.
| Work | Who does it | Status |
|---|---|---|
| Crawl and refresh pages, extract their text | Miners fetch assigned pages; validators re-check a sample of their tasks | Running on SN22 |
| Embed accepted text with a selected model | Miners run the model on their GPUs; validators recompute a sample | Built; opens with Desearch's embedding model |
| Train retrieval models, improve ranking, extraction and coverage | Model builders and data operators | Later programs, announced before they open |
| Build, test and serve the index | The Desearch team | Team-operated; queries never wait on miner work |
Miners crawl: they take tasks, fetch the assigned pages and upload the extracted text. They are paid by their share of accepted pages, and can test their miner locally before registering. Miner setup → · Test locally → · Emission →
Validators check: they re-fetch a random sample of miners' tasks, compare it with what the miner returned, and set weights from the accepted work. A miner caught returning bad pages loses what it earned since its last good check. Validator setup →
The rules are published and every result is public. How it works →
The search API is available today for developers building AI agents and research tools. Get a key in the console and start with the documentation.
| SDK | Install |
|---|---|
| Python | pip install desearch-py |
| JavaScript / TypeScript | npm install desearch-js |
| MCP server | npm install -g desearch-mcp-server |
| Guide | What's inside |
|---|---|
| Miner setup | Install, register, configure, run, how you earn, monitoring |
| Test your miner locally | Run your miner against a local task API and validator before registering |
| Validator setup | Install, register, configure, run, automatic upgrades, monitoring |
| Emission | How miners' shares are worked out and what raises them |
| Architecture | How the bot, task API, miners, validators, storage and engine work together |
| Embedding tasks | What embed tasks will contain and how they are checked, before they open |
| Engine | The search index and API, and how new pages reach it |
| Task API | Endpoints, rounds, scoring rules, public logs and the storage layout |
| Desearch 2.0 | The direction: why an index, the first phase, incentives and participation |
- Discord: questions, mining and validating support
- X and Telegram: announcements
- Blog: guides and engineering write-ups
Released under the MIT License.