Memtle

A local-first memory palace for AI assistants. Single static binary backed by embedded SQLite (turso). No Python, no ChromaDB, no API keys.

This repository publishes the Rust crate and CLI as memtle.

---

Why

The Python version used ChromaDB + SQLite. Under multiple simultaneous MCP clients, SQLite locking caused dropped writes. ChromaDB also carried a large dependency footprint and required Python to be installed.

This reimplementation:

• Ships as a single self-contained binary
• Replaces ChromaDB semantic search with a keyword inverted index (BM25-style scoring via drawer_words)
• Fixes the concurrency problem at the turso layer
• Keeps all MCP tools and all CLI commands fully compatible

Trade-off: Keyword search instead of embedding-based semantic search. Semantic search is deferred until an embedded model is available without network dependencies.

---

Installation

git clone https://github.com/ProjectViVy/memtle.git
cd memtle
cargo build --release
# binary is at: target/release/memtle

Optionally copy to a location on your PATH:

cp target/release/memtle ~/.local/bin/memtle

Precompiled binaries (GitHub Actions Artifacts)

Each artifact on the GitHub Actions page lists a sha256 digest. Verify the downloaded zip against that digest before extracting:

# macOS
shasum -a 256 memtle-macos-<git-short-sha>.zip

# Linux
sha256sum memtle-linux-<git-short-sha>.zip

The output should match the digest shown on the Artifacts page (strip the leading sha256: prefix before comparing).

On macOS, the extracted binary will also be quarantined because it was not distributed through the App Store or a notarised installer. After extracting, remove the quarantine attribute before running:

xattr -d com.apple.quarantine ./memtle

---

MCP Setup (Claude Code)

claude mcp add memtle -- /path/to/memtle mcp

The MCP server runs as a JSON-RPC 2.0 process over stdio. All 26 tools are available immediately after the server starts.

On first use, call memtle_status — it returns the full memory protocol and AAAK dialect spec in the response, so the AI learns how to use the palace during wake-up.

---

Quick Start

# 1. Initialise a project (creates memtle.yaml)
memtle init ~/my-project

# 2. Mine project files into the palace
memtle mine ~/my-project

# 3. Mine conversation transcripts
memtle mine ~/.claude/projects/ --mode convos

# 4. Search
memtle search "chromadb locking"

# 5. Generate wake-up context (L0 identity + L1 essential story)
memtle wake-up

See USAGE.md for the full CLI reference, configuration options, and MCP tool descriptions.

---

Use As A Rust Crate

[dependencies]
memtle = { git = "https://github.com/ProjectViVy/memtle", default-features = false }
serde_json = "1"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

#[tokio::main]
async fn main() -> memtle::error::Result<()> {
    let toolkit = memtle::toolkit::MemtleToolkit::open("palace.db").await?;

    let tools = toolkit.tool_definitions();
    assert!(!tools.is_empty());

    let status = toolkit
        .call_json("memtle_status", serde_json::json!({}))
        .await?;
    println!("{status}");

    Ok(())
}

Stable embedding surface:

• memtle::MemtleToolkit
• memtle::toolkit
• memtle::tools
• memtle::error

Feature flags:

• default = ["cli", "mcp"]
• default-features = false for library-only embedding without the CLI binary or MCP server

---

Architecture

src/
  main.rs              Entry point: clap dispatch → open_palace() → handler
  db.rs                open_db(), query_all() helpers over turso::Connection
  schema.rs            DDL: 6 tables + indexes, ensure_schema()
  config.rs            MemtleConfig ($XDG_DATA_HOME/memtle/config.json) + ProjectConfig (memtle.yaml)
  error.rs             thiserror Error enum

  cli/                 One file per subcommand
    init.rs            Entity discovery + optional LLM refinement + room detection → write memtle.yaml
    search.rs          CLI search output
    wakeup.rs          L0 + L1 assembly and print
    compress.rs        AAAK batch compression
    split.rs           Mega-file session splitter
    status.rs          Palace stats display
    repair.rs          Backup + rebuild inverted index

  llm/                 LLM provider abstraction (optional, used by init --llm)
    client.rs          LlmProvider trait + Ollama / OpenAI-compat / Anthropic impls
    refine.rs          Batched entity refinement: classify, drop, reclassify via LLM

  palace/
    miner.rs           Project file scanner + chunker + drawer writer; MineParams struct
    convo_miner.rs     Conversation file scanner + normaliser + drawer writer
    drawer.rs          add_drawer(), file_already_mined(), inverted index maintenance
    chunker.rs         chunk_text(): 800-char chunks with 100-char overlap
    search.rs          search_memories(): inverted index query with relevance scoring
    room_detect.rs     70+ folder-to-room mappings, detect_room(), detect_rooms_from_folders()
    query_sanitizer.rs 4-step sanitizer: strip system-prompt contamination from search queries
    entities.rs        Shared DetectedEntity type used by entity_detect and project_scanner
    entity_detect.rs   Person vs project heuristic classifier for prose files
    entity_confirm.rs  Interactive / --yes entity confirmation UX
    known_entities.rs  Global entity registry (~/.local/share/memtle/known_entities.json)
    project_scanner.rs Manifest parsing + git author scanning → DetectedDict
    session_scanner.rs Claude Code project dir scanning for entity discovery
    corpus_origin.rs   Heuristic + LLM detection of whether corpus is AI conversation
    layers.rs          L0 identity + L1 essential story assembly
    graph.rs           BFS traversal, auto-tunnel detection, explicit tunnel CRUD

  kg/
    mod.rs             Entity + triple CRUD
    query.rs           query_entity(), kg_timeline()

  normalize/           Chat export parsers → canonical transcript text
    claude_code.rs     JSONL (Claude Code); strip_noise() removes UI chrome / system-reminder tags
    claude_ai.rs       JSON array (Claude.ai) + privacy export format
    codex.rs           JSONL (OpenAI Codex CLI)
    chatgpt.rs         ChatGPT export JSON
    slack.rs           Slack export JSON
    gemini_cli.rs      Gemini CLI JSONL (detects via session_metadata sentinel)

  dialect/             AAAK compression
    mod.rs             compress(): header + content line assembly
    emotions.rs        38 emotion codes, keyword → code mapping
    topics.rs          Topic extraction with proper-noun frequency boost

  extract/             Memory type classifier (used in general extraction mode)
    mod.rs             5-type classifier: decision, preference, milestone, problem, emotional
    markers.rs         ~80 regex patterns

  mcp/
    mod.rs             Async stdio JSON-RPC 2.0 event loop
    protocol.rs        PALACE_PROTOCOL, AAAK_SPEC, 26 tool schemas (diary tools include optional wing)
    tools.rs           Tool dispatch + all 26 handler implementations

---

Differences from the Python version

Area	Python	Rust
Search	ChromaDB semantic / embedding	Keyword inverted index (BM25-style)
memtle_search score field	similarity (0–1 cosine)	similarity (word hit count)
Storage	ChromaDB + SQLite	Single turso (SQLite) file
Binary size	~100MB Python env	~13MB binary
Concurrency	SQLite locking issues	WAL mode; resolved at turso layer
Duplicate detection	0.9 cosine threshold	Keyword overlap threshold
Entity registry	Wikipedia lookups	Heuristic only (deferred)
Onboarding wizard	Interactive	Interactive (memtle onboard)
ChromaDB import	N/A	Not implemented (deferred)
Gitignore support	Full (projects)	Full (ignore crate)
Repair command	Yes	Yes (memtle repair)
Conversation formats	Limited	Extended (+ Codex CLI, Gemini CLI)
MCP error responses	Generic	Generic
Query sanitizer	Yes (issue #333)	Yes (ported from the Python version)

---

Code Style

See STYLEGUIDE.md for the full coding conventions: assertions, loop bounds, no-abbreviation naming, function length limits, clippy configuration, and the "always say why" comment rule.

---

Tests

cargo nextest run