Repository navigation
Corpus: user documents in Laya's retrieval stack - #56
Merged
Merged
Conversation
Users can share local files (PDF, DOCX, Markdown, text, CSV, JSON, HTML) by dropping them anywhere on the window, picking them on the new Corpus page or attaching them from the chat composer. The engine parses, chunks (page and heading aware), embeds into the shared laya_memory collection, indexes in a new documents_fts table, links identifiers to entities and writes a map-reduce summary at ingest. Files shared by path are read in place and recorded by path, sha256 and mtime (storage_mode=reference); only browser multipart uploads keep a copy. The Corpus page shows source state (changed on disk, moved) with re-sync and relocate actions. Chat retrieval fuses document chunks with cards and events, replies cite [doc:ID p.N], conversations keep attached document ids, and the list_documents / search_documents / read_document tools ship to chat (gated) and MCP (read scope). Cards show related documents; Coherence clusters carry linked documents. Every card-only Chroma reader now filters with CARD_CONTENT_FILTER so document vectors never surface as cards. Engine: migration 074, laya/documents package, api/documents_api.py, pypdf + mammoth pinned and locked. UI: /corpus route, lib/components/corpus, documents store with a tested reducer, shared fileDrop utility. Design: engine/docs/corpus-documents.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q64rmiHsZE6sRH1JrEAHqG
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Closing the add sheet now only hides it: the batch keeps uploading and the engine keeps processing. A footer indicator shows the current batch (active document, done count, progress bar) and reopens the sheet with the stage tiles; once everything finishes it reads "N documents added" (or how many failed) until dismissed. Submission moved out of the component so it survives the sheet being hidden, and a new drop replaces finished tiles from the previous batch. Also drops the BETA marker from the Coherence navigation entry. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q64rmiHsZE6sRH1JrEAHqG
Embedding a batch of long chunks ran the nomic encoder out of GPU memory on Apple silicon (a 28 GiB allocation for 32 chunks). embed_batch now retries a failed slice one vector at a time before failing, the default batch size is 8, and chunk_units cuts any chunk above 3000 characters at whitespace (a table or listing with no sentence boundary could reach several thousand) so no single sequence is unbounded. The batch progress moves from the footer, where it overlapped the centre text, to a banner under the title bar beside the update banner: current document, done count and progress bar while active, then "N documents added" or "N failed" with Details and dismiss. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q64rmiHsZE6sRH1JrEAHqG
A sheet tile whose document was deleted while processing kept counting as
active, so the banner stayed at "1/2" forever. A submitted tile with no
matching document in the store is now a finished tile ("Removed"): the
activity counter, the sheet's Done/Ask buttons and the clear-on-dismiss all
treat it as done, and a new drop replaces it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q64rmiHsZE6sRH1JrEAHqG
Adds Corpus to the README feature list and tree, the landing page (feature card, showcase section with the Corpus screenshot, cost breakdown), the docs site (sidebar section, Documents / Adding Documents / Chat, Cards & Coherence pages, model roles note, prompt files, settings), llms.txt, the privacy page, and the internal docs: architecture diagrams and a Corpus pipeline section, API contracts (chat document_ids, conversation attachments, the documents routes, WebSocket messages), database schema (documents tables, documents_fts, migrations 071-074), project structure, decision log phase 14, tuning parameters and the pipeline lifecycle diagram. Registers the three Corpus prompts (chat_documents, document_summary_map, document_summary_reduce) as overridable files in ~/.laya/prompts/. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q64rmiHsZE6sRH1JrEAHqG
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q64rmiHsZE6sRH1JrEAHqG
Paths given to from-path and relocate must be absolute, are resolved through symlinks, must have a supported extension (checked before any filesystem lookup) and must not be inside ~/.laya. Addresses CodeQL py/path-injection alerts #23 and #24. The document type chip now sits after the status label so chips align at the right edge of every row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Corpus, a peer of Pulse, Omni and Coherence: the user hands Laya a local file (PDF, DOCX, Markdown, text, CSV, JSON, HTML) and it becomes part of the hybrid retrieval stack.
storage_mode=reference); only browser multipart uploads keep a copy under~/.laya/data/documents. The engine never writes to a shared file.engine/laya/documents/): parser registry (onePARSERper format, mirroringegress/platforms/), page/heading-aware chunking overdb/chunking.chunk_segments(chunk_textoutput unchanged, golden-tested), batched embedding intolaya_memoryasdocument_chunk/document_summaryvectors,documents_fts(BM25, LIKE fallback), identifier and name linking to entities, map-reduce LLM summary. Each stage broadcastsdocument_updated; stalled runs are re-queued at startup.chat_conversations.document_ids) force retrieval with a larger budget and pin their summaries into the prompt; replies cite[doc:ID p.N](persisted inreferenced_documents);list_documents/search_documents/read_documenttools ship when the space has documents or the turn mentions one, and reach MCP under the read scope automatically.CARD_CONTENT_FILTER(content_type $nin) applied to the six Chroma readers that assume every hit is a card (chat, trace, related_context, entity_resolution, context_grouping, card_tools)./corpusroute with select-and-ask, source-state badges (changed on disk → Re-sync, moved → Locate), document view with page deep links, add-document sheet with live progress, chat paperclip and chips, citation links, conversation list chip. Sharedlib/utils/fileDrop.tsnow serves both the Run Agent modal and the global drop.pypdf>=6,<7,mammoth>=1.9,<2(pure Python), locks regenerated;check-locks.shandsmoke-install.shpass.Design doc:
engine/docs/corpus-documents.md.Test plan
cd engine && pytest— 1460 passed (new: chunking, parsers incl. scanned/encrypted PDF and DOCX fixtures, linking, summary driver, ingest state machine, documents API, tools; extended: fts, chat, tool gating, mcp scope, trace, spaces)scripts/check-locks.sh -v,scripts/smoke-install.sh(fresh venv, 1463 passed)cd ui && npx svelte-check && npm test— 0 errors/warnings, 274 tests (new reducer tests)ready; 409 duplicate, 415 unsupported; rename, resync (unchanged), chat with an attachment runs document retrieval with no errors; delete leaves the referenced source byte-identical[doc:ID p.N]link, select two documents and "Ask", edit/move the source to see Re-sync/Locate, open a card whose ticket key the document mentions🤖 Generated with Claude Code
https://claude.ai/code/session_01Q64rmiHsZE6sRH1JrEAHqG