You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Memory: ingest images into the brain (description and text from PNG/JPEG/WebP/HEIC) #7066
Adding an image to the brain fails today. OpenHuman's document converter chain (crates/openhuman-core/src/memory/convert.rs, TinyMemory's ConverterChain) has converters for plain text, HTML and Markdown, and, since #7034, PDF, Word, PowerPoint and Excel. No converter claims an image, so the chain answers UnsupportedFormat, and the image is not stored.
#6718 listed blob_ref extraction for images ("the chat model for images") as an ingest path. It was not built, and this issue tracks it on its own so #6718 can close.
What is needed
A converter for common image formats (PNG, JPEG, WebP, HEIC at least) that produces text the brain can index:
a description of the image;
any text in it (screenshots, scanned pages, whiteboards).
A decision on how that text is produced:
a vision model through the existing inference path, which costs credits per image and adds latency; or
local OCR, which is cheaper but has no description, and adds a native dependency. That must stay out of the always-on kernel floor (scripts/kernel-floor.limits).
Long work goes off the request path, like the Office converter does (spawn_blocking / a background job).
A clear error for formats that are still unsupported, shown to the user rather than a silent drop.
Tests: a fixture per format, a failure case, and a check that the cost of each ingest is known.
Open questions
Should a scanned PDF (image-only pages) go through the same path?
What happens to PII redaction on text taken out of images?
Image ingest (PNG/JPEG/WebP/HEIC) into memory: needs a converter producing text via vision model or OCR, plus limits and error handling. This is a new functionality request, not a bug.
Triage: needs opinion — “Memory: ingest images into the brain (description and text from PNG/JPEG/WebP/HEIC)” needs a maintainer decision on product scope/priority or fresh reproduction evidence before its status can be settled. Please advise whether to pursue, narrow, or close it.
Problem
Adding an image to the brain fails today. OpenHuman's document converter chain (
crates/openhuman-core/src/memory/convert.rs, TinyMemory'sConverterChain) has converters for plain text, HTML and Markdown, and, since #7034, PDF, Word, PowerPoint and Excel. No converter claims an image, so the chain answersUnsupportedFormat, and the image is not stored.#6718 listed
blob_refextraction for images ("the chat model for images") as an ingest path. It was not built, and this issue tracks it on its own so #6718 can close.What is needed
scripts/kernel-floor.limits).spawn_blocking/ a background job).Open questions
Related