Skip to content

Memory: ingest images into the brain (description and text from PNG/JPEG/WebP/HEIC) #7066

Description

@CodeGhost21

Problem

Adding an image to the brain fails today. OpenHuman's document converter chain (crates/openhuman-core/src/memory/convert.rs, TinyMemory's ConverterChain) has converters for plain text, HTML and Markdown, and, since #7034, PDF, Word, PowerPoint and Excel. No converter claims an image, so the chain answers UnsupportedFormat, and the image is not stored.

#6718 listed blob_ref extraction for images ("the chat model for images") as an ingest path. It was not built, and this issue tracks it on its own so #6718 can close.

What is needed

  • A converter for common image formats (PNG, JPEG, WebP, HEIC at least) that produces text the brain can index:
    • a description of the image;
    • any text in it (screenshots, scanned pages, whiteboards).
  • A decision on how that text is produced:
    • a vision model through the existing inference path, which costs credits per image and adds latency; or
    • local OCR, which is cheaper but has no description, and adds a native dependency. That must stay out of the always-on kernel floor (scripts/kernel-floor.limits).
  • Long work goes off the request path, like the Office converter does (spawn_blocking / a background job).
  • A clear error for formats that are still unsupported, shown to the user rather than a silent drop.
  • Tests: a fixture per format, a failure case, and a check that the cost of each ingest is known.

Open questions

  • Should a scanned PDF (image-only pages) go through the same path?
  • What happens to PII redaction on text taken out of images?
  • What are the size and dimension limits per image?

Related

Activity

  1. added
    priority: p3Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect.
    on Oct 7, 2026
  2. tinysweeper commented on Oct 7, 2026

    @tinysweeper

    Image ingest (PNG/JPEG/WebP/HEIC) into memory: needs a converter producing text via vision model or OCR, plus limits and error handling. This is a new functionality request, not a bug.

    Labelled priority: p3.

  3. aryash45 commented on Oct 7, 2026

    @aryash45
    Contributor

    @CodeGhost21 @senamakel @M3gA-Mind can i take this feature request

  4. senamakel commented on Oct 9, 2026

    @senamakel
    Member

    Triage: needs opinion — “Memory: ingest images into the brain (description and text from PNG/JPEG/WebP/HEIC)” needs a maintainer decision on product scope/priority or fresh reproduction evidence before its status can be settled. Please advise whether to pursue, narrow, or close it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority: p3Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect.source: teamIssue opened by a repository membertriage: needs-opinionValid issue awaiting a maintainer product or priority decision

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions