[MMAP] Add in-memory (mmap) stream index mapping. by rfsaliev · Pull Request #310 · intel/ScalableVectorSearch

rfsaliev · 2026-04-03T13:25:29Z

This pull request adds to CPP Runtime API support for loading vector search indices (FlatIndex and VamanaIndex) from memory-mapped files and memory buffers, in addition to existing stream-based loading. It introduces new API methods, updates internal implementations to manage mapped resources, and adds comprehensive tests to ensure correct behavior for all supported storage types.

Major new features:

Added map_to_file and map_to_memory static methods to both FlatIndex and VamanaIndex classes, allowing indices to be loaded directly from memory-mapped files or memory buffers, in addition to streams. [1] [2]
Implemented the corresponding logic in the source files, including resource management to keep memory-mapped streams alive as long as the index exists. [1] [2]

Internal implementation changes:

Added and updated constructors and helper methods in FlatIndexImpl and VamanaIndexImpl to support mapping from streams, and to manage the lifetime of mapped streams via unique_ptr members. [1] [2] [3] [4] [5]
Included necessary headers for memory stream support in relevant files. [1] [2] [3] [4]
Updated dataset loading logic to enforce that view allocators are only used with memory streams, improving safety and correctness.

Testing improvements:

Added a generic test helper (write_and_map_index) and comprehensive test cases to verify correct loading and querying of indices from memory-mapped files for all supported storage kinds and index types. [1] [2] [3] [4] [5] [6] [7]

rfsaliev · 2026-04-14T14:32:50Z

+
+    // Load from a memory-mapped file.
+    // The file is expected to be in the format produced by save().
+    static Status map_to_file(
+        VamanaIndex** index, const char* path, MetricType metric, StorageKind storage_kind
+    ) noexcept;
+
+    // Load from a memory buffer.
+    // The buffer is expected to be in the format produced by save().
+    static Status map_to_memory(
+        VamanaIndex** index,
+        const void* data,
+        size_t size,
+        MetricType metric,
+        StorageKind storage_kind
+    ) noexcept;


@alexanderguzhva, is this API suitable?

…ests - Introduced `memstream.h` and `memstream.cpp` for memory-mapped stream functionality. - Updated `io.h` and `simple.h` to include memory stream support. - Enhanced `simple.cpp` and `flat.cpp` tests to validate loading from memory streams.

…emory and memory-mapped stream support

…View

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds mmap/in-memory stream support for loading indices and datasets (zero-copy “view” loading) across the core library and C++ runtime API, plus tests to validate behavior.

Changes:

Introduces mmstream/mmstreambuf, is_memory_stream, and pointer helpers (current_ptr/begin_ptr/end_ptr) plus a MemoryStreamAllocator to enable view-based, zero-copy loading.
Updates loading paths (e.g., SimpleData, SimpleGraph, Vamana graph selection) to support view allocators and to enforce memory-stream requirements.
Extends C++ runtime API with map_to_file / map_to_memory for Flat and Vamana indices and adds new coverage.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
include/svs/core/io/memstream.h	Adds mmapped streambuf/istream + memory-stream utilities and allocator for zero-copy loads
tests/svs/core/io/memstream.cpp	Adds unit tests for mmstream, pointer helpers, and memory-stream detection
include/svs/core/data/simple.h	Adds view-only load path from `ContextFreeLoadTable` + memory-stream enforcement
include/svs/core/data/io.h	Skips `populate()` for view allocators; enforces memory-stream constraint
include/svs/core/graph/graph.h	Allows forwarding allocator args during stream load to support view allocators
include/svs/orchestrators/vamana.h	Adjusts graph type selection for view-allocated data (mmapped/index views)
include/svs/lib/array.h	Generalizes “view allocator” DenseArray specialization to support MemoryStreamAllocator
include/svs/quantization/scalar/scalar.h	Fixes min/max init and adjusts SQDataset load APIs + adds mutable `get_datum`
bindings/cpp/include/svs/runtime/flat_index.h	Adds `map_to_file` / `map_to_memory` for FlatIndex
bindings/cpp/src/flat_index.cpp	Implements FlatIndex mapping APIs
bindings/cpp/src/flat_index_impl.h	Holds mapped stream lifetime and adds `map_to_stream` implementation
bindings/cpp/include/svs/runtime/vamana_index.h	Adds `map_to_file` / `map_to_memory` for VamanaIndex
bindings/cpp/src/vamana_index.cpp	Implements VamanaIndex mapping APIs
bindings/cpp/src/vamana_index_impl.h	Holds mapped stream lifetime and adds `map_to_stream` implementation
bindings/cpp/tests/runtime_test.cpp	Adds runtime tests for mapping APIs across storage kinds
tests/svs/core/data/simple.cpp	Adds tests for loading SimpleDataView from stringstream + error-path test
tests/svs/quantization/scalar/scalar.cpp	Adds SQDataset view-load test from stringstream
tests/svs/index/flat/flat.cpp	Adds FlatIndex view-load tests from stringstream and mmapped file
tests/svs/index/vamana/index.cpp	Adds Vamana view-load tests from stringstream/mmapped file + SQ view-load test
tests/CMakeLists.txt	Registers new memstream test source

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 8 comments.

Copilot · 2026-04-22T12:53:32Z

+template <typename CharT, typename Traits>
+struct StreambufAccessor : std::basic_streambuf<CharT, Traits> {
+    static CharT* get(std::basic_streambuf<CharT, Traits>* buf) noexcept {
+        // NOLINTNEXTLINE(cppcoreguidelines-pro-type-static-cast-downcast)
+        return static_cast<StreambufAccessor*>(buf)->gptr();
+    }
+
+    static CharT* begin(std::basic_streambuf<CharT, Traits>* buf) noexcept {
+        // NOLINTNEXTLINE(cppcoreguidelines-pro-type-static-cast-downcast)
+        return static_cast<StreambufAccessor*>(buf)->eback();
+    }
+
+    static CharT* end(std::basic_streambuf<CharT, Traits>* buf) noexcept {
+        // NOLINTNEXTLINE(cppcoreguidelines-pro-type-static-cast-downcast)
+        return static_cast<StreambufAccessor*>(buf)->egptr();
+    }
+};


The static_cast<StreambufAccessor*>(buf) downcast is undefined behavior because buf does not actually point to a StreambufAccessor object. This can break across standard library implementations/RTTI modes and is a memory-safety risk. Prefer a defined mechanism: either (a) only support current_ptr/begin_ptr/end_ptr for SVS-owned streambuf types that expose gptr/eback/egptr via a public virtual interface, or (b) explicitly dynamic_cast to known concrete types (basic_mmstreambuf, basic_spanbuf) and return nullptr for std::basic_stringbuf (or require callers to use ispanstream/mmstream for zero-copy).

Copilot · 2026-04-22T12:53:33Z

+template <typename T, typename CharT, typename Traits = std::char_traits<CharT>>
+[[nodiscard]] T* current_ptr(std::basic_istream<CharT, Traits>& stream) noexcept {
+    static_assert(sizeof(CharT) == 1, "current_ptr requires a 1-byte character type.");
+    if (!is_memory_stream(stream)) {
+        return nullptr;
+    }
+
+    auto* buf = stream.rdbuf();
+    auto begin = detail::StreambufAccessor<CharT, Traits>::begin(buf);
+    auto end = detail::StreambufAccessor<CharT, Traits>::end(buf);
+    if (begin == end) {
+        return nullptr;
+    }
+    auto raw = detail::StreambufAccessor<CharT, Traits>::get(buf);
+
+    // NOLINTNEXTLINE(cppcoreguidelines-pro-type-reinterpret-cast)
+    return reinterpret_cast<T*>(raw);
+}


reinterpret_cast<T*>(raw) can produce misaligned T* when the stream position is not aligned for T (common when reading past headers), which is undefined behavior if the returned pointer is dereferenced. Add an alignment check (e.g., reinterpret_cast<std::uintptr_t>(raw) % alignof(T) == 0) and return nullptr/throw when misaligned, and consider also validating that the remaining buffer size is at least sizeof(T) (or documenting that callers must ensure alignment/size).

Copilot · 2026-04-22T12:53:33Z

+    basic_mmstreambuf* open(
+        const std::filesystem::path& path,
+        std::ios_base::openmode mode = std::ios_base::in | std::ios_base::out
+    ) {


basic_mmstreambuf::open defaults to in | out, which selects ReadWrite mappings by default even though the streambuf explicitly disallows writing (setp(&empty_, &empty_)). This can cause unexpected failures when mapping read-only files and requests stronger permissions than necessary. Consider defaulting to std::ios_base::in (read-only) and only enabling ReadWrite when the implementation actually supports writes for the put area.

Copilot · 2026-04-22T12:53:33Z

+        auto perm =
+            (mode & std::ios_base::out) ? MemoryMapper::ReadWrite : MemoryMapper::ReadOnly;


basic_mmstreambuf::open defaults to in | out, which selects ReadWrite mappings by default even though the streambuf explicitly disallows writing (setp(&empty_, &empty_)). This can cause unexpected failures when mapping read-only files and requests stronger permissions than necessary. Consider defaulting to std::ios_base::in (read-only) and only enabling ReadWrite when the implementation actually supports writes for the put area.

Copilot · 2026-04-22T12:53:34Z

+    [[nodiscard]] pointer allocate(size_type n) {
+        T* current = current_ptr<T>(stream_);
+        if (current == nullptr) {
+            throw std::runtime_error("Failed to obtain current pointer from memory stream."
+            );
+        }
+        pointer result = current;
+        stream_.seekg(n * sizeof(T), std::ios_base::cur);
+        if (!stream_) {
+            throw std::runtime_error("Failed to advance memory stream after allocation.");
+        }
+        return result;
+    }


n * sizeof(T) can overflow size_t (especially for n derived from serialized sizes), leading to an incorrect/negative seek and potentially returning a pointer to an undersized region. Add an overflow check before multiplication and ensure the seek distance fits into std::streamoff/std::streamsize on the target platform.

Copilot · 2026-04-22T12:53:34Z

+#include <array>
+#include <cstddef>
+#include <filesystem>
+#include <fstream>
+#include <span>
+#include <sstream>
+#include <string>
+#include <system_error>
+#include <utility>


std::strlen is used but <cstring> is not included in this test file, which can cause compilation failures depending on standard library headers included transitively. Add #include <cstring>.

Copilot · 2026-04-22T12:53:34Z

+    CATCH_REQUIRE(base_ptr != nullptr);
+    CATCH_REQUIRE(*base_ptr == 'H');
+
+    for (std::size_t i = 0; i < std::strlen(text); ++i) {


std::strlen is used but <cstring> is not included in this test file, which can cause compilation failures depending on standard library headers included transitively. Add #include <cstring>.

Copilot · 2026-04-22T12:53:35Z

+    svs_test::prepare_temp_directory();
+    auto temp_dir = svs_test::temp_directory();
+    auto filename = temp_dir / "index_test.bin";
+
+    // Serialize
+    std::ofstream out(filename, std::ios::binary);


The mapped-index tests always write to the fixed filename index_test.bin, which risks collisions if tests are ever executed concurrently (e.g., parallel CTest) or re-entered. Prefer a unique filename per test invocation (e.g., incorporate PID, timestamp, or a monotonically increasing counter / use UniqueTempDirectory like other tests).

…stream&) to accept custom accessors

…views Co-authored-by: Copilot <copilot@github.com>

ibhati

Looks great to me!

ethanglaser · 2026-04-23T16:10:09Z

Please address CI failure before merging

ethanglaser · 2026-04-23T16:13:25Z

And if it is expected to require updates from private source to work correctly then we'll need to update https://github.com/intel/ScalableVectorSearch/blob/main/bindings/cpp/CMakeLists.txt#L126 with LTO shared lib build from the private PR (can publish as nightly here)

Co-authored-by: Copilot <copilot@github.com>

…ed allocator handling Co-authored-by: Copilot <copilot@github.com>

rfsaliev requested a review from razdoburdin April 3, 2026 13:25

rfsaliev force-pushed the rfsaliev/mmap-loading branch 2 times, most recently from 1b35ca7 to 9e5588f Compare April 14, 2026 13:32

rfsaliev commented Apr 14, 2026

View reviewed changes

Base automatically changed from dev/razdoburdin_streaming to main April 16, 2026 09:55

rfsaliev added 5 commits April 16, 2026 15:40

Enhance mmap functionality for SimpleGraph and Vamana index with in-m…

cec2c73

…emory and memory-mapped stream support

Add scalar quantization mem-mapping tests

ee59fc8

Refactor SimpleData mmapping using MemoryStreamAllocator rather than …

6963335

…View

Add mem-map CPP Runtime API for Flat and (static)Vamana

a5ed391

rfsaliev force-pushed the rfsaliev/mmap-loading branch from 9e5588f to a5ed391 Compare April 16, 2026 14:25

Refactor mmstream/mmstreambuf using MMapPtr

f52cf19

rfsaliev marked this pull request as ready for review April 20, 2026 09:54

rfsaliev requested review from ahuber21, ethanglaser, ibhati, mihaic and yuejiaointel as code owners April 20, 2026 09:54

rfsaliev requested a review from Copilot April 20, 2026 10:00

Copilot AI reviewed Apr 20, 2026

View reviewed changes

Copilot started reviewing on behalf of rfsaliev April 20, 2026 10:26 View session

rfsaliev added 3 commits April 21, 2026 15:05

Enable rebinding for blocked allocator

82e026f

Address some review comments

d2c6cf2

Fix map_to_memory() by using ispanstream

f04b1c2

rfsaliev force-pushed the rfsaliev/mmap-loading branch from 748365f to f04b1c2 Compare April 22, 2026 12:43

rfsaliev requested a review from Copilot April 22, 2026 12:43

Copilot AI reviewed Apr 22, 2026

View reviewed changes

Copilot started reviewing on behalf of rfsaliev April 22, 2026 14:35 View session

rfsaliev and others added 2 commits April 22, 2026 17:24

Enhance DefaultWriteAccessor with vector_reader and update populate(i…

9d50d0e

…stream&) to accept custom accessors

Refactor SimpleData load methods to handle istream compatibility for …

1ffec19

…views Co-authored-by: Copilot <copilot@github.com>

ibhati approved these changes Apr 23, 2026

View reviewed changes

rfsaliev and others added 2 commits April 24, 2026 10:17

Add copy constructor and stream accessor to MemoryStreamAllocator

bf1f6a3

Co-authored-by: Copilot <copilot@github.com>

Fix CPP Runtime map_to_stream() and SimpleData load method for improv…

ab51fef

…ed allocator handling Co-authored-by: Copilot <copilot@github.com>

		auto perm =
		(mode & std::ios_base::out) ? MemoryMapper::ReadWrite : MemoryMapper::ReadOnly;

Conversation

rfsaliev commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rfsaliev Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

ibhati left a comment

Choose a reason for hiding this comment

Uh oh!

ethanglaser commented Apr 23, 2026

Uh oh!

ethanglaser commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rfsaliev commented Apr 3, 2026 •

edited

Loading

ethanglaser commented Apr 23, 2026 •

edited

Loading