Comparing pg_vectorsearch with alternatives on a real dataset

[vemonet]

Hi, I am benchmarking various postgres semantic similarity extensions on the same embeddings for our dataset of ~1.5M datasets

I first ran this benchmark on pg17 with pgvectorscale (adding it as reference to see the improvements)

Then saw the pg_vectrosearch release, so I re-ran it on pg18, comparing:

• pgvectorscale DiskANN (from Tiger Data too)
• pgvector HNSW on halfvec, with the pgvector default build settings (m = 16, ef_construction = 64)
• VectorChord vchordrq (IVF + RaBitQ, reranks from the table) and vchordg (graph)
• pg_vectorsearch PRISM, the new extension released 5 days ago (0.1.0, alpha, PostgreSQL 18+, PostgreSQL license). Same family as vchordrq (IVF + RaBitQ, reranks from the table) (see release comments here: https://news.ycombinator.com/item?id=50086165)

I mainly looked at scalability on machines with limited RAM (that's all I care, ain't nobody got 60GB RAM to waste in a postgres db, we got models to train). It's not a really large dataset, but it's already pushing "search in RAM" methods to the limit

Setup: 1.44M datasets, 3.7M vectors of 768 dims. Each index type built one after the other on the same rows, then queried with 100 real user searches from our logs. Recall is measured against an exact search, the hybrid latency is the full search query (semantic + BM25 on the 3 fields), throughput with 8 concurrent clients. Measured in a docker VM (16GB, 14 cores), each type once with full memory and once with postgres capped at 4GB to simulate a small server, hence the ranges. Running on macOS m4 max

PostgreSQL 17

	vchordrq	pgvectorscale diskann	pgvector hnsw halfvec	vchordrq, vectors in index	vchordg
Vector indexes (need to stay in RAM)	0.71 GB	2.38 GB	7.10 GB	15.0 GB	15.1 GB
Max vectors in RAM on a 16GB server (est.)	~30M	~14M	~5M	~2.7M (less than today)	~2.7M (less than today)
Build time	3.2 min	54-62 min (single-threaded)	3.4 min	4.0 min	29 min
Peak RAM during build	2.6 GB	5.5 GB	~3 GB	not measured	not measured
Best recall@100	0.97	0.66-0.71	0.65	0.97	0.88
ANN latency per field at that recall	33-45 ms	46-52 ms	92 ms	38 ms	85 ms
Hybrid search p50 / p95	101-189 / 158-404 ms	157-223 / 291-439 ms	221-506 / 494-1399 ms	160 / 314 ms	239 / 521 ms
Throughput (8 clients)	28-30 qps	46-56 qps	39-111 qps	79 qps	59 qps

pgvectorscale 0.9.0 parallel build was broken for us: built in parallel, the description index had a recall@100 of 0.18 whatever the search settings, 0.76 when built single-threaded. So diskann has to be built single-threaded, which takes ~1h.

PostgreSQL 18

Local DB upgraded in place with pg_upgrade --link

	pg_vectorsearch prism	vchordrq	pgvectorscale diskann
Vector indexes (need to stay in RAM)	0.96 GB	0.71 GB	2.38 GB
Max vectors in RAM on a 16GB server (est.)	~26M	~30M	~14M
Build time	10.4 min	3.1 min	53 min (single-threaded)
Peak RAM during build	1.6 GB (512MB, no workers) to 12.4 GB (4GB, 8 workers)	2.6 GB	5.5 GB
Best recall@100	0.97	0.97	0.66-0.69
ANN latency per field at that recall	21-27 ms	31-41 ms	45-48 ms
Recall at default / cheap settings	0.75 in 5 ms	0.47 in 8 ms	0.43 in 9 ms
Hybrid search p50 / p95	76-137 / 116-255 ms	106-153 / 170-232 ms	131-136 / 235-239 ms
Throughput (8 clients)	76-87 qps	54-64 qps	66-83 qps

Recall is hard on our data for every type, because of the queries: HNSW is stuck at 0.65 (it returns at most 33 keywords results whatever ef_search), diskann does not go above ~0.7. vchordrq and prism reach 0.97 but only by scanning a big share of their lists (~40% for vchordrq at 800 probes, ~30% for prism at 1600), with their default settings they are at 0.47 and 0.75

Results are pretty good. The permissive licensing, company history of qualitative and reliable open source contributions, and synergia with pg_textsearch give pg_vectorsearch an obvious edge imo

Congrats and thanks!

Do you have a rough idea around when beta and v1 might land for pg_vectorsearch?

[erimatnor]

Thanks for these numbers @vemonet. We are working systematically towards a beta and GA, but we have no dates to announce at this time.

[saolof]

Apart from the performance the license may also be worth including in the table, since pgvector/pgvectorscale/pg_vectorsearch are postgresql licensed while vectorchord is AGPLv3/ELv2 licensed.