arcadedb
arcadedb copied to clipboard
ArcadeData
Vector search is much faster than hnswlib, but returns different neighbors
First, kudos on the default vector index (JVector) — the speedups are substantial.
We compared the default implementation against the legacy hnswlib index on both small (~10k vectors) and large (~86k vectors) datasets:
-
Index build
- JVector: ~0.6–0.7s for ~86k vectors
- hnswlib: ~25–30 min for the same data
-
Query latency
- JVector (after warmup): ~0.002–0.004s
- hnswlib: ~0.02–0.05s
Performance-wise, JVector is clearly superior.
Differences in retrieval behavior
However, we consistently observe differences in neighbor selection compared to hnswlib.
These differences:
- Appear on both small and large movielens datasets
- Are not limited to distance scaling or ordering
- Persist after warmup (though results stabilize)
Concrete examples
Query: Jurassic Park (1993)
For the same embedding model and query vector:
JVector
1. Jurassic World (2015) (distance: 0.0952)
2. Jurassic Attack (2012) (distance: 0.0992)
3. Jurassic World Dominion (2022) (distance: 0.1018)
4. Jurassic Hunt (2021) (distance: 0.1153)
5. The Jurassic Games (2018) (distance: 0.1183)
hnswlib
1. Jurassic World (2015) (distance: 0.1904)
2. Jurassic Attack (2012) (distance: 0.1983)
3. Jurassic World Dominion (2022) (distance: 0.2036)
4. Jurassic Hunt (2021) (distance: 0.2307)
5. The Jurassic Games (2018) (distance: 0.2366)
Query: Forrest Gump (1994)
Another example (same query, same model):
JVector
1. Favor, The (1994) (distance: 0.1848)
2. What Happened Was... (1994) (distance: 0.1882)
3. Pillertrillaren (1994) (distance: 0.1915)
4. In Custody (1994) (distance: 0.1927)
5. Pulp Fiction (1994) (distance: 0.1928)
hnswlib
1. War, The (1994) (distance: 0.3277)
2. In the Army Now (1994) (distance: 0.3507)
3. This Means War (2012) (distance: 0.3533)
4. Fortunes of War (1994) (distance: 0.3554)
5. Men of War (1994) (distance: 0.3559)
These are different but still semantically reasonable results, indicating that both systems retrieve valid neighbors, but with different selection semantics. We are not expecting strict equivalence with hnswlib, but would like to understand the intended recall / determinism guarantees of the default index.
Additional observations / hypothesis
Based on logs and behavior, it seems likely that the differences are influenced by:
-
Lazy graph construction in JVector (first query triggers graph build)
-
Graph rebuilds from pages, where:
- Deleted / obsolete entries are skipped
- Active vector count may differ slightly from total records
-
Different graph traversal and entry-point heuristics compared to classic HNSW
In particular, early queries (before full graph materialization) can return different neighbors than later queries, and even after warmup, the neighbor sets are not identical to hnswlib.
This looks like a design tradeoff rather than a correctness bug, but it would be helpful to understand the guarantees.
Questions
-
Is the difference in neighbor selection expected by design given JVector’s hybrid / lazy graph construction?
-
Is there a way to:
- Force eager graph construction
- Avoid query-time rebuilds
- Improve determinism or recall guarantees?
-
Are there documented expectations around equivalence vs classic HNSW behavior?
The performance gains are excellent; we mainly want to understand the expected correctness and equivalence guarantees of the default vector index.
Happy to provide a minimal reproducer if useful.
Are you using the same similarity measure?
yes (cosine). everything is the same in this experiment, except the vector search implementations (jvector vs. hnswlib).
Are you using the same similarity measure?
yes (cosine). everything is the same in this experiment, except the vector search implementations (jvector vs. hnswlib).
oh yeah jvector's cosine metric (range [0, 1]) is half of hnswlib metric (range [0, 2])
oh yeah jvector's cosine metric (range [0, 1]) is half of hnswlib metric (range [0, 2])
Is it a bug or a feaure?
Is it a bug or a feaure?
A point of view. There a different valid ways to shift a cosine image, you can 1.0*cos(x)+1.0 or 0.5*cos(x)+0.5. This should't make much difference for the similarity measure as it is just a scale difference, but maybe the annihilation error made by the addition is differing?
oh yeah jvector's cosine metric (range [0, 1]) is half of hnswlib metric (range [0, 2])
Is it a bug or a feaure?
I think either way is fine, as long as it's consistent. I prefer range [0, 1]
looked more into the warmup & persistence issue of jvector at https://github.com/ArcadeData/arcadedb/issues/2915
@robfrank @gramian
I looked more into jvector vs. hnswlib returning different results.
# Benchmark Results
This benchmark compares ArcadeDB's vector index implementations (JVector vs HNSWLib) across multiple dataset sizes and index parameters.
## Dataset Information
**1. SIFT-128-Euclidean (`sift-128-euclidean.hdf5`)**
- **File Size:** 501 MB
- **Total Vectors:** 1,000,000 (1 Million)
- **Dimensions:** 128
- **Test Queries:** 10,000
* **Name**: sift-128-euclidean
* **Source**: http://ann-benchmarks.com/sift-128-euclidean.hdf5
* **Metric**: Euclidean
* **K Values**: [10, 50, 100]
## Targets & Use Cases
| k | Recall target | Hit target | Use case |
| ------ | ------------- | ---------- | -------------------------- |
| 10 | 0.85–0.95 | 0.90–0.95 | Small corpora only |
| **50** | **≥ 0.95** | **≥ 0.99** | **Default RAG** |
| 100 | 0.90–0.93 | ≈ 1.0 | Very large / noisy corpora |
As k increases, Recall@k can go down — but Hit@k must go up.
**Note:**
- **Parameter Semantics**: `m` and `ef` are not exactly equivalent between implementations:
- **JVector**: `m` -> `max_connections`, `ef` -> `beam_width`.
- **HNSWLib**: `m` -> `m`, `ef` -> `ef`.
- **HNSWLib**: `ef_construction` is fixed at 200. `ef` controls search recall/speed dynamically.
- **Metric Equations**:
- **Euclidean (JVector)**: Similarity = $1 / (1 + d^2)$ (Higher is better)
- **Euclidean (HNSWLib)**: Distance = $d^2$ (Lower is better)
- **Cosine (JVector)**: Distance = $(1 - \cos(\theta)) / 2$ (Lower is better)
- **Cosine (HNSWLib)**: Distance = $1 - \cos(\theta)$ (Lower is better)
## Scenario: Tiny
- **Dimensions**: 128
- **Vector Count**: 1000
- **Queries**: 10
- **Data Load Time**: 0.0016s
- **Ground Truth Time**: 0.0008s
- **DB Setup Time**: 1.3248s
### k = 10
| m | ef | J_Recall (Avg±Std) | H_Recall (Avg±Std) | J_Hit (Avg) | H_Hit (Avg) | J_Build(s) | H_Build(s) | J_Lat(ms) | H_Lat(ms) |
| --- | --- | ------------------ | ------------------ | ----------- | ----------- | ---------- | ---------- | --------- | ---------- |
| 8 | 16 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.3060 | 2.2226 | 6.61±4.14 | 12.15±3.97 |
| 8 | 32 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.1125 | 1.6152 | 4.93±2.70 | 2.58±0.25 |
| 8 | 64 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0878 | 1.2914 | 2.28±0.41 | 3.77±0.30 |
| 8 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0567 | 1.1799 | 2.76±0.60 | 3.20±1.47 |
| 8 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0632 | 1.0869 | 2.31±0.75 | 2.88±0.09 |
| 16 | 16 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0267 | 1.1120 | 1.83±0.45 | 1.66±0.13 |
| 16 | 32 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0398 | 1.2263 | 1.84±0.21 | 4.01±1.24 |
| 16 | 64 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0517 | 1.1085 | 3.37±1.77 | 1.74±0.12 |
| 16 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0452 | 0.9787 | 2.10±0.67 | 2.34±0.46 |
| 16 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0449 | 1.0525 | 2.91±1.39 | 3.37±0.18 |
| 32 | 16 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0192 | 1.1375 | 1.83±0.42 | 2.40±0.33 |
| 32 | 32 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0225 | 1.1067 | 1.83±0.47 | 2.35±0.12 |
| 32 | 64 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0323 | 1.1705 | 2.12±0.37 | 1.67±0.27 |
| 32 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0428 | 1.0222 | 2.04±0.33 | 2.40±0.17 |
| 32 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0672 | 1.2103 | 2.65±0.38 | 5.51±0.19 |
| 64 | 16 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0260 | 1.5606 | 2.43±0.13 | 1.92±0.18 |
| 64 | 32 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0296 | 1.6095 | 2.00±0.35 | 3.78±0.32 |
| 64 | 64 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0635 | 1.3177 | 2.73±0.38 | 4.77±8.11 |
| 64 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0433 | 1.6287 | 2.07±0.23 | 3.05±0.20 |
| 64 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0880 | 1.3130 | 3.06±0.24 | 3.23±0.10 |
| 128 | 16 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0420 | 2.3575 | 1.44±0.20 | 1.97±0.13 |
| 128 | 32 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0299 | 2.1802 | 1.96±0.45 | 2.01±0.23 |
| 128 | 64 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0289 | 2.3398 | 2.17±0.13 | 2.99±0.49 |
| 128 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0463 | 2.4498 | 2.21±0.10 | 3.43±0.23 |
| 128 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0544 | 2.4382 | 1.88±0.26 | 6.05±7.97 |
### k = 50
| m | ef | J_Recall (Avg±Std) | H_Recall (Avg±Std) | J_Hit (Avg) | H_Hit (Avg) | J_Build(s) | H_Build(s) | J_Lat(ms) | H_Lat(ms) |
| --- | --- | ------------------ | ------------------ | ----------- | ----------- | ---------- | ---------- | --------- | ---------- |
| 8 | 16 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.3060 | 2.2226 | 6.61±4.14 | 12.15±3.97 |
| 8 | 32 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.1125 | 1.6152 | 4.93±2.70 | 2.58±0.25 |
| 8 | 64 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0878 | 1.2914 | 2.28±0.41 | 3.77±0.30 |
| 8 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0567 | 1.1799 | 2.76±0.60 | 3.20±1.47 |
| 8 | 256 | 0.9960±0.0080 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0632 | 1.0869 | 2.31±0.75 | 2.88±0.09 |
| 16 | 16 | 0.9980±0.0060 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0267 | 1.1120 | 1.83±0.45 | 1.66±0.13 |
| 16 | 32 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0398 | 1.2263 | 1.84±0.21 | 4.01±1.24 |
| 16 | 64 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0517 | 1.1085 | 3.37±1.77 | 1.74±0.12 |
| 16 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0452 | 0.9787 | 2.10±0.67 | 2.34±0.46 |
| 16 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0449 | 1.0525 | 2.91±1.39 | 3.37±0.18 |
| 32 | 16 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0192 | 1.1375 | 1.83±0.42 | 2.40±0.33 |
| 32 | 32 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0225 | 1.1067 | 1.83±0.47 | 2.35±0.12 |
| 32 | 64 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0323 | 1.1705 | 2.12±0.37 | 1.67±0.27 |
| 32 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0428 | 1.0222 | 2.04±0.33 | 2.40±0.17 |
| 32 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0672 | 1.2103 | 2.65±0.38 | 5.51±0.19 |
| 64 | 16 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0260 | 1.5606 | 2.43±0.13 | 1.92±0.18 |
| 64 | 32 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0296 | 1.6095 | 2.00±0.35 | 3.78±0.32 |
| 64 | 64 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0635 | 1.3177 | 2.73±0.38 | 4.77±8.11 |
| 64 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0433 | 1.6287 | 2.07±0.23 | 3.05±0.20 |
| 64 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0880 | 1.3130 | 3.06±0.24 | 3.23±0.10 |
| 128 | 16 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0420 | 2.3575 | 1.44±0.20 | 1.97±0.13 |
| 128 | 32 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0299 | 2.1802 | 1.96±0.45 | 2.01±0.23 |
| 128 | 64 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0289 | 2.3398 | 2.17±0.13 | 2.99±0.49 |
| 128 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0463 | 2.4498 | 2.21±0.10 | 3.43±0.23 |
| 128 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0544 | 2.4382 | 1.88±0.26 | 6.05±7.97 |
### k = 100
| m | ef | J_Recall (Avg±Std) | H_Recall (Avg±Std) | J_Hit (Avg) | H_Hit (Avg) | J_Build(s) | H_Build(s) | J_Lat(ms) | H_Lat(ms) |
| --- | --- | ------------------ | ------------------ | ----------- | ----------- | ---------- | ---------- | --------- | ---------- |
| 8 | 16 | 0.9860±0.0120 | 0.9980±0.0060 | 1.0000 | 1.0000 | 0.3060 | 2.2226 | 6.61±4.14 | 12.15±3.97 |
| 8 | 32 | 0.9910±0.0094 | 0.9980±0.0060 | 1.0000 | 1.0000 | 0.1125 | 1.6152 | 4.93±2.70 | 2.58±0.25 |
| 8 | 64 | 0.9940±0.0080 | 0.9980±0.0060 | 1.0000 | 1.0000 | 0.0878 | 1.2914 | 2.28±0.41 | 3.77±0.30 |
| 8 | 128 | 0.9950±0.0081 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0567 | 1.1799 | 2.76±0.60 | 3.20±1.47 |
| 8 | 256 | 0.9900±0.0110 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0632 | 1.0869 | 2.31±0.75 | 2.88±0.09 |
| 16 | 16 | 0.9970±0.0046 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0267 | 1.1120 | 1.83±0.45 | 1.66±0.13 |
| 16 | 32 | 0.9990±0.0030 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0398 | 1.2263 | 1.84±0.21 | 4.01±1.24 |
| 16 | 64 | 0.9990±0.0030 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0517 | 1.1085 | 3.37±1.77 | 1.74±0.12 |
| 16 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0452 | 0.9787 | 2.10±0.67 | 2.34±0.46 |
| 16 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0449 | 1.0525 | 2.91±1.39 | 3.37±0.18 |
| 32 | 16 | 1.0000±0.0000 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0192 | 1.1375 | 1.83±0.42 | 2.40±0.33 |
| 32 | 32 | 0.9990±0.0030 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0225 | 1.1067 | 1.83±0.47 | 2.35±0.12 |
| 32 | 64 | 1.0000±0.0000 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0323 | 1.1705 | 2.12±0.37 | 1.67±0.27 |
| 32 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0428 | 1.0222 | 2.04±0.33 | 2.40±0.17 |
| 32 | 256 | 0.9990±0.0030 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0672 | 1.2103 | 2.65±0.38 | 5.51±0.19 |
| 64 | 16 | 1.0000±0.0000 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0260 | 1.5606 | 2.43±0.13 | 1.92±0.18 |
| 64 | 32 | 1.0000±0.0000 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0296 | 1.6095 | 2.00±0.35 | 3.78±0.32 |
| 64 | 64 | 1.0000±0.0000 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0635 | 1.3177 | 2.73±0.38 | 4.77±8.11 |
| 64 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0433 | 1.6287 | 2.07±0.23 | 3.05±0.20 |
| 64 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0880 | 1.3130 | 3.06±0.24 | 3.23±0.10 |
| 128 | 16 | 0.9990±0.0030 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0420 | 2.3575 | 1.44±0.20 | 1.97±0.13 |
| 128 | 32 | 1.0000±0.0000 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0299 | 2.1802 | 1.96±0.45 | 2.01±0.23 |
| 128 | 64 | 1.0000±0.0000 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0289 | 2.3398 | 2.17±0.13 | 2.99±0.49 |
| 128 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0463 | 2.4498 | 2.21±0.10 | 3.43±0.23 |
| 128 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0544 | 2.4382 | 1.88±0.26 | 6.05±7.97 |
## Scenario: Small
- **Dimensions**: 128
- **Vector Count**: 10000
- **Queries**: 100
- **Data Load Time**: 0.0039s
- **Ground Truth Time**: 0.0444s
- **DB Setup Time**: 1.1554s
### k = 10
| m | ef | J_Recall (Avg±Std) | H_Recall (Avg±Std) | J_Hit (Avg) | H_Hit (Avg) | J_Build(s) | H_Build(s) | J_Lat(ms) | H_Lat(ms) |
| --- | --- | ------------------ | ------------------ | ----------- | ----------- | ---------- | ---------- | ---------- | ---------- |
| 8 | 16 | 0.9560±0.0852 | 0.9980±0.0199 | 1.0000 | 1.0000 | 0.2399 | 38.2958 | 3.24±12.13 | 5.03±12.92 |
| 8 | 32 | 0.9850±0.0456 | 0.9980±0.0199 | 1.0000 | 1.0000 | 0.1574 | 35.8491 | 2.04±0.55 | 4.45±11.59 |
| 8 | 64 | 0.9870±0.0439 | 0.9980±0.0199 | 1.0000 | 1.0000 | 0.1801 | 35.8696 | 2.76±1.03 | 5.27±12.61 |
| 8 | 128 | 0.9880±0.0431 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.3277 | 37.9060 | 2.16±0.37 | 5.40±11.97 |
| 8 | 256 | 0.9940±0.0237 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.3496 | 41.5154 | 1.91±0.50 | 9.69±12.09 |
| 16 | 16 | 0.9990±0.0099 | 0.9980±0.0199 | 1.0000 | 1.0000 | 0.1849 | 28.8327 | 4.07±12.49 | 3.60±1.01 |
| 16 | 32 | 0.9990±0.0099 | 0.9980±0.0199 | 1.0000 | 1.0000 | 0.2870 | 32.9817 | 2.27±0.65 | 5.10±12.48 |
| 16 | 64 | 1.0000±0.0000 | 0.9980±0.0199 | 1.0000 | 1.0000 | 0.5399 | 34.1274 | 2.36±0.67 | 6.79±17.56 |
| 16 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 1.0970 | 31.0514 | 3.26±1.43 | 6.14±11.06 |
| 16 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.8088 | 30.4205 | 2.60±0.51 | 9.61±14.70 |
| 32 | 16 | 1.0000±0.0000 | 0.9980±0.0199 | 1.0000 | 1.0000 | 0.2526 | 24.8727 | 3.20±0.56 | 3.82±0.98 |
| 32 | 32 | 1.0000±0.0000 | 0.9980±0.0199 | 1.0000 | 1.0000 | 0.2658 | 24.9629 | 2.53±0.36 | 4.25±11.26 |
| 32 | 64 | 1.0000±0.0000 | 0.9980±0.0199 | 1.0000 | 1.0000 | 0.4837 | 25.7623 | 3.00±0.93 | 6.21±19.20 |
| 32 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.5397 | 26.9713 | 4.21±11.84 | 6.74±16.08 |
| 32 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.8179 | 22.6187 | 4.25±10.59 | 6.99±13.08 |
| 64 | 16 | 0.9990±0.0099 | 0.9980±0.0199 | 1.0000 | 1.0000 | 0.2165 | 25.0248 | 2.90±0.82 | 5.45±13.39 |
| 64 | 32 | 1.0000±0.0000 | 0.9980±0.0199 | 1.0000 | 1.0000 | 0.2993 | 26.1632 | 3.18±0.89 | 5.08±12.94 |
| 64 | 64 | 1.0000±0.0000 | 0.9980±0.0199 | 1.0000 | 1.0000 | 0.4756 | 28.5828 | 3.20±1.09 | 7.28±17.04 |
| 64 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.9861 | 26.3436 | 3.51±0.88 | 5.79±12.50 |
| 64 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 1.1579 | 25.5012 | 4.29±1.00 | 8.01±14.05 |
| 128 | 16 | 1.0000±0.0000 | 0.9980±0.0199 | 1.0000 | 1.0000 | 0.2726 | 29.4576 | 2.58±0.44 | 4.38±11.98 |
| 128 | 32 | 1.0000±0.0000 | 0.9980±0.0199 | 1.0000 | 1.0000 | 0.2686 | 29.9900 | 4.38±11.46 | 5.69±13.79 |
| 128 | 64 | 1.0000±0.0000 | 0.9980±0.0199 | 1.0000 | 1.0000 | 0.4299 | 31.8733 | 3.64±0.91 | 3.18±0.62 |
| 128 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.5181 | 29.9800 | 3.22±0.82 | 5.83±10.78 |
| 128 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.9942 | 27.5722 | 4.08±1.05 | 9.65±17.75 |
### k = 50
| m | ef | J_Recall (Avg±Std) | H_Recall (Avg±Std) | J_Hit (Avg) | H_Hit (Avg) | J_Build(s) | H_Build(s) | J_Lat(ms) | H_Lat(ms) |
| --- | --- | ------------------ | ------------------ | ----------- | ----------- | ---------- | ---------- | ---------- | ---------- |
| 8 | 16 | 0.9358±0.0598 | 0.9948±0.0115 | 1.0000 | 1.0000 | 0.2399 | 38.2958 | 3.24±12.13 | 5.03±12.92 |
| 8 | 32 | 0.9604±0.0475 | 0.9948±0.0115 | 1.0000 | 1.0000 | 0.1574 | 35.8491 | 2.04±0.55 | 4.45±11.59 |
| 8 | 64 | 0.9708±0.0399 | 0.9948±0.0115 | 1.0000 | 1.0000 | 0.1801 | 35.8696 | 2.76±1.03 | 5.27±12.61 |
| 8 | 128 | 0.9726±0.0385 | 0.9984±0.0054 | 1.0000 | 1.0000 | 0.3277 | 37.9060 | 2.16±0.37 | 5.40±11.97 |
| 8 | 256 | 0.9738±0.0343 | 0.9998±0.0020 | 1.0000 | 1.0000 | 0.3496 | 41.5154 | 1.91±0.50 | 9.69±12.09 |
| 16 | 16 | 0.9896±0.0209 | 0.9982±0.0064 | 1.0000 | 1.0000 | 0.1849 | 28.8327 | 4.07±12.49 | 3.60±1.01 |
| 16 | 32 | 0.9964±0.0111 | 0.9982±0.0064 | 1.0000 | 1.0000 | 0.2870 | 32.9817 | 2.27±0.65 | 5.10±12.48 |
| 16 | 64 | 0.9982±0.0057 | 0.9982±0.0064 | 1.0000 | 1.0000 | 0.5399 | 34.1274 | 2.36±0.67 | 6.79±17.56 |
| 16 | 128 | 0.9976±0.0071 | 0.9994±0.0034 | 1.0000 | 1.0000 | 1.0970 | 31.0514 | 3.26±1.43 | 6.14±11.06 |
| 16 | 256 | 0.9968±0.0084 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.8088 | 30.4205 | 2.60±0.51 | 9.61±14.70 |
| 32 | 16 | 0.9956±0.0112 | 0.9986±0.0058 | 1.0000 | 1.0000 | 0.2526 | 24.8727 | 3.20±0.56 | 3.82±0.98 |
| 32 | 32 | 0.9988±0.0055 | 0.9986±0.0058 | 1.0000 | 1.0000 | 0.2658 | 24.9629 | 2.53±0.36 | 4.25±11.26 |
| 32 | 64 | 0.9998±0.0020 | 0.9986±0.0058 | 1.0000 | 1.0000 | 0.4837 | 25.7623 | 3.00±0.93 | 6.21±19.20 |
| 32 | 128 | 0.9994±0.0044 | 0.9996±0.0028 | 1.0000 | 1.0000 | 0.5397 | 26.9713 | 4.21±11.84 | 6.74±16.08 |
| 32 | 256 | 0.9996±0.0028 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.8179 | 22.6187 | 4.25±10.59 | 6.99±13.08 |
| 64 | 16 | 0.9964±0.0114 | 0.9986±0.0058 | 1.0000 | 1.0000 | 0.2165 | 25.0248 | 2.90±0.82 | 5.45±13.39 |
| 64 | 32 | 0.9992±0.0039 | 0.9986±0.0058 | 1.0000 | 1.0000 | 0.2993 | 26.1632 | 3.18±0.89 | 5.08±12.94 |
| 64 | 64 | 0.9998±0.0020 | 0.9986±0.0058 | 1.0000 | 1.0000 | 0.4756 | 28.5828 | 3.20±1.09 | 7.28±17.04 |
| 64 | 128 | 1.0000±0.0000 | 0.9996±0.0028 | 1.0000 | 1.0000 | 0.9861 | 26.3436 | 3.51±0.88 | 5.79±12.50 |
| 64 | 256 | 0.9996±0.0028 | 1.0000±0.0000 | 1.0000 | 1.0000 | 1.1579 | 25.5012 | 4.29±1.00 | 8.01±14.05 |
| 128 | 16 | 0.9956±0.0115 | 0.9986±0.0058 | 1.0000 | 1.0000 | 0.2726 | 29.4576 | 2.58±0.44 | 4.38±11.98 |
| 128 | 32 | 1.0000±0.0000 | 0.9986±0.0058 | 1.0000 | 1.0000 | 0.2686 | 29.9900 | 4.38±11.46 | 5.69±13.79 |
| 128 | 64 | 0.9998±0.0020 | 0.9986±0.0058 | 1.0000 | 1.0000 | 0.4299 | 31.8733 | 3.64±0.91 | 3.18±0.62 |
| 128 | 128 | 0.9998±0.0020 | 0.9996±0.0028 | 1.0000 | 1.0000 | 0.5181 | 29.9800 | 3.22±0.82 | 5.83±10.78 |
| 128 | 256 | 0.9998±0.0020 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.9942 | 27.5722 | 4.08±1.05 | 9.65±17.75 |
### k = 100
| m | ef | J_Recall (Avg±Std) | H_Recall (Avg±Std) | J_Hit (Avg) | H_Hit (Avg) | J_Build(s) | H_Build(s) | J_Lat(ms) | H_Lat(ms) |
| --- | --- | ------------------ | ------------------ | ----------- | ----------- | ---------- | ---------- | ---------- | ---------- |
| 8 | 16 | 0.8955±0.0689 | 0.9852±0.0181 | 1.0000 | 1.0000 | 0.2399 | 38.2958 | 3.24±12.13 | 5.03±12.92 |
| 8 | 32 | 0.9235±0.0569 | 0.9852±0.0181 | 1.0000 | 1.0000 | 0.1574 | 35.8491 | 2.04±0.55 | 4.45±11.59 |
| 8 | 64 | 0.9372±0.0487 | 0.9852±0.0181 | 1.0000 | 1.0000 | 0.1801 | 35.8696 | 2.76±1.03 | 5.27±12.61 |
| 8 | 128 | 0.9388±0.0483 | 0.9941±0.0092 | 1.0000 | 1.0000 | 0.3277 | 37.9060 | 2.16±0.37 | 5.40±11.97 |
| 8 | 256 | 0.9422±0.0446 | 0.9997±0.0017 | 1.0000 | 1.0000 | 0.3496 | 41.5154 | 1.91±0.50 | 9.69±12.09 |
| 16 | 16 | 0.9683±0.0300 | 0.9947±0.0094 | 1.0000 | 1.0000 | 0.1849 | 28.8327 | 4.07±12.49 | 3.60±1.01 |
| 16 | 32 | 0.9845±0.0183 | 0.9947±0.0094 | 1.0000 | 1.0000 | 0.2870 | 32.9817 | 2.27±0.65 | 5.10±12.48 |
| 16 | 64 | 0.9885±0.0136 | 0.9947±0.0094 | 1.0000 | 1.0000 | 0.5399 | 34.1274 | 2.36±0.67 | 6.79±17.56 |
| 16 | 128 | 0.9883±0.0144 | 0.9979±0.0043 | 1.0000 | 1.0000 | 1.0970 | 31.0514 | 3.26±1.43 | 6.14±11.06 |
| 16 | 256 | 0.9873±0.0155 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.8088 | 30.4205 | 2.60±0.51 | 9.61±14.70 |
| 32 | 16 | 0.9788±0.0226 | 0.9956±0.0083 | 1.0000 | 1.0000 | 0.2526 | 24.8727 | 3.20±0.56 | 3.82±0.98 |
| 32 | 32 | 0.9929±0.0108 | 0.9956±0.0083 | 1.0000 | 1.0000 | 0.2658 | 24.9629 | 2.53±0.36 | 4.25±11.26 |
| 32 | 64 | 0.9965±0.0071 | 0.9956±0.0083 | 1.0000 | 1.0000 | 0.4837 | 25.7623 | 3.00±0.93 | 6.21±19.20 |
| 32 | 128 | 0.9983±0.0051 | 0.9982±0.0038 | 1.0000 | 1.0000 | 0.5397 | 26.9713 | 4.21±11.84 | 6.74±16.08 |
| 32 | 256 | 0.9978±0.0052 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.8179 | 22.6187 | 4.25±10.59 | 6.99±13.08 |
| 64 | 16 | 0.9834±0.0203 | 0.9956±0.0083 | 1.0000 | 1.0000 | 0.2165 | 25.0248 | 2.90±0.82 | 5.45±13.39 |
| 64 | 32 | 0.9949±0.0081 | 0.9956±0.0083 | 1.0000 | 1.0000 | 0.2993 | 26.1632 | 3.18±0.89 | 5.08±12.94 |
| 64 | 64 | 0.9991±0.0032 | 0.9956±0.0083 | 1.0000 | 1.0000 | 0.4756 | 28.5828 | 3.20±1.09 | 7.28±17.04 |
| 64 | 128 | 0.9993±0.0026 | 0.9982±0.0038 | 1.0000 | 1.0000 | 0.9861 | 26.3436 | 3.51±0.88 | 5.79±12.50 |
| 64 | 256 | 0.9994±0.0028 | 1.0000±0.0000 | 1.0000 | 1.0000 | 1.1579 | 25.5012 | 4.29±1.00 | 8.01±14.05 |
| 128 | 16 | 0.9846±0.0166 | 0.9956±0.0083 | 1.0000 | 1.0000 | 0.2726 | 29.4576 | 2.58±0.44 | 4.38±11.98 |
| 128 | 32 | 0.9969±0.0059 | 0.9956±0.0083 | 1.0000 | 1.0000 | 0.2686 | 29.9900 | 4.38±11.46 | 5.69±13.79 |
| 128 | 64 | 0.9982±0.0052 | 0.9956±0.0083 | 1.0000 | 1.0000 | 0.4299 | 31.8733 | 3.64±0.91 | 3.18±0.62 |
| 128 | 128 | 0.9994±0.0024 | 0.9982±0.0038 | 1.0000 | 1.0000 | 0.5181 | 29.9800 | 3.22±0.82 | 5.83±10.78 |
| 128 | 256 | 0.9993±0.0026 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.9942 | 27.5722 | 4.08±1.05 | 9.65±17.75 |
# Benchmark Results
This benchmark compares ArcadeDB's vector index implementations (JVector vs HNSWLib) across multiple dataset sizes and index parameters.
## Dataset Information
**2. GloVe-100-Angular (`glove-100-angular.hdf5`)**
- **File Size:** 463 MB
- **Total Vectors:** 1,183,514 (~1.2 Million)
- **Dimensions:** 100
- **Test Queries:** 10,000
* **Name**: glove-100-angular
* **Source**: http://ann-benchmarks.com/glove-100-angular.hdf5
* **Metric**: Cosine
* **K Values**: [10, 50, 100]
## Targets & Use Cases
| k | Recall target | Hit target | Use case |
| ------ | ------------- | ---------- | -------------------------- |
| 10 | 0.85–0.95 | 0.90–0.95 | Small corpora only |
| **50** | **≥ 0.95** | **≥ 0.99** | **Default RAG** |
| 100 | 0.90–0.93 | ≈ 1.0 | Very large / noisy corpora |
As k increases, Recall@k can go down — but Hit@k must go up.
**Note:**
- **Parameter Semantics**: `m` and `ef` are not exactly equivalent between implementations:
- **JVector**: `m` -> `max_connections`, `ef` -> `beam_width`.
- **HNSWLib**: `m` -> `m`, `ef` -> `ef`.
- **HNSWLib**: `ef_construction` is fixed at 200. `ef` controls search recall/speed dynamically.
- **Metric Equations**:
- **Euclidean (JVector)**: Similarity = $1 / (1 + d^2)$ (Higher is better)
- **Euclidean (HNSWLib)**: Distance = $d^2$ (Lower is better)
- **Cosine (JVector)**: Distance = $(1 - \cos(\theta)) / 2$ (Lower is better)
- **Cosine (HNSWLib)**: Distance = $1 - \cos(\theta)$ (Lower is better)
## Scenario: Tiny
- **Dimensions**: 100
- **Vector Count**: 1000
- **Queries**: 10
- **Data Load Time**: 0.0024s
- **Ground Truth Time**: 0.0025s
- **DB Setup Time**: 1.8284s
### k = 10
| m | ef | J_Recall (Avg±Std) | H_Recall (Avg±Std) | J_Hit (Avg) | H_Hit (Avg) | J_Build(s) | H_Build(s) | J_Lat(ms) | H_Lat(ms) |
| --- | --- | ------------------ | ------------------ | ----------- | ----------- | ---------- | ---------- | --------- | ----------- |
| 8 | 16 | 0.9000±0.0775 | 0.9900±0.0300 | 1.0000 | 1.0000 | 0.5872 | 4.1287 | 8.58±3.06 | 7.36±2.83 |
| 8 | 32 | 0.9000±0.1000 | 0.9900±0.0300 | 1.0000 | 1.0000 | 0.0923 | 3.3700 | 3.47±1.11 | 4.94±1.81 |
| 8 | 64 | 0.9400±0.0663 | 0.9900±0.0300 | 1.0000 | 1.0000 | 0.0921 | 3.1004 | 3.06±1.60 | 4.30±0.40 |
| 8 | 128 | 0.9300±0.0640 | 0.9900±0.0300 | 1.0000 | 1.0000 | 0.0561 | 3.0017 | 3.09±0.80 | 4.09±0.52 |
| 8 | 256 | 0.9000±0.0775 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0766 | 2.7670 | 2.21±0.36 | 11.88±14.15 |
| 16 | 16 | 0.9900±0.0300 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0511 | 3.1033 | 1.39±0.26 | 3.19±0.29 |
| 16 | 32 | 0.9700±0.0640 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0911 | 3.0323 | 2.06±0.18 | 3.61±0.40 |
| 16 | 64 | 0.9900±0.0300 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0415 | 2.5198 | 2.71±1.50 | 4.32±0.28 |
| 16 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0525 | 2.8273 | 2.29±0.60 | 5.27±0.38 |
| 16 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0683 | 2.7886 | 3.88±1.96 | 6.73±0.67 |
| 32 | 16 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0392 | 1.8623 | 2.44±0.37 | 3.73±0.25 |
| 32 | 32 | 0.9900±0.0300 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0454 | 1.6662 | 1.64±0.14 | 3.93±0.63 |
| 32 | 64 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0598 | 1.9019 | 2.57±0.23 | 6.56±8.91 |
| 32 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0674 | 1.9971 | 1.92±0.12 | 5.13±0.60 |
| 32 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.1037 | 2.0801 | 1.99±0.11 | 4.68±0.18 |
| 64 | 16 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0540 | 1.8359 | 1.84±0.48 | 2.28±0.12 |
| 64 | 32 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0685 | 1.8867 | 1.96±0.35 | 2.27±0.08 |
| 64 | 64 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.2086 | 1.7060 | 1.83±0.35 | 3.20±0.60 |
| 64 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.2343 | 1.7253 | 2.01±0.08 | 2.99±0.21 |
| 64 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.1688 | 1.7036 | 1.94±0.17 | 6.20±0.47 |
| 128 | 16 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0898 | 2.8904 | 3.43±2.18 | 3.19±0.14 |
| 128 | 32 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0904 | 3.0701 | 2.61±0.69 | 3.53±0.28 |
| 128 | 64 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.2103 | 2.9961 | 2.96±1.41 | 3.74±0.49 |
| 128 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.5450 | 3.1606 | 5.86±1.06 | 4.73±0.15 |
| 128 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.5648 | 2.7474 | 2.86±0.23 | 5.36±1.15 |
### k = 50
| m | ef | J_Recall (Avg±Std) | H_Recall (Avg±Std) | J_Hit (Avg) | H_Hit (Avg) | J_Build(s) | H_Build(s) | J_Lat(ms) | H_Lat(ms) |
| --- | --- | ------------------ | ------------------ | ----------- | ----------- | ---------- | ---------- | --------- | ----------- |
| 8 | 16 | 0.7900±0.0873 | 0.9720±0.0183 | 1.0000 | 1.0000 | 0.5872 | 4.1287 | 8.58±3.06 | 7.36±2.83 |
| 8 | 32 | 0.8220±0.0724 | 0.9720±0.0183 | 1.0000 | 1.0000 | 0.0923 | 3.3700 | 3.47±1.11 | 4.94±1.81 |
| 8 | 64 | 0.8720±0.0508 | 0.9720±0.0183 | 1.0000 | 1.0000 | 0.0921 | 3.1004 | 3.06±1.60 | 4.30±0.40 |
| 8 | 128 | 0.8620±0.0817 | 0.9880±0.0160 | 1.0000 | 1.0000 | 0.0561 | 3.0017 | 3.09±0.80 | 4.09±0.52 |
| 8 | 256 | 0.8600±0.0727 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0766 | 2.7670 | 2.21±0.36 | 11.88±14.15 |
| 16 | 16 | 0.9360±0.0463 | 0.9960±0.0080 | 1.0000 | 1.0000 | 0.0511 | 3.1033 | 1.39±0.26 | 3.19±0.29 |
| 16 | 32 | 0.9600±0.0358 | 0.9960±0.0080 | 1.0000 | 1.0000 | 0.0911 | 3.0323 | 2.06±0.18 | 3.61±0.40 |
| 16 | 64 | 0.9800±0.0179 | 0.9960±0.0080 | 1.0000 | 1.0000 | 0.0415 | 2.5198 | 2.71±1.50 | 4.32±0.28 |
| 16 | 128 | 0.9760±0.0250 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0525 | 2.8273 | 2.29±0.60 | 5.27±0.38 |
| 16 | 256 | 0.9700±0.0300 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0683 | 2.7886 | 3.88±1.96 | 6.73±0.67 |
| 32 | 16 | 0.9900±0.0100 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0392 | 1.8623 | 2.44±0.37 | 3.73±0.25 |
| 32 | 32 | 0.9880±0.0098 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0454 | 1.6662 | 1.64±0.14 | 3.93±0.63 |
| 32 | 64 | 0.9960±0.0080 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0598 | 1.9019 | 2.57±0.23 | 6.56±8.91 |
| 32 | 128 | 0.9980±0.0060 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0674 | 1.9971 | 1.92±0.12 | 5.13±0.60 |
| 32 | 256 | 0.9920±0.0133 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.1037 | 2.0801 | 1.99±0.11 | 4.68±0.18 |
| 64 | 16 | 0.9980±0.0060 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0540 | 1.8359 | 1.84±0.48 | 2.28±0.12 |
| 64 | 32 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0685 | 1.8867 | 1.96±0.35 | 2.27±0.08 |
| 64 | 64 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.2086 | 1.7060 | 1.83±0.35 | 3.20±0.60 |
| 64 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.2343 | 1.7253 | 2.01±0.08 | 2.99±0.21 |
| 64 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.1688 | 1.7036 | 1.94±0.17 | 6.20±0.47 |
| 128 | 16 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0898 | 2.8904 | 3.43±2.18 | 3.19±0.14 |
| 128 | 32 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0904 | 3.0701 | 2.61±0.69 | 3.53±0.28 |
| 128 | 64 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.2103 | 2.9961 | 2.96±1.41 | 3.74±0.49 |
| 128 | 128 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.5450 | 3.1606 | 5.86±1.06 | 4.73±0.15 |
| 128 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.5648 | 2.7474 | 2.86±0.23 | 5.36±1.15 |
### k = 100
| m | ef | J_Recall (Avg±Std) | H_Recall (Avg±Std) | J_Hit (Avg) | H_Hit (Avg) | J_Build(s) | H_Build(s) | J_Lat(ms) | H_Lat(ms) |
| --- | --- | ------------------ | ------------------ | ----------- | ----------- | ---------- | ---------- | --------- | ----------- |
| 8 | 16 | 0.7450±0.0720 | 0.9590±0.0192 | 1.0000 | 1.0000 | 0.5872 | 4.1287 | 8.58±3.06 | 7.36±2.83 |
| 8 | 32 | 0.7950±0.0518 | 0.9590±0.0192 | 1.0000 | 1.0000 | 0.0923 | 3.3700 | 3.47±1.11 | 4.94±1.81 |
| 8 | 64 | 0.8220±0.0360 | 0.9590±0.0192 | 1.0000 | 1.0000 | 0.0921 | 3.1004 | 3.06±1.60 | 4.30±0.40 |
| 8 | 128 | 0.8330±0.0398 | 0.9830±0.0127 | 1.0000 | 1.0000 | 0.0561 | 3.0017 | 3.09±0.80 | 4.09±0.52 |
| 8 | 256 | 0.8190±0.0630 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0766 | 2.7670 | 2.21±0.36 | 11.88±14.15 |
| 16 | 16 | 0.9250±0.0216 | 0.9930±0.0090 | 1.0000 | 1.0000 | 0.0511 | 3.1033 | 1.39±0.26 | 3.19±0.29 |
| 16 | 32 | 0.9480±0.0183 | 0.9930±0.0090 | 1.0000 | 1.0000 | 0.0911 | 3.0323 | 2.06±0.18 | 3.61±0.40 |
| 16 | 64 | 0.9650±0.0180 | 0.9930±0.0090 | 1.0000 | 1.0000 | 0.0415 | 2.5198 | 2.71±1.50 | 4.32±0.28 |
| 16 | 128 | 0.9660±0.0229 | 0.9980±0.0040 | 1.0000 | 1.0000 | 0.0525 | 2.8273 | 2.29±0.60 | 5.27±0.38 |
| 16 | 256 | 0.9580±0.0227 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.0683 | 2.7886 | 3.88±1.96 | 6.73±0.67 |
| 32 | 16 | 0.9870±0.0110 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0392 | 1.8623 | 2.44±0.37 | 3.73±0.25 |
| 32 | 32 | 0.9860±0.0128 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0454 | 1.6662 | 1.64±0.14 | 3.93±0.63 |
| 32 | 64 | 0.9930±0.0064 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0598 | 1.9019 | 2.57±0.23 | 6.56±8.91 |
| 32 | 128 | 0.9940±0.0049 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0674 | 1.9971 | 1.92±0.12 | 5.13±0.60 |
| 32 | 256 | 0.9940±0.0080 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.1037 | 2.0801 | 1.99±0.11 | 4.68±0.18 |
| 64 | 16 | 0.9970±0.0046 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0540 | 1.8359 | 1.84±0.48 | 2.28±0.12 |
| 64 | 32 | 0.9990±0.0030 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0685 | 1.8867 | 1.96±0.35 | 2.27±0.08 |
| 64 | 64 | 1.0000±0.0000 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.2086 | 1.7060 | 1.83±0.35 | 3.20±0.60 |
| 64 | 128 | 1.0000±0.0000 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.2343 | 1.7253 | 2.01±0.08 | 2.99±0.21 |
| 64 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.1688 | 1.7036 | 1.94±0.17 | 6.20±0.47 |
| 128 | 16 | 0.9980±0.0040 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0898 | 2.8904 | 3.43±2.18 | 3.19±0.14 |
| 128 | 32 | 1.0000±0.0000 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.0904 | 3.0701 | 2.61±0.69 | 3.53±0.28 |
| 128 | 64 | 1.0000±0.0000 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.2103 | 2.9961 | 2.96±1.41 | 3.74±0.49 |
| 128 | 128 | 1.0000±0.0000 | 0.9990±0.0030 | 1.0000 | 1.0000 | 0.5450 | 3.1606 | 5.86±1.06 | 4.73±0.15 |
| 128 | 256 | 1.0000±0.0000 | 1.0000±0.0000 | 1.0000 | 1.0000 | 0.5648 | 2.7474 | 2.86±0.23 | 5.36±1.15 |
## Scenario: Small
- **Dimensions**: 100
- **Vector Count**: 10000
- **Queries**: 100
- **Data Load Time**: 0.0091s
- **Ground Truth Time**: 0.0459s
- **DB Setup Time**: 1.1918s
### k = 10
| m | ef | J_Recall (Avg±Std) | H_Recall (Avg±Std) | J_Hit (Avg) | H_Hit (Avg) | J_Build(s) | H_Build(s) | J_Lat(ms) | H_Lat(ms) |
| --- | --- | ------------------ | ------------------ | ----------- | ----------- | ---------- | ---------- | ---------- | ----------- |
| 8 | 16 | 0.6110±0.2800 | 0.9100±0.1170 | 0.9700 | 1.0000 | 0.1405 | 93.7114 | 4.25±17.39 | 10.34±17.13 |
| 8 | 32 | 0.7110±0.2541 | 0.9100±0.1170 | 1.0000 | 1.0000 | 0.3402 | 92.6237 | 1.76±0.47 | 7.81±14.18 |
| 8 | 64 | 0.7380±0.2440 | 0.9100±0.1170 | 1.0000 | 1.0000 | 0.2117 | 98.0199 | 1.87±0.31 | 10.97±20.68 |
| 8 | 128 | 0.7550±0.2291 | 0.9320±0.0947 | 1.0000 | 1.0000 | 0.6585 | 92.4708 | 3.57±12.54 | 15.73±32.10 |
| 8 | 256 | 0.7780±0.2287 | 0.9750±0.0589 | 1.0000 | 1.0000 | 1.1054 | 93.3903 | 2.68±1.25 | 16.56±17.02 |
| 16 | 16 | 0.8310±0.1917 | 0.9790±0.0496 | 1.0000 | 1.0000 | 0.5578 | 207.3885 | 2.63±0.61 | 16.66±17.41 |
| 16 | 32 | 0.8720±0.1631 | 0.9790±0.0496 | 1.0000 | 1.0000 | 1.0065 | 219.8400 | 3.08±1.95 | 15.92±18.87 |
| 16 | 64 | 0.9090±0.1258 | 0.9790±0.0496 | 1.0000 | 1.0000 | 1.1894 | 205.4681 | 3.94±2.39 | 14.96±16.48 |
| 16 | 128 | 0.9120±0.1329 | 0.9850±0.0357 | 1.0000 | 1.0000 | 0.5079 | 205.7746 | 3.34±0.42 | 19.60±23.86 |
| 16 | 256 | 0.9150±0.1260 | 0.9960±0.0196 | 1.0000 | 1.0000 | 0.5171 | 206.0412 | 4.33±15.73 | 30.87±24.15 |
| 32 | 16 | 0.9320±0.1057 | 0.9930±0.0292 | 1.0000 | 1.0000 | 0.4430 | 181.1624 | 3.92±0.98 | 15.46±17.07 |
| 32 | 32 | 0.9520±0.0922 | 0.9930±0.0292 | 1.0000 | 1.0000 | 0.6128 | 179.0480 | 2.91±0.50 | 16.25±19.20 |
| 32 | 64 | 0.9620±0.0718 | 0.9930±0.0292 | 1.0000 | 1.0000 | 0.5703 | 179.0236 | 4.14±0.62 | 17.03±20.79 |
| 32 | 128 | 0.9690±0.0578 | 0.9960±0.0196 | 1.0000 | 1.0000 | 0.7548 | 181.1532 | 4.99±10.51 | 20.67±24.51 |
| 32 | 256 | 0.9670±0.0633 | 0.9990±0.0099 | 1.0000 | 1.0000 | 1.1226 | 181.2866 | 4.60±13.08 | 32.49±27.34 |
| 64 | 16 | 0.9370±0.0956 | 0.9950±0.0260 | 1.0000 | 1.0000 | 0.6293 | 73.6658 | 3.72±0.65 | 12.12±17.53 |
| 64 | 32 | 0.9520±0.0768 | 0.9950±0.0260 | 1.0000 | 1.0000 | 0.8149 | 76.2756 | 3.47±0.72 | 15.40±26.76 |
| 64 | 64 | 0.9630±0.0744 | 0.9950±0.0260 | 1.0000 | 1.0000 | 1.9456 | 75.6124 | 3.64±0.93 | 10.29±17.51 |
| 64 | 128 | 0.9780±0.0502 | 0.9970±0.0171 | 1.0000 | 1.0000 | 1.9361 | 81.6227 | 3.43±0.39 | 19.66±28.72 |
| 64 | 256 | 0.9690±0.0595 | 0.9990±0.0099 | 1.0000 | 1.0000 | 2.5369 | 76.4818 | 4.21±0.94 | 20.74±24.60 |
| 128 | 16 | 0.9190±0.1247 | 0.9950±0.0260 | 1.0000 | 1.0000 | 0.8615 | 75.0088 | 5.02±11.17 | 12.01±22.29 |
| 128 | 32 | 0.9560±0.0804 | 0.9950±0.0260 | 1.0000 | 1.0000 | 1.1306 | 72.9703 | 5.54±0.80 | 9.64±15.93 |
| 128 | 64 | 0.9720±0.0584 | 0.9950±0.0260 | 1.0000 | 1.0000 | 2.3965 | 72.2925 | 4.38±1.11 | 14.87±27.57 |
| 128 | 128 | 0.9670±0.0664 | 0.9970±0.0171 | 1.0000 | 1.0000 | 6.1258 | 71.2034 | 3.21±0.51 | 12.94±16.78 |
| 128 | 256 | 0.9720±0.0549 | 0.9990±0.0099 | 1.0000 | 1.0000 | 6.5363 | 75.5057 | 3.37±0.61 | 27.00±32.66 |
### k = 50
| m | ef | J_Recall (Avg±Std) | H_Recall (Avg±Std) | J_Hit (Avg) | H_Hit (Avg) | J_Build(s) | H_Build(s) | J_Lat(ms) | H_Lat(ms) |
| --- | --- | ------------------ | ------------------ | ----------- | ----------- | ---------- | ---------- | ---------- | ----------- |
| 8 | 16 | 0.5424±0.2219 | 0.8508±0.1143 | 1.0000 | 1.0000 | 0.1405 | 93.7114 | 4.25±17.39 | 10.34±17.13 |
| 8 | 32 | 0.6122±0.2052 | 0.8508±0.1143 | 1.0000 | 1.0000 | 0.3402 | 92.6237 | 1.76±0.47 | 7.81±14.18 |
| 8 | 64 | 0.6528±0.2028 | 0.8508±0.1143 | 1.0000 | 1.0000 | 0.2117 | 98.0199 | 1.87±0.31 | 10.97±20.68 |
| 8 | 128 | 0.6696±0.1926 | 0.8828±0.0930 | 1.0000 | 1.0000 | 0.6585 | 92.4708 | 3.57±12.54 | 15.73±32.10 |
| 8 | 256 | 0.6774±0.1984 | 0.9522±0.0522 | 1.0000 | 1.0000 | 1.1054 | 93.3903 | 2.68±1.25 | 16.56±17.02 |
| 16 | 16 | 0.7544±0.1585 | 0.9400±0.0544 | 1.0000 | 1.0000 | 0.5578 | 207.3885 | 2.63±0.61 | 16.66±17.41 |
| 16 | 32 | 0.8144±0.1367 | 0.9400±0.0544 | 1.0000 | 1.0000 | 1.0065 | 219.8400 | 3.08±1.95 | 15.92±18.87 |
| 16 | 64 | 0.8322±0.1250 | 0.9400±0.0544 | 1.0000 | 1.0000 | 1.1894 | 205.4681 | 3.94±2.39 | 14.96±16.48 |
| 16 | 128 | 0.8460±0.1191 | 0.9586±0.0413 | 1.0000 | 1.0000 | 0.5079 | 205.7746 | 3.34±0.42 | 19.60±23.86 |
| 16 | 256 | 0.8544±0.1138 | 0.9904±0.0159 | 1.0000 | 1.0000 | 0.5171 | 206.0412 | 4.33±15.73 | 30.87±24.15 |
| 32 | 16 | 0.8800±0.0937 | 0.9764±0.0236 | 1.0000 | 1.0000 | 0.4430 | 181.1624 | 3.92±0.98 | 15.46±17.07 |
| 32 | 32 | 0.9092±0.0742 | 0.9764±0.0236 | 1.0000 | 1.0000 | 0.6128 | 179.0480 | 2.91±0.50 | 16.25±19.20 |
| 32 | 64 | 0.9224±0.0703 | 0.9764±0.0236 | 1.0000 | 1.0000 | 0.5703 | 179.0236 | 4.14±0.62 | 17.03±20.79 |
| 32 | 128 | 0.9346±0.0561 | 0.9850±0.0195 | 1.0000 | 1.0000 | 0.7548 | 181.1532 | 4.99±10.51 | 20.67±24.51 |
| 32 | 256 | 0.9288±0.0594 | 0.9974±0.0067 | 1.0000 | 1.0000 | 1.1226 | 181.2866 | 4.60±13.08 | 32.49±27.34 |
| 64 | 16 | 0.8836±0.0838 | 0.9812±0.0215 | 1.0000 | 1.0000 | 0.6293 | 73.6658 | 3.72±0.65 | 12.12±17.53 |
| 64 | 32 | 0.9094±0.0790 | 0.9812±0.0215 | 1.0000 | 1.0000 | 0.8149 | 76.2756 | 3.47±0.72 | 15.40±26.76 |
| 64 | 64 | 0.9338±0.0637 | 0.9812±0.0215 | 1.0000 | 1.0000 | 1.9456 | 75.6124 | 3.64±0.93 | 10.29±17.51 |
| 64 | 128 | 0.9374±0.0549 | 0.9884±0.0168 | 1.0000 | 1.0000 | 1.9361 | 81.6227 | 3.43±0.39 | 19.66±28.72 |
| 64 | 256 | 0.9384±0.0531 | 0.9980±0.0060 | 1.0000 | 1.0000 | 2.5369 | 76.4818 | 4.21±0.94 | 20.74±24.60 |
| 128 | 16 | 0.8682±0.1018 | 0.9818±0.0214 | 1.0000 | 1.0000 | 0.8615 | 75.0088 | 5.02±11.17 | 12.01±22.29 |
| 128 | 32 | 0.9072±0.0775 | 0.9818±0.0214 | 1.0000 | 1.0000 | 1.1306 | 72.9703 | 5.54±0.80 | 9.64±15.93 |
| 128 | 64 | 0.9296±0.0649 | 0.9818±0.0214 | 1.0000 | 1.0000 | 2.3965 | 72.2925 | 4.38±1.11 | 14.87±27.57 |
| 128 | 128 | 0.9378±0.0600 | 0.9886±0.0168 | 1.0000 | 1.0000 | 6.1258 | 71.2034 | 3.21±0.51 | 12.94±16.78 |
| 128 | 256 | 0.9388±0.0592 | 0.9980±0.0060 | 1.0000 | 1.0000 | 6.5363 | 75.5057 | 3.37±0.61 | 27.00±32.66 |
### k = 100
| m | ef | J_Recall (Avg±Std) | H_Recall (Avg±Std) | J_Hit (Avg) | H_Hit (Avg) | J_Build(s) | H_Build(s) | J_Lat(ms) | H_Lat(ms) |
| --- | --- | ------------------ | ------------------ | ----------- | ----------- | ---------- | ---------- | ---------- | ----------- |
| 8 | 16 | 0.4837±0.1877 | 0.7963±0.1177 | 1.0000 | 1.0000 | 0.1405 | 93.7114 | 4.25±17.39 | 10.34±17.13 |
| 8 | 32 | 0.5500±0.1755 | 0.7963±0.1177 | 1.0000 | 1.0000 | 0.3402 | 92.6237 | 1.76±0.47 | 7.81±14.18 |
| 8 | 64 | 0.5842±0.1738 | 0.7963±0.1177 | 1.0000 | 1.0000 | 0.2117 | 98.0199 | 1.87±0.31 | 10.97±20.68 |
| 8 | 128 | 0.5978±0.1736 | 0.8379±0.0987 | 1.0000 | 1.0000 | 0.6585 | 92.4708 | 3.57±12.54 | 15.73±32.10 |
| 8 | 256 | 0.6058±0.1735 | 0.9292±0.0569 | 1.0000 | 1.0000 | 1.1054 | 93.3903 | 2.68±1.25 | 16.56±17.02 |
| 16 | 16 | 0.6958±0.1397 | 0.9081±0.0619 | 1.0000 | 1.0000 | 0.5578 | 207.3885 | 2.63±0.61 | 16.66±17.41 |
| 16 | 32 | 0.7567±0.1326 | 0.9081±0.0619 | 1.0000 | 1.0000 | 1.0065 | 219.8400 | 3.08±1.95 | 15.92±18.87 |
| 16 | 64 | 0.7761±0.1251 | 0.9081±0.0619 | 1.0000 | 1.0000 | 1.1894 | 205.4681 | 3.94±2.39 | 14.96±16.48 |
| 16 | 128 | 0.7928±0.1195 | 0.9351±0.0486 | 1.0000 | 1.0000 | 0.5079 | 205.7746 | 3.34±0.42 | 19.60±23.86 |
| 16 | 256 | 0.8002±0.1141 | 0.9833±0.0177 | 1.0000 | 1.0000 | 0.5171 | 206.0412 | 4.33±15.73 | 30.87±24.15 |
| 32 | 16 | 0.8335±0.0944 | 0.9576±0.0321 | 1.0000 | 1.0000 | 0.4430 | 181.1624 | 3.92±0.98 | 15.46±17.07 |
| 32 | 32 | 0.8706±0.0781 | 0.9576±0.0321 | 1.0000 | 1.0000 | 0.6128 | 179.0480 | 2.91±0.50 | 16.25±19.20 |
| 32 | 64 | 0.8872±0.0736 | 0.9576±0.0321 | 1.0000 | 1.0000 | 0.5703 | 179.0236 | 4.14±0.62 | 17.03±20.79 |
| 32 | 128 | 0.8932±0.0668 | 0.9747±0.0236 | 1.0000 | 1.0000 | 0.7548 | 181.1532 | 4.99±10.51 | 20.67±24.51 |
| 32 | 256 | 0.8914±0.0648 | 0.9954±0.0078 | 1.0000 | 1.0000 | 1.1226 | 181.2866 | 4.60±13.08 | 32.49±27.34 |
| 64 | 16 | 0.8340±0.0884 | 0.9654±0.0282 | 1.0000 | 1.0000 | 0.6293 | 73.6658 | 3.72±0.65 | 12.12±17.53 |
| 64 | 32 | 0.8711±0.0866 | 0.9654±0.0282 | 1.0000 | 1.0000 | 0.8149 | 76.2756 | 3.47±0.72 | 15.40±26.76 |
| 64 | 64 | 0.9006±0.0666 | 0.9654±0.0282 | 1.0000 | 1.0000 | 1.9456 | 75.6124 | 3.64±0.93 | 10.29±17.51 |
| 64 | 128 | 0.9017±0.0656 | 0.9800±0.0202 | 1.0000 | 1.0000 | 1.9361 | 81.6227 | 3.43±0.39 | 19.66±28.72 |
| 64 | 256 | 0.9026±0.0634 | 0.9968±0.0061 | 1.0000 | 1.0000 | 2.5369 | 76.4818 | 4.21±0.94 | 20.74±24.60 |
| 128 | 16 | 0.8259±0.0976 | 0.9659±0.0279 | 1.0000 | 1.0000 | 0.8615 | 75.0088 | 5.02±11.17 | 12.01±22.29 |
| 128 | 32 | 0.8636±0.0814 | 0.9659±0.0279 | 1.0000 | 1.0000 | 1.1306 | 72.9703 | 5.54±0.80 | 9.64±15.93 |
| 128 | 64 | 0.8959±0.0678 | 0.9659±0.0279 | 1.0000 | 1.0000 | 2.3965 | 72.2925 | 4.38±1.11 | 14.87±27.57 |
| 128 | 128 | 0.9026±0.0712 | 0.9802±0.0201 | 1.0000 | 1.0000 | 6.1258 | 71.2034 | 3.21±0.51 | 12.94±16.78 |
| 128 | 256 | 0.9075±0.0694 | 0.9968±0.0061 | 1.0000 | 1.0000 | 6.5363 | 75.5057 | 3.37±0.61 | 27.00±32.66 |
So basically as the dataset size increases, jvector's performance gets worse than hnswlib. Obviously the glove-100-angular dataset is more challenging than the sift-128-euclidean dataset.
Although the number of vectors used for search is not even that big, even high max_connections and beam_width, jvector's recall wouldn't get so high. I also ran a test with 100,000 vectors, and for topk=50, I never got average recall higher than 80% with jvector. I have no doubt that jvector is much faster, but not sure if the performance is adequate for production. Do you think I should also raise this issue to the jvector team?
@tae898 I think the issue is the lazy loading of vectors. We leave the vectors in the documents/vertices, so when the find neighbors is called, the jvector graph (serialized in ArcadeDB pages and lazy loaded) is used, but then the final compare of vectors is done in RAM, that means it's retrieved from the record (document/vertex/etc).
We could try to store the vector ALSO within the jvector graph, this could drastically improve performance on read and slow down the creation of the graph.
I could create a new setting to do this so we can run a benchmark later.
@tae898 how can I run this test on my machine so I can profile ArcadeDB and see where it's spending most of the time?
@lvca
At the moment, the test is done by me internally pulling the latest snapshot of arcadedb from Docker Hub, unzipping the necessary JARs, bundling them up in python wheel, install the python wheel, and then testing it locally. Ofc, I have my own script achieving this, but it won't be so straightforward to you go through this process.
I don't mind redoing a test myself if a fix is made to the main branch and is pushed to the docker hub. If you still wanna test it, I can think about making this more accessible.
@lvca
https://github.com/humemai/arcadedb-embedded-python/blob/48dbaa2fadbfe02f56a957b629d7b596d9703796/bindings/python/examples/06_vector_search_recommendations.py
is the script to run this. I'm rerunning it with two datasets
./run_with_memory_monitor.sh 06-angular "ARCADEDB_JVM_MAX_HEAP='32g' ARCADEDB_JVM_ARGS='-Xms32g' python3 benchmark_vector_params.py --dataset glove-100-angular"
and
./run_with_memory_monitor.sh 06-euclidean "ARCADEDB_JVM_MAX_HEAP='32g' ARCADEDB_JVM_ARGS='-Xms32g' python3 benchmark_vector_params.py --dataset sift-128-euclidean"
I'll give you the log and the recall results.
@lvca @robfrank
I finished running the experiments (ArcadeDB's jvector vs. FAISS's HNSW)
https://github.com/humemai/arcadedb-embedded-python/blob/main/bindings/python/examples/benchmark-vector/README.md
TL;DR
- Speed: ArcadeDB (JVector) is surprisingly fast, often matching or beating in-memory FAISS.
- Recall: ArcadeDB has much lower recall than FAISS (likely due to lazy loading issues).
- Persistence: Works correctly, but has a significant "warmup" latency on the first query.
- Verdict: Promising performance, but recall issues need addressing for high-precision production use.