Ideas

[ankane]

Please create a new issue to discuss any ideas or share your own.

HNSW

• [ ] Add support for streaming I/O - hnsw-read-stream branch (WIP)
• [ ] Improve colocation of neighbors on disk for index builds - hnsw-build-order branch
• [ ] Add support for inline filtering
  • [ ] HQANN - hqann3 branch (WIP)
  • [ ] ACORN
• [ ] Add time budget option - time-budget branch
• [ ] Use scan->kill_prior_tuple - kill-prior-tuple branch (WIP)
• [ ] Add support for index-only scans - hnsw-index-only-v2 branch
• [ ] Add support for INCLUDE clause with CREATE INDEX (not more performant since still accesses heap)
• [ ] Improve performance of on-disk index builds by keeping neighbors and closer in memory

IVFFlat

• [ ] Add support for inline filtering - ivfflat-filtering branch (needs different design for better performance)
• [ ] Add support for product quantization
• [ ] Add support for another coarse quantizer
• [ ] Add support for parallel k-means
• [ ] Update parallel workers calculation - ivfflat-parallel-workers branch
• [ ] Increase default value of ivfflat.probes
• [ ] Add support for setting default probes on index level
• [ ] Scan additional lists if number of entries is low after scanning probes
• [ ] Use pairing heap for index scan for performance - stages and pairingheap branches
• [ ] Use tuplesort_set_bound for performance - bound branch (not needed w/ pairing heap)
• [ ] Add support for parallel index scans (planner gets cost estimate but doesn't use) - parallel-index-scan4 branch
• [ ] Use Lloyd's or MO-Elkan's k-means when not enough memory for Elkan's
• [ ] Use FB-Elkan's k-means for faster index creation
• [ ] Use mini-batch k-means for index creation for reduced memory - minibatch branch
• [ ] Add support for index-only scans - ivfflat-index-only branch
• [ ] Add support for INCLUDE clause with CREATE INDEX
• [ ] Prevent compression for index tuples - ivfflat-storage branch

Indexes

• [ ] Include vectors with zero norm in indexes for cosine distance - index-zero-norm branch

Types

• [ ] Add support for fp8 vectors (e4m3) - minivec branch
• [ ] Add support for int8 vectors - intvec branch

Functions

• [ ] Add subscript function - subscript branch (requires Postgres 14+)
• [ ] Add angular_distance function - angular_distance branch - #234
• [ ] Add random_vector function - random_vector branch
• [ ] Add functions to view lists and/or pages like pageinspect (require superuser)
• [ ] Add function to estimate index balance (require superuser?) - #105

Installation

• [ ] Create installer for Windows - #108