TorchPQ icon indicating copy to clipboard operation
TorchPQ copied to clipboard

pre-compute approximate pairwise distances between vectors to use in cuML UMAP

Open choucavalier opened this issue 1 year ago • 1 comments

hey there! i'm having perf issues because cuml.UMAP does not scale well with the size of my dataset.

probably because it performs a greed KNN algorithm, thereby computing billions of distances.

do you know if it's possible to use TorchPQ to pre-compute a sparse distance matrix i can pass to cuml.UMAP here?

choucavalier avatar Dec 05 '23 15:12 choucavalier

Hi, this is currently not possible with TorchPQ.

DeMoriarty avatar Dec 12 '23 23:12 DeMoriarty