Romain Beaumont
Romain Beaumont
It's not expected the memory usage would go higher than specified by this option. It might have been introduced by recent changes in faiss or autofaiss For now you can...
How many images are you working with ? Storing images as individual files tend to break down for many reasons after 100k or so On Sat, Jan 18, 2025, 14:59...
I advise you use a sharded format like webtorrent or tfrecords. Training software can be made much faster when reading such sharded format On Mon, Jan 20, 2025, 02:21 Philip...
Sure feel free to send a PR. On Mon, Jan 20, 2025, 20:34 Philip Brown ***@***.***> wrote: > I dont think thats valid in this case. Training is entirely gated...
Building an hnsw is indeed one of the slowest adding method, especially with random vectors. This is calling faiss index.add If you want to optimize for speed of building an...
What kind of local disk do you have ?
Save it to disk/hdfs as parquet and use the distributed implementation of autofaiss, see readme for details of usage On Thu, May 25, 2023, 12:48 Vikas Dubey ***@***.***> wrote: >...
Did you check the files are actually there ? If yes post a ls of them and then compare with the pattern used to find them in embedding reader Then...
Please put a ls in the code to check what is in tmp0a9ynoy_ Then please read that list file function code and see why it can't find it On Thu,...
Ok so next try and run that function independently and try to see what is wrong with it On Thu, Mar 21, 2024, 5:29 PM Loreto Parisi ***@***.***> wrote: >...