autofaiss
autofaiss copied to clipboard
use embedding reader start:end feature to get a proper sampled training set
same for the evaluation set
currently we use the first N vectors for both training and evaluation which is not ideal, especially if the embedding set is not randomly shuffled