big-ann-benchmarks icon indicating copy to clipboard operation
big-ann-benchmarks copied to clipboard

Fix issue with dataset filenames

Open magdalendobson opened this issue 9 months ago • 0 comments

The generic function get_dataset_fn() implemented in the DatasetCompetitionFormat class in benchmark/datasets.py did not work properly, as it contained some code that was specific to the billion-size datasets, and caused some runs to fail as they searched for the wrong dataset name. I have amended that class to use a more general function, and I implemented a subclass BillionScaleDatasetCompetitionFormat that took care of the filenames for the billion-size datasets. Now the get_dataset_fn() method should work properly for every dataset.

Requesting @harsha-simhadri or @maumueller review.

magdalendobson avatar May 16 '24 20:05 magdalendobson