Sean MacAvaney

Any chance you could reconsider and release the pre-trained models?

Awesome- thanks! The original TK and TKL would be great.

Should be being built right here: https://github.com/Georgetown-IR-Lab/OpenNIR/blob/master/onir/datasets/msmarco.py#L237 Nothing obvious looks wrong. It's especially strange because the query file was built properly, and it's doing essentially the same thing: https://github.com/Georgetown-IR-Lab/OpenNIR/blob/master/onir/datasets/msmarco.py#L210

From correspondence with @aminbigdeli -- it seems the `minidev.qrels` file was empty. I need to look into why this happened + correct the bug.

Hi Stéphane, Unfortunately the `deterministic` indicator only corresponds to `torch.backends.cudnn.deterministic` flag-- which doesn't actually control for differences across specific GPUs or CUDA versions. Anecdotally, I've seen that different GPUs can...

Our setup for running the experiment was: - OS: Ubuntu 18.04 - GPU: GeForce GTX 1080 Ti - NVIDIA Driver Version: 418.67 - CUDA Version: 10.1 So there are differences...

It should be the initial commit: [ca14dfa5e7...](https://github.com/Georgetown-IR-Lab/OpenNIR/tree/ca14dfa5e7cfef3fbbb35efbb4e7df0f1fbde590) Note that you'll need to clear the `~/data/onir` directory (or rename it), otherwise it will use the indices built from the newer version.

Hmmm, fascinating! Thanks for running these tests. The BM25 discrepancies are puzzling, as well as the performance differences between Vanilla BERT and CEDR-KNRM. I'm out of ideas about what could...

Sounds good- thanks! Just to be clear-- `matchzoo-py==1.0` worked, correct?

I think rather than adding it to the requirements.txt (which would mean it's required for all installations of OpenNIR), I'll add a check in `matchzoo.py` to see that it's a...