Sebastian Husch Lee

@tstadel Nice work! Thanks for the thorough investigation. > My suggestion would be to fix the passing of FARMReader's `use_confidence_scores` down to the Evaluator and with setting that to False...

Whatever defaults we decide on, @julian-risch mentioned that we should include docstrings and docs on the website explaining when to set `use_confidence_scores` either `True` or `False`, which I think sounds...

Hi @tstadel I definitely agree with your second and third points. However, I'm a little uncertain about the first suggestion about adding `use_confidence_scores_for_ranking` to `eval()` and `eval_on_file()`. It seems unintuitive...

After some discussion with @Timoeller, @ju-gu, @mathislucka and @tstadel we agreed that we would like the `FARMReader.eval` and `FARMReader.eval_on_file` functions to behave the same way as the `FARMReader.predict` functions. So...

This has been resolved with PR #2856. To be clear reproducing the metrics reported on HF only works if `use_confidence_scores` is set to `False` when initializing a `FARMReader` in Haystack.

Hi @anakin87, thanks again for opening a PR for this issue! Is it safe to close this issue now?

Hey, @anakin87 I see what you are saying. It sounds like there is still more to be done on this topic in the future. I think the best course of...

@tstadel This PR is ready for review.

Hey, @yli223 we do not currently support the M1 GPU. We would need to implement the changes explained by @mathislucka above in Haystack. In addition, we need also need to...

That's great! I would say that anywhere the user passes an option to `initialize_device_settings` should have the option of passing a list of `devices` instead. Similar to what is already...