tstadel

Just a quick status from my side. I'm still investigating: my goal is to have a clear view on what this means for: - Training - Eval - Inference in...

Ok, so here's the result of my analysis, most of it confirms @MichelBartels. However, besides `use_no_answer_legacy_confidence` the eval and inference results are also affected by `use_confidence_scores`/`use_confidence_scores_for_ranking`. Both default values have...

I ran eval_on_file with `use_confidence_scores=False` on #2856 branch like this: ```python from haystack.nodes import FARMReader reader = FARMReader( model_name_or_path='deepset/roberta-base-squad2', use_gpu=True, max_seq_len=386, use_confidence_scores=False ) metrics = reader.eval_on_file( data_dir='/home/tstad/Downloads', test_filename='dev-v2.0.json', ) ```...

Yes definitely! We have to mention that setting `use_confidence_scores` affects ranking of no_answers. Here's what I'd suggest: - complete https://github.com/deepset-ai/haystack/pull/2856 with adding a new param `use_confidence_scores_for_ranking` to `eval()` and `eval_on_file()`....

@sjrl Adding `use_no_answer_legacy_confidence` to the constructor would introduce more confusion in my opinion. Even if we correctly pass it down to the prediction head it won't have any effect for...

@sjrl are we good with this plan? Can we close this issue?

To use docker inside CI jobs, check out https://github.com/deepset-ai/haystack-hub-api/blob/master/.github/workflows/integration-and-deploy.yml

Extended stacktrace: ``` >>> Pipeline.load_from_yaml('/home/tstad/git/haystack/test/samples/pipeline/bad-pipeline.yaml') >>> Pipeline.load_from_yaml('/home/tstad/git/haystack/test/samples/pipeline/bad-pipeline.yaml') INFO - haystack.pipelines.config - Missing definition for node of type BM25Retriever. Looking into local classes... INFO - haystack.nodes._json_schema - Creating schema for 'BM25Retriever'...

Maybe extending the param types with an anyOf like in https://stackoverflow.com/questions/22565005/json-schema-validate-a-number-or-null-value could help.

@Schokomensch feel free to request a review from me and @TuanaCelik on your PR.