3rd Retriever in the pipeline is not shown in evaluation metrics

[Koenlaermans]

Describe the bug When you add a 3rd retriever to the pipeline as input to the JoinDocuments node, only the first 2 are shown in evaluation.

eval_pipe = Pipeline()
  eval_pipe.add_node(component=retriever, name="Retriever", inputs=["Query"])
  eval_pipe.add_node(component=retriever2, name="Retriever2", inputs=["Query"])
  eval_pipe.add_node(component=retriever3, name="Retriever3", inputs=["Query"])
  eval_pipe.add_node(component=JoinDocuments(join_mode="reciprocal_rank_fusion"),     name="JoinResults", inputs=["Retriever", "Retriever2", "Retriever3"])

I found this after evaluating this and using the function calculate_metrics: only these properties are available.

Expected behavior I Expect all 3 retrievers to be added in the evaluation metrics

System: Using Google Colab T4 GPU

• Haystack version (commit or version number): latest
• DocumentStore: elastic search
• Retriever: EmbeddingRetriever (3)

[julian-risch]

@tstadel Have you come across this problem before? Any idea why the evaluation results of the last retriever are not included when JoinDocuments is used?

[tstadel]

No, I haven't seen this so far. But afaik I haven't seen the usage of three retrievers either. I would suspect the problem somewhere here https://github.com/deepset-ai/haystack/blob/179e9cea08f00f25c785e371c4ca1b092837bb4c/haystack/pipelines/base.py#L580-L600

Looks like the else block is not merging the _debug dict which we use for eval.

[masci]

This looks like a corner case, closing as won't fix