DocumentJoiner receives unfiltered documents when using LLM-generated filters

[bglearning]

Note: This issue seems to be a pipeline.run bug and is fixed by the subgraphs branch.

In a pipeline with dynamic llm-generated filters (based on query, similar to this blog-post) i.e. basically query -> prompt-builder -> llm -> filter-parser -> retriever -> document_joiner the retriever generates filtered documents but the document_joiner ignores the generated filter.

Example Colab.

For Fix

It doesn't occur in the subgraphs branch. (Colab)