Bug when computing relavant_docs (qrels)

Open tonywu71 opened this issue 1 year ago • 0 comments

Description

At the following line, the relavant_docs dictionary is overridden instead of getting updated. This causes relavant_docs to only contain the last seen query. This is problematic in datasets with duplicate queries (e.g. DocVQA) as the output score is not correctly computed.

https://github.com/illuin-tech/vidore-benchmark/blob/b8609716e6b5240e8100c8a861c381c0c738c4de/src/vidore_benchmark/retrievers/vision_retriever.py#L122

Nov 11 '24 17:11 tonywu71