memory overflow in bm25 retrieval loop when constructing dataset

[yjqangiai]

When I attempted to construct the training data, I encountered a memory overflow (128GB) when performing bm25 retrieval iteratively on CPU. The memory is rapidly increased in the loop proccess. So I would like to ask how to solve it. Thanks.