jobflow icon indicating copy to clipboard operation
jobflow copied to clipboard

Slow resolve reference for large output DB

Open gpetretto opened this issue 8 months ago • 2 comments

I have realized that, when the size of the output DB increases, resolving the references becomes a bottleneck for the execution of the jobs. I have a DB with ~8000 jobs outputs in atomate2 and resolving the references for the store_inputs job for an elastic flow was taking hours. Introducing a mongodb index on the output collection with {uuid: 1, index: -1} led to a huge speedup. Admittedly, I am not working with a very powerful DB, but I expect that this kind of problem would affect even more powerful machines as the DB size grows bigger.

I am opening this issue to check if I was the only one experiencing this kind of problem and to know if there is a set of suggested indexes to be added to the output DB. Maybe there is margin for some optimization in the code? Or at least it could be good to perform some analysis of the most common queris and add a list of suggested indexes to the documentation.

gpetretto avatar Jun 12 '24 09:06 gpetretto