xlearn icon indicating copy to clipboard operation
xlearn copied to clipboard

xlearn python API's .predict method in doesn't kill the created threads after execution in python API, which leads to resource exhausted.

Open HovhannesManushyan opened this issue 2 years ago • 2 comments

I was getting strange resource exhausted bug when running xlearn fm model predict method for a while.

When I profiled the processes via htop, I have noticed that the number of threads gradually increases by 8 when invoking model.predict("model/model.out", f"output/output.txt") which leads to resource exhausted when the number of threads reaches a critical level.

One solution, I found to solve this problem is invoke the model.predict in a separate process via the multiprocessing module, however this solution is extremely slow in cases when model.predict needs to be invoked many times.

Is there a way to kill the created threads after the execution of the predict method has completed?

HovhannesManushyan avatar Aug 24 '21 06:08 HovhannesManushyan

This problem could be solved by building the command line xlearn and then executing the binary using Python's subprocess module.

HovhannesManushyan avatar Sep 10 '21 12:09 HovhannesManushyan

we have the same problem!

litchi6666 avatar Oct 21 '21 06:10 litchi6666