xlearn
xlearn copied to clipboard
xlearn python API's .predict method in doesn't kill the created threads after execution in python API, which leads to resource exhausted.
I was getting strange resource exhausted bug when running xlearn fm model predict method for a while.
When I profiled the processes via htop, I have noticed that the number of threads gradually increases by 8 when invoking model.predict("model/model.out", f"output/output.txt")
which leads to resource exhausted when the number of threads reaches a critical level.
One solution, I found to solve this problem is invoke the model.predict in a separate process via the multiprocessing module, however this solution is extremely slow in cases when model.predict needs to be invoked many times.
Is there a way to kill the created threads after the execution of the predict method has completed?
This problem could be solved by building the command line xlearn and then executing the binary using Python's subprocess module.
we have the same problem!