ctransformers icon indicating copy to clipboard operation
ctransformers copied to clipboard

Out of memory exits process

Open kczimm opened this issue 2 years ago • 0 comments

Thanks for creating this library! We are using ctransformers at PostgresML to support GGUF models from Huggingface. We need to detect and recover from CUDA out of memory errors. Currently, it appears that ctransformers exits the process whenever a CUDA error is encountered.

Is there any plan to add error handling to ctransformers? Ideally, we would get some sort of Exception in Python that we can handle.

kczimm avatar Nov 01 '23 14:11 kczimm