ctransformers icon indicating copy to clipboard operation
ctransformers copied to clipboard
verified publisher icon marella

Multiple GPU support

Open physaikat opened this issue 2 years ago • 0 comments

I have 3 GPTQ models to consume and I have 4 GPUs available. How can I mention which model to load in which GPU? If I do not mention it is trying to load all models in cuda:0 and it is crashing.

physaikat avatar Aug 21 '23 04:08 physaikat