Select cuda device via API.

[Ph0rk0z]

There are extensions using this for text generation, as you know. It would be very nice to be able to select which GPU to use via the API. Right now you are only able to choose GPU or CPU and there is no way to do CUDA_VISIBLE_DEVICES since it does not run stand-alone.

I have 2 GPU and would like to use one for TTS and one for the LLM without having to patch the code manually.

[matheus786]

3f3f3f3

[doevent]

It worked for me:

import torch
torch.cuda.set_device(device_id)

[gkucsko]

thanks yeah putting it on enhancements list for next version