os.environ['CUDA_VISIBLE_DEVICES'] usage error

[WarpedAnimation]

In your initalization.py module, you have the following code:

if args.gpu_device_id is not None:
    os.environ['CUDA_VISIBLE_DEVICES'] = str(args.gpu_device_id)
    print("Set device to:", args.gpu_device_id)

In Windows, this usage is NOT doing what you apparently think it is. A common misconception is that setting this variable to the desired device id will make only THAT device available.

In fact, it DOES NOT do that. For instance, if I have 2 GPUs, 0 and 1, and I want to use GPU ID 1, your code will set os.environ['CUDA_VISIBLE_DEVICES'] = "1". What this does is set the number of devices available to 1, and makes the only device available GPU 0. This is obviously NOT what is desired.

The fix (for cuda devices) is to either NOT set the os.environ['CUDA_VISIBLE_DEVICES'] variable at all, or special case it the devices are "cuda", and instead, change your code in memory_management.py to:

def get_torch_device(): global directml_enabled global cpu_state if directml_enabled: global directml_device return directml_device if cpu_state == CPUState.MPS: return torch.device("mps") if cpu_state == CPUState.CPU: return torch.device("cpu") else: if is_intel_xpu(): return torch.device("xpu", torch.xpu.current_device()) else: if not args.gpu_device_id is None: return torch.device(int(args.gpu_device_id)) else: return torch.device(torch.cuda.current_device())

[ZedOud]

This might be useful.

CUDA_VISIBLE_DEVICES is used to specify which GPU’s should be visible to a CUDA application. Only the devices whose index or UUID is present in the sequence are visible to CUDA applications and they are enumerated in the order of the sequence.

When CUDA_VISIBLE_DEVICES is set before launching the control daemon, the devices will be remapped by the MPS server. This means that if your system has devices 0, 1 and 2, and if CUDA_VISIBLE_DEVICES is set to “0,2”, then when a client connects to the server it will see the remapped devices - device 0 and a device 1. Therefore, keeping CUDA_VISIBLE_DEVICES set to “0,2” when launching the client would lead to an error.

I'm also curious as to the differences with how --device-id and --gpu-device-id works.

I think what you are proposing is maybe a more correct and reliable and clear solution?

[WarpedAnimation]

This might be useful.

CUDA_VISIBLE_DEVICES is used to specify which GPU’s should be visible to a CUDA application. Only the devices whose index or UUID is present in the sequence are visible to CUDA applications and they are enumerated in the order of the sequence. When CUDA_VISIBLE_DEVICES is set before launching the control daemon, the devices will be remapped by the MPS server. This means that if your system has devices 0, 1 and 2, and if CUDA_VISIBLE_DEVICES is set to “0,2”, then when a client connects to the server it will see the remapped devices - device 0 and a device 1. Therefore, keeping CUDA_VISIBLE_DEVICES set to “0,2” when launching the client would lead to an error.

I'm also curious as to the differences with how --device-id and --gpu-device-id works.

I think what you are proposing is maybe a more correct and reliable and clear solution?

As far as --device-id and --gpu-device-id go, I had to slightly modify the code on my machine to get it to work correctly on Windows. I had to change the arg name to be --gpu_device_id because it didn't like the '-' in the name(s). As far as I know, --gpu-device-id is introduced by the webui forge code, whereas the --device-id arg is from the original stable diffusion code. Basically, --device-id is no longer important with webui forge. I should note that I routinely run webui forge on my gpu 1, and comfyui on my gpu 0, so I know this stuff works with a few code changes and the change that I suggested in my original problem report above.