Amount of effort required to make it work in Windows using DirectML

[FatCache]

Turns out ROC.M is not supported by default in Windows. It requires Ubuntu. It appears DirectML is a bridge to use Torch in Windows and offer ROC.M support.

https://learn.microsoft.com/en-us/windows/ai/directml/gpu-pytorch-windows

I don't have any knowledge of how Machine Learning model work or implemented but can work my way around using Python.

How much effort is it to get it to work in Windows using DirectML ?

From the example, do we just need to import import torch_directml and throughout the code, put in the attribute .to(torch_directml.device())

I can see inside text_generation.py, tensor is used quiet a bit there.

[oobabooga]

I have zero knowledge about DirectML, but maybe someone can help get this implemented.

[Spencer-Dawson]

So if I'm understanding various blog posts correctly(and assuming those posts are still accurate)...

If you use CUDA in WSL2 it uses a version of the CUDA driver in WSL that then uses a DirectML vitual device to represent your Windows device driver via some kind of DirectX12 compatibility layer. This requires specific driver support from the Windows GPU driver which Nvidia provides, but AMD, and Intel do not.

But CUDA isn't the only option for pytorch in WSL. There is DirectML pytorch (mentioned above in FatCache's post) which will abstract to all directX12 capable GPUs including Nvidia, AMD, and Intel. I get the impression Intel uses OneAPI for this in their driver, but I haven't a clue about the others.

One obvious issue with swapping using CUDA devices for pytorch with DirectML is that it would presumably break non Linux compatibility. This could be avoided by supporting both and choosing which one to load at runtime through some kind of interface.

[Ph0rk0z]

If by some miracle shark and mrq voice cloning work using directml I will give it a try. Especially if it's just a matter of replacing torch import with torch_directml.

[Spencer-Dawson]

Someone in another bug thread mentioned a directml port of stable-diffusion-ui(I'd link, but honestly I lost the thread). I haven't tested it, but I diffed it against the original to get an idea of what changes would be needed. Looking at the diff(click on files changed) it doesn't appear to be a simple drop in tweak of just changing a few torch commands, but I would imagine it would be doable for someone used to troubleshooting low level model stuff anyway.

https://github.com/AUTOMATIC1111/stable-diffusion-webui/compare/master...lshqqytiger:stable-diffusion-webui-directml:master

[Ph0rk0z]

Yea.. I'm going to use this and MRQ as a blueprint. Shark dumped out some stuff on windows with my AMD but it's using vulkan. If ai voice cloning works it should be doable... wish bitsandbytes rocm would work on windows tho. Can't do much with 8gb.

[FatCache]

I should have sent out an update. I did put effort to see how far I can get it to work in DirectML with no knowledge how tensor works for any of this. After back and forth, I got stuck here and threw in the towel.

https://github.com/oobabooga/text-generation-webui/blob/026d60bd3424b5426c5ef80632aa6b71fe12d4c5/modules/models.py#L49

I have no idea how can AutoModelForCausalLM be ported to work in DirectML. This is beyond the stage of leveraging instance of torch and mapping it do DirectML device.

Only if AutoModelForCausalLM had an attribute to map to a directML device.

[github-actions[bot]]

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.