integration of stable diffusion and some other interesting machine learning tools

[lamikr]

trafficstars

At the moment the rocm sdk builder stack is providing the base for integrating and using the many nice ML projects but does not in itself include them.

Some of the projects like openai whisper are easy to take in use with pip install but I am thinking that there could perhaps be room for adding a second layer of install of apps directly. They would not be installed automatically but we could still have binfo files to build them after the core has been build..

At the moment I am thinking some language tools, audio tools and visual tools that could be easily usable. (whisper, automatic1111, comfyui, sd.cpp-webui, shark for example ?)

Not sure whether this second layer would also include only the libraries itself or also the attempts to trying out to improve the default models that these projects provide.

[mritunjaymusale]

I would like to add that when working with stable-diffusion whenever you install extentions it causes the packages to update and thus breaking stuff on rocm's python side.

[mritunjaymusale]

Continuation of using stable diffusion:

    from transformers import (
  File "<frozen importlib._bootstrap>", line 1229, in _handle_fromlist
  File "/opt/rocm_sdk_612/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1077, in __getattr__
    value = getattr(module, name)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rocm_sdk_612/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1076, in __getattr__
    module = self._get_module(self._class_to_module[name])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rocm_sdk_612/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1088, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.models.clip.modeling_clip because of the following error (look up to see its traceback):
No module named 'triton.ops'
➜  stable-diffusion-webui git:(master) 

I got this error after I tried to use a clean version of stable-diffusion using this guide.

I did a clean install of the rocm_sdk including what I mentioned here to avoid the aotriton error

My guess is the version of clip library that stable-diff repo wants to use is replacing triton with clip's preferred version

[lamikr]

@mritunjaymusale Sorry, I have not had myself much time yet to work with the stable diffusion. @daniandtheweb Do you have any idea?

[daniandtheweb]

If you're using this stable diffusion https://github.com/AUTOMATIC1111/stable-diffusion-webui, it currently only supports Python 3.10 so it could be related to that. I've read that Python 3.11 works fine in the webui itself but the extensions still need to be updated by their developers so I think the only thing to do is to wait for the webui to officially support Python 3.11, meanwhile you can just try other webuis like SD.Next or ComfyUI (AUTOMATIC1111 development is quite slower than those other two so It could be a while before we see an updated Python version).

[lamikr]

I tested the comfyui, by creating a venv from rocm sdk python and then by installing the whl files that rocm sdk builder builded and then finally the comfyui itself. I only downloaded and tested with one model file but it worked ok.

I noticed that you have also made own UI: https://github.com/daniandtheweb/sd.cpp-webui Do you think it would be possible to get a binfo file that one?

[lamikr]

• we can now build and launch following extra tools with the latest rocm_sdk_builder. What is missing is the better documentation for this so that people know it. Not sure whether it should be in main readme.md or in some extra page? All tools have also simple one-command launch-script that will work for starting their server on rocm sdk environment easily.

./babs.sh -b binfo/extra/stable-diffusion-webui.binfo
./babs.sh -b binfo/extra/vllm.binfo
./babs.sh -b binfo/extra/llama_cpp.binfo

[daniandtheweb]

I tested the comfyui, by creating a venv from rocm sdk python and then by installing the whl files that rocm sdk builder builded and then finally the comfyui itself. I only downloaded and tested with one model file but it worked ok.

I noticed that you have also made own UI: https://github.com/daniandtheweb/sd.cpp-webui Do you think it would be possible to get a binfo file that one?

I've just read the message, sure, it could be integrated quite easily. It's basically a UI that creates the required command for stable-diffusion.cpp to use (it integrates other functionality such as a gallery and saved preferences). The binfo file should have to clone my repo, clone stable-diffusion.cpp, build it and finally copying the saved result in sd.cpp-webui's main folder. My ui already includes a launch script that creates the venv so I think that would be all.

[daniandtheweb]

• we can now build and launch following extra tools with the latest rocm_sdk_builder. What is missing is the better documentation for this so that people know it. Not sure whether it should be in main readme.md or in some extra page? All tools have also simple one-command launch-script that will work for starting their server on rocm sdk environment easily.

./babs.sh -b binfo/extra/stable-diffusion-webui.binfo
./babs.sh -b binfo/extra/vllm.binfo
./babs.sh -b binfo/extra/llama_cpp.binfo

If this doesn't take much space it should be implemented in the main readme.md page. The instructions could tell about an hypotetical command that lists all the available extra tools and the general command to install them, this way the readme could stay clean but the functionality would stay still clear and easy to use.

[lamikr]

I have now done pretty extensive update to README.md, spend 2 days for reorganizing and rewriting it. Let me know if you think that something is still missing. If you have interest, try to submit a patch to add sd.cpp-webinfo.bnfo file. I can try to help, if you need some help.