InvokeAI [WIP] ONNX Support

Note: this branch based on #3548, not on main

While find out what needs to be done to implement onnx, found that I can do draft of it pretty quickly, so... here it is) Supports LoRA and TI. As example - cat with sadcatmeme lora:

Jun 20 '23 23:06 StAlKeR7779

Hi @StAlKeR7779 can you provide some environment details where you ran the basic version and with versions of the packages being used to test this. It would be helpful for testing. Just can you just post the message related to command: "pip freeze" for the above details.

Jun 23 '23 12:06 saikrishna2893

Hi @StAlKeR7779 can you provide some environment details where you ran the basic version and with versions of the packages being used to test this. It would be helpful for testing. Just can you just post the message related to command: "pip freeze" for the above details.

I run it in ubuntu 22.04 wsl on windows 10. My venv currently really messy and don't think that it could provide helpful information, but it should run with default install + onnx and onnxruntime packages. But... here my venv packages:

absl-py==1.4.0
accelerate==0.19.0
addict==2.4.0
aiohttp==3.8.4
aiosignal==1.3.1
albumentations==1.3.0
antlr4-python3-runtime==4.9.3
anyio==3.6.2
async-timeout==4.0.2
attrs==23.1.0
basicsr==1.4.2
bidict==0.22.1
boltons==23.0.0
cachetools==5.3.0
certifi==2022.12.7
charset-normalizer==3.1.0
click==8.1.3
clip-anytorch==2.5.2
cmake==3.26.3
coloredlogs==15.0.1
compel==1.2.1
contourpy==1.0.7
controlnet-aux==0.0.5
cycler==0.11.0
datasets==2.12.0
diffusers==0.17.1
dill==0.3.6
dnspython==2.2.1
dynamicprompts==0.27.0
easing-functions==1.0.4
einops==0.6.1
eventlet==0.33.3
facexlib==0.3.0
fastapi==0.88.0
fastapi-events==0.8.0
fastapi-socketio==0.0.10
filelock==3.12.0
filterpy==1.4.5
Flask==2.1.3
Flask-Cors==3.0.10
Flask-SocketIO==5.3.0
flaskwebgui==1.0.3
flatbuffers==23.5.26
fonttools==4.39.3
frozenlist==1.3.3
fsspec==2023.4.0
ftfy==6.1.1
future==0.18.3
gfpgan==1.3.8
google-auth==2.17.3
google-auth-oauthlib==1.0.0
greenlet==2.0.2
grpcio==1.54.0
h11==0.14.0
httptools==0.5.0
huggingface-hub==0.14.1
humanfriendly==10.0
idna==3.4
imageio==2.28.1
importlib-metadata==6.6.0
-e git+https://github.com/invoke-ai/InvokeAI@54b74427f4376716c3b30d5f4ee3aa3fd0c2daa4#egg=InvokeAI
itsdangerous==2.1.2
Jinja2==3.1.2
joblib==1.2.0
kiwisolver==1.4.4
lazy_loader==0.2
lit==16.0.3
llvmlite==0.40.0
lmdb==1.4.1
Markdown==3.4.3
markdown-it-py==2.2.0
MarkupSafe==2.1.2
matplotlib==3.7.1
mdurl==0.1.2
mpmath==1.3.0
multidict==6.0.4
multiprocess==0.70.14
mypy-extensions==1.0.0
networkx==3.1
npyscreen==4.10.5
numba==0.57.0
numpy==1.23.5
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==8.9.1.23
oauthlib==3.2.2
omegaconf==2.3.0
onnx==1.14.0
onnxruntime==1.15.1
onnxruntime-gpu==1.15.1
onnxruntime-openvino @ file:///mnt/f/tmp/onnxruntime_openvino-1.14.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
opencv-python==4.7.0.72
opencv-python-headless==4.7.0.72
openvino==2022.3.0
packaging==23.1
pandas==2.0.1
picklescan==0.0.9
Pillow==9.5.0
polygraphy==0.47.1
prompt-toolkit==3.0.38
protobuf==4.22.4
psutil==5.9.4
pyarrow==12.0.0
pyasn1==0.5.0
pyasn1-modules==0.3.0
pydantic==1.10.7
pyDeprecate==0.3.2
Pygments==2.15.1
pyparsing==3.0.9
PyPatchMatch==1.0.0
pyre-extensions==0.0.29
pyreadline3==3.4.1
python-dateutil==2.8.2
python-dotenv==1.0.0
python-engineio==4.4.1
python-multipart==0.0.6
python-socketio==5.8.0
pytorch-lightning==1.7.7
pytorch-triton==2.1.0+9820899b38
pytz==2023.3
PyWavelets==1.4.1
PyYAML==6.0
qudida==0.0.4
realesrgan==0.3.0
regex==2023.5.5
requests==2.28.2
requests-oauthlib==1.3.1
responses==0.18.0
rich==13.3.5
rsa==4.9
safetensors==0.3.1
scikit-image==0.20.0
scikit-learn==1.2.2
scipy==1.10.1
Send2Trash==1.8.2
sentencepiece==0.1.99
six==1.16.0
sniffio==1.3.0
starlette==0.22.0
sympy==1.11.1
tb-nightly==2.14.0a20230506
tensorboard==2.13.0
tensorboard-data-server==0.7.0
tensorrt==8.6.1
tensorrt-bindings==8.6.1
tensorrt-libs==8.6.1
test-tube==0.7.5
threadpoolctl==3.1.0
tifffile==2023.4.12
timm==0.9.2
tokenizers==0.13.3
tomli==2.0.1
torch==2.1.0.dev20230531+cu118
torchaudio==2.1.0.dev20230531+cu118
torchmetrics==0.11.4
torchsde==0.2.5
torchvision==0.16.0.dev20230531+cu118
tqdm==4.65.0
trampoline==0.1.2
transformers==4.29.2
triton==2.0.0
typing-inspect==0.8.0
typing_extensions==4.5.0
tzdata==2023.3
urllib3==1.26.15
uvicorn==0.21.1
uvloop==0.17.0
watchfiles==0.19.0
wcwidth==0.2.6
websockets==11.0.2
Werkzeug==2.3.3
xformers==0.0.19
xxhash==3.2.0
yapf==0.33.0
yarl==1.9.2
zipp==3.15.0

Jun 24 '23 17:06 StAlKeR7779

Hi @StAlKeR7779 can you provide some environment details where you ran the basic version and with versions of the packages being used to test this. It would be helpful for testing. Just can you just post the message related to command: "pip freeze" for the above details.

I run it in ubuntu 22.04 wsl on windows 10. My venv currently really messy and don't think that it could provide helpful information, but it should run with default install + onnx and onnxruntime packages. But... here my venv packages:

absl-py==1.4.0
accelerate==0.19.0
addict==2.4.0
aiohttp==3.8.4
aiosignal==1.3.1
albumentations==1.3.0
antlr4-python3-runtime==4.9.3
anyio==3.6.2
async-timeout==4.0.2
attrs==23.1.0
basicsr==1.4.2
bidict==0.22.1
boltons==23.0.0
cachetools==5.3.0
certifi==2022.12.7
charset-normalizer==3.1.0
click==8.1.3
clip-anytorch==2.5.2
cmake==3.26.3
coloredlogs==15.0.1
compel==1.2.1
contourpy==1.0.7
controlnet-aux==0.0.5
cycler==0.11.0
datasets==2.12.0
diffusers==0.17.1
dill==0.3.6
dnspython==2.2.1
dynamicprompts==0.27.0
easing-functions==1.0.4
einops==0.6.1
eventlet==0.33.3
facexlib==0.3.0
fastapi==0.88.0
fastapi-events==0.8.0
fastapi-socketio==0.0.10
filelock==3.12.0
filterpy==1.4.5
Flask==2.1.3
Flask-Cors==3.0.10
Flask-SocketIO==5.3.0
flaskwebgui==1.0.3
flatbuffers==23.5.26
fonttools==4.39.3
frozenlist==1.3.3
fsspec==2023.4.0
ftfy==6.1.1
future==0.18.3
gfpgan==1.3.8
google-auth==2.17.3
google-auth-oauthlib==1.0.0
greenlet==2.0.2
grpcio==1.54.0
h11==0.14.0
httptools==0.5.0
huggingface-hub==0.14.1
humanfriendly==10.0
idna==3.4
imageio==2.28.1
importlib-metadata==6.6.0
-e git+https://github.com/invoke-ai/InvokeAI@54b74427f4376716c3b30d5f4ee3aa3fd0c2daa4#egg=InvokeAI
itsdangerous==2.1.2
Jinja2==3.1.2
joblib==1.2.0
kiwisolver==1.4.4
lazy_loader==0.2
lit==16.0.3
llvmlite==0.40.0
lmdb==1.4.1
Markdown==3.4.3
markdown-it-py==2.2.0
MarkupSafe==2.1.2
matplotlib==3.7.1
mdurl==0.1.2
mpmath==1.3.0
multidict==6.0.4
multiprocess==0.70.14
mypy-extensions==1.0.0
networkx==3.1
npyscreen==4.10.5
numba==0.57.0
numpy==1.23.5
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==8.9.1.23
oauthlib==3.2.2
omegaconf==2.3.0
onnx==1.14.0
onnxruntime==1.15.1
onnxruntime-gpu==1.15.1
onnxruntime-openvino @ file:///mnt/f/tmp/onnxruntime_openvino-1.14.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
opencv-python==4.7.0.72
opencv-python-headless==4.7.0.72
openvino==2022.3.0
packaging==23.1
pandas==2.0.1
picklescan==0.0.9
Pillow==9.5.0
polygraphy==0.47.1
prompt-toolkit==3.0.38
protobuf==4.22.4
psutil==5.9.4
pyarrow==12.0.0
pyasn1==0.5.0
pyasn1-modules==0.3.0
pydantic==1.10.7
pyDeprecate==0.3.2
Pygments==2.15.1
pyparsing==3.0.9
PyPatchMatch==1.0.0
pyre-extensions==0.0.29
pyreadline3==3.4.1
python-dateutil==2.8.2
python-dotenv==1.0.0
python-engineio==4.4.1
python-multipart==0.0.6
python-socketio==5.8.0
pytorch-lightning==1.7.7
pytorch-triton==2.1.0+9820899b38
pytz==2023.3
PyWavelets==1.4.1
PyYAML==6.0
qudida==0.0.4
realesrgan==0.3.0
regex==2023.5.5
requests==2.28.2
requests-oauthlib==1.3.1
responses==0.18.0
rich==13.3.5
rsa==4.9
safetensors==0.3.1
scikit-image==0.20.0
scikit-learn==1.2.2
scipy==1.10.1
Send2Trash==1.8.2
sentencepiece==0.1.99
six==1.16.0
sniffio==1.3.0
starlette==0.22.0
sympy==1.11.1
tb-nightly==2.14.0a20230506
tensorboard==2.13.0
tensorboard-data-server==0.7.0
tensorrt==8.6.1
tensorrt-bindings==8.6.1
tensorrt-libs==8.6.1
test-tube==0.7.5
threadpoolctl==3.1.0
tifffile==2023.4.12
timm==0.9.2
tokenizers==0.13.3
tomli==2.0.1
torch==2.1.0.dev20230531+cu118
torchaudio==2.1.0.dev20230531+cu118
torchmetrics==0.11.4
torchsde==0.2.5
torchvision==0.16.0.dev20230531+cu118
tqdm==4.65.0
trampoline==0.1.2
transformers==4.29.2
triton==2.0.0
typing-inspect==0.8.0
typing_extensions==4.5.0
tzdata==2023.3
urllib3==1.26.15
uvicorn==0.21.1
uvloop==0.17.0
watchfiles==0.19.0
wcwidth==0.2.6
websockets==11.0.2
Werkzeug==2.3.3
xformers==0.0.19
xxhash==3.2.0
yapf==0.33.0
yarl==1.9.2
zipp==3.15.0

Sure thanks for the information @StAlKeR7779. We are trying to run in Windows with python v3.9, facing execution errors with respect to cross_attention modules. When trying to run command "invokeai --web"

Jun 26 '23 06:06 saikrishna2893

@saikrishna2893 It's because you running old invoke web, it's not supposed to running. We left it here for now, to be able check old sources. I don't know how to run with builded web files, but I running currently as: 1 console:

cd invokeai
source venv/bin/activate
python scripts/invoke-web.py

2 console:

cd invokeai/frontend/web/
yarn install
yarn dev

you also need before to initialize venv for 1 console:

cd invokeai
python -m venv venv
source venv/bin/activate
pip install -e .[xformers] --use-pep517 --extra-index-url https://download.pytorch.org/whl/cu117

Jun 26 '23 23:06 StAlKeR7779

@saikrishna2893 It's because you running old invoke web, it's not supposed to running. We left it here for now, to be able check old sources. I don't know how to run with builded web files, but I running currently as: 1 console:
cd invokeai
source venv/bin/activate
python scripts/invoke-web.py
2 console:
cd invokeai/frontend/web/
yarn install
yarn dev
you also need before to initialize venv for 1 console:
cd invokeai
python -m venv venv
source venv/bin/activate
pip install -e .[xformers] --use-pep517 --extra-index-url https://download.pytorch.org/whl/cu117

Sure thanks for the information. Will test accordingly.

Jun 27 '23 05:06 saikrishna2893

@saikrishna2893

You probably already know but if it helps, once I had main installed right, I run both commands in one console window (for ease of use) to get the webui to load using:

source /venv/bin/activate
cd /InvokeAI/invokeai/frontend/web/
yarn install
yarn dev & ( cd /InvokeAI/ && invokeai-web )

Jun 27 '23 08:06 mickr777

@saikrishna2893 It's because you running old invoke web, it's not supposed to running. We left it here for now, to be able check old sources. I don't know how to run with builded web files, but I running currently as: 1 console:
cd invokeai
source venv/bin/activate
python scripts/invoke-web.py
2 console:
cd invokeai/frontend/web/
yarn install
yarn dev
you also need before to initialize venv for 1 console:
cd invokeai
python -m venv venv
source venv/bin/activate
pip install -e .[xformers] --use-pep517 --extra-index-url https://download.pytorch.org/whl/cu117

Hi @StAlKeR7779 sorry to bother, with python 3.9 and by executing above mentioned steps, facing runtime errors with inspect module with error as "module 'inspect' has no attribute 'get_annotations'". When checked for the error, found that get_annotations is supported in python 3.10 and more. Just want to enquire if you are also using python3.9 or 3.10 version. For Openvino and onnxruntime-openvino module, actually 3.9 is required. So just want to enquire on this.

Jun 30 '23 04:06 saikrishna2893

(.test_Invoke) sysuser@sysuser-5995X:~/Documents/Lalith/InvokeAI$ python scripts/invokeai-model-install.py 
[2023-07-26 04:11:56,775]::[InvokeAI]::INFO --> scanning /home/sysuser/invokeai/models for new models
An exception has occurred: unsupported operand type(s) for /: 'PosixPath' and 'NoneType' Details:
Traceback (most recent call last):
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/frontend/install/model_install.py", line 791, in main
    select_and_download_models(opt)
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/frontend/install/model_install.py", line 677, in select_and_download_models
    installer = ModelInstall(config, prediction_type_helper=helper)
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/install/model_install_backend.py", line 94, in __init__
    self.mgr = model_manager or ModelManager(config.model_conf_path)
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 338, in __init__
    self._read_models(config)
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 361, in _read_models
    self.scan_models_directory()
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 908, in scan_models_directory
    imported_models = self.autoimport()
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 948, in autoimport
    directories = {config.root_path / x for x in [config.autoimport_dir,
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 948, in <setcomp>
    directories = {config.root_path / x for x in [config.autoimport_dir,
TypeError: unsupported operand type(s) for /: 'PosixPath' and 'NoneType'

Press any key to continue...    
(.test_Invoke) sysuser@sysuser-5995X:~/Documents/Lalith/InvokeAI$ python scripts/invokeai-web.py 
/home/sysuser/Documents/Lalith/InvokeAI/.test_Invoke/lib/python3.10/site-packages/torchvision/transforms/functional_tensor.py:5: UserWarning: The torchvision.transforms.functional_tensor module is deprecated in 0.15 and will be **removed in 0.17**. Please don't rely on it. You probably just need to use APIs in torchvision.transforms.functional or in torchvision.transforms.v2.functional.
  warnings.warn(
>> patchmatch.patch_match: INFO - Compiling and loading c extensions from "/home/sysuser/Documents/Lalith/InvokeAI/.test_Invoke/lib/python3.10/site-packages/patchmatch".
>> patchmatch.patch_match: ERROR - patchmatch failed to load or compile (Command 'make clean && make' returned non-zero exit status 2.).
>> patchmatch.patch_match: INFO - Refer to https://invoke-ai.github.io/InvokeAI/installation/060_INSTALL_PATCHMATCH/ for installation instructions.
[2023-07-26 04:12:14,236]::[InvokeAI]::INFO --> Patchmatch not loaded (nonfatal)
INFO:     Started server process [44116]
INFO:     Waiting for application startup.
[2023-07-26 04:12:14,646]::[InvokeAI]::INFO --> scanning /home/sysuser/invokeai/models for new models
ERROR:    Traceback (most recent call last):
  File "/home/sysuser/Documents/Lalith/InvokeAI/.test_Invoke/lib/python3.10/site-packages/starlette/routing.py", line 671, in lifespan
    async with self.lifespan_context(app):
  File "/home/sysuser/Documents/Lalith/InvokeAI/.test_Invoke/lib/python3.10/site-packages/starlette/routing.py", line 566, in __aenter__
    await self._router.startup()
  File "/home/sysuser/Documents/Lalith/InvokeAI/.test_Invoke/lib/python3.10/site-packages/starlette/routing.py", line 648, in startup
    await handler()
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/app/api_app.py", line 78, in startup_event
    ApiDependencies.initialize(
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/app/api/dependencies.py", line 120, in initialize
    model_manager=ModelManagerService(config, logger),
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/app/services/model_manager_service.py", line 326, in __init__
    self.mgr = ModelManager(
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 338, in __init__
    self._read_models(config)
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 361, in _read_models
    self.scan_models_directory()
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 908, in scan_models_directory
    imported_models = self.autoimport()
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 948, in autoimport
    directories = {config.root_path / x for x in [config.autoimport_dir,
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 948, in <setcomp>
    directories = {config.root_path / x for x in [config.autoimport_dir,
TypeError: unsupported operand type(s) for /: 'PosixPath' and 'NoneType'

ERROR:    Application startup failed. Exiting.
Task was destroyed but it is pending!
task: <Task pending name='Task-3' coro=<FastAPIEventService.__dispatch_from_queue() running at /home/sysuser/Documents/Lalith/InvokeAI/invokeai/app/api/events.py:48> wait_for=<Future pending cb=[Task.task_wakeup()]>>

Was able to invoke the main branch but the same is failing with this branch Not able to host with scripts/invokeai-web.py @StAlKeR7779

Jul 26 '23 11:07 lalith-mcw

(.test_Invoke) sysuser@sysuser-5995X:~/Documents/Lalith/InvokeAI$ python scripts/invokeai-model-install.py 
[2023-07-26 04:11:56,775]::[InvokeAI]::INFO --> scanning /home/sysuser/invokeai/models for new models
An exception has occurred: unsupported operand type(s) for /: 'PosixPath' and 'NoneType' Details:
Traceback (most recent call last):
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/frontend/install/model_install.py", line 791, in main
    select_and_download_models(opt)
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/frontend/install/model_install.py", line 677, in select_and_download_models
    installer = ModelInstall(config, prediction_type_helper=helper)
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/install/model_install_backend.py", line 94, in __init__
    self.mgr = model_manager or ModelManager(config.model_conf_path)
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 338, in __init__
    self._read_models(config)
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 361, in _read_models
    self.scan_models_directory()
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 908, in scan_models_directory
    imported_models = self.autoimport()
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 948, in autoimport
    directories = {config.root_path / x for x in [config.autoimport_dir,
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 948, in <setcomp>
    directories = {config.root_path / x for x in [config.autoimport_dir,
TypeError: unsupported operand type(s) for /: 'PosixPath' and 'NoneType'

Press any key to continue...    
(.test_Invoke) sysuser@sysuser-5995X:~/Documents/Lalith/InvokeAI$ python scripts/invokeai-web.py 
/home/sysuser/Documents/Lalith/InvokeAI/.test_Invoke/lib/python3.10/site-packages/torchvision/transforms/functional_tensor.py:5: UserWarning: The torchvision.transforms.functional_tensor module is deprecated in 0.15 and will be **removed in 0.17**. Please don't rely on it. You probably just need to use APIs in torchvision.transforms.functional or in torchvision.transforms.v2.functional.
  warnings.warn(
>> patchmatch.patch_match: INFO - Compiling and loading c extensions from "/home/sysuser/Documents/Lalith/InvokeAI/.test_Invoke/lib/python3.10/site-packages/patchmatch".
>> patchmatch.patch_match: ERROR - patchmatch failed to load or compile (Command 'make clean && make' returned non-zero exit status 2.).
>> patchmatch.patch_match: INFO - Refer to https://invoke-ai.github.io/InvokeAI/installation/060_INSTALL_PATCHMATCH/ for installation instructions.
[2023-07-26 04:12:14,236]::[InvokeAI]::INFO --> Patchmatch not loaded (nonfatal)
INFO:     Started server process [44116]
INFO:     Waiting for application startup.
[2023-07-26 04:12:14,646]::[InvokeAI]::INFO --> scanning /home/sysuser/invokeai/models for new models
ERROR:    Traceback (most recent call last):
  File "/home/sysuser/Documents/Lalith/InvokeAI/.test_Invoke/lib/python3.10/site-packages/starlette/routing.py", line 671, in lifespan
    async with self.lifespan_context(app):
  File "/home/sysuser/Documents/Lalith/InvokeAI/.test_Invoke/lib/python3.10/site-packages/starlette/routing.py", line 566, in __aenter__
    await self._router.startup()
  File "/home/sysuser/Documents/Lalith/InvokeAI/.test_Invoke/lib/python3.10/site-packages/starlette/routing.py", line 648, in startup
    await handler()
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/app/api_app.py", line 78, in startup_event
    ApiDependencies.initialize(
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/app/api/dependencies.py", line 120, in initialize
    model_manager=ModelManagerService(config, logger),
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/app/services/model_manager_service.py", line 326, in __init__
    self.mgr = ModelManager(
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 338, in __init__
    self._read_models(config)
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 361, in _read_models
    self.scan_models_directory()
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 908, in scan_models_directory
    imported_models = self.autoimport()
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 948, in autoimport
    directories = {config.root_path / x for x in [config.autoimport_dir,
  File "/home/sysuser/Documents/Lalith/InvokeAI/invokeai/backend/model_management/model_manager.py", line 948, in <setcomp>
    directories = {config.root_path / x for x in [config.autoimport_dir,
TypeError: unsupported operand type(s) for /: 'PosixPath' and 'NoneType'

ERROR:    Application startup failed. Exiting.
Task was destroyed but it is pending!
task: <Task pending name='Task-3' coro=<FastAPIEventService.__dispatch_from_queue() running at /home/sysuser/Documents/Lalith/InvokeAI/invokeai/app/api/events.py:48> wait_for=<Future pending cb=[Task.task_wakeup()]>>

Was able to invoke the main branch but the same is failing with this branch Not able to host with scripts/invokeai-web.py @StAlKeR7779

That's really strange that you're seeing that error. it seems like one of the following directories are being improperly instantialized:

config.autoimport_dir,
config.lora_dir,
config.embedding_dir,
controlnet_dir

I just made some updates to model management. You may want to try emptying your configs/models.yaml and retrying now.

Jul 27 '23 13:07 brandonrising

I just did some testing of this branch. My focus was on testing that this works on an 8GB GPU, not on exhaustive feature testing, but I noticed a few things along the way.

Test Environment:

Python 3.10.12
GPU: RTX 2070 (8GB VRAM)

Test Steps:

Fresh pip install. I tested both pip install -e .[xformers] and pip install -e .[xformers] --use-pep517 --extra-index-url https://download.pytorch.org/whl/cu117, since I had previously seen differences in onnx behaviour caused by the resulting differences in installed Nvidia libs.
Add this ONNX model via symlink: ln -s ~/invokeai_onnx/stable_diffusion_onnx ~/invokeai/models/sd-1/onnx/stable_diffusion_onnx.
Test text-to-image linear UI with both ONNX and non-ONNX models.
Test image-to-image linear UI with both ONNX and non-ONNX models.
Test text-to-image linear UI with ONNX model and non-ONNX LoRA.

Performance:

UNet denoising:
- ONNX v1.5 model denoising steps: 50/50 [00:11<00:00, 4.45it/s]
- Torch v1.5 model denoising steps: 50/50 [00:05<00:00, 8.86it/s]
I don't have timing logs to quantify it, but the final VAE decoding step is visibly very slow with ONNX due to the known first-load issue. (The blurred latent preview stops changing and then it takes another ~5 seconds for the VAE to decode the final image.)

Notes:

I tried to download an HF ONNX model via the UI. It downloaded the entire model and then failed. Was this expected to work?
After installing with --extra-index-url https://download.pytorch.org/whl/cu117 (as recommended in our docs), I get Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements to ensure all dependencies are met..
- This will cause some models to run very slowly on the CPU, and other models to crash if they contain ONNX ops without CPU support (e.g. GroupNorm).
I get this warning message in the logs: Failed to create TensorrtExecutionProvider. Please reference https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html#requirements to ensure all dependencies are met.
- Do you see this too @brandonrising ? I suspect that me might see better performance if we can get the TRT execution provider working. Then again, it doesn't really matter if it doesn't solve the first-load slowness.
The ONNX model is not listed as an option in the image-to-image linear UI.
LoRAs do not seem to be applied correctly to the ONNX model "yoda" prompt with pokemon LoRA on non-ONNX v1.5 model: "yoda" prompt with pokemon LoRA on ONNX v1.5 model:

Jul 27 '23 15:07 RyanJDick

I just did some testing of this branch. My focus was on testing that this works on an 8GB GPU, not on exhaustive feature testing, but I noticed a few things along the way.

Test Environment:

* Python 3.10.12

* GPU: RTX 2070 (8GB VRAM)

Test Steps:

* Fresh pip install. I tested both `pip install -e .[xformers]` and `pip install -e .[xformers] --use-pep517 --extra-index-url https://download.pytorch.org/whl/cu117`, since I had previously seen differences in onnx behaviour caused by the resulting differences in installed Nvidia libs.

* Add [this](https://huggingface.co/axodoxian/stable_diffusion_onnx/tree/main) ONNX model via symlink: `ln -s ~/invokeai_onnx/stable_diffusion_onnx ~/invokeai/models/sd-1/onnx/stable_diffusion_onnx`.

* Test text-to-image linear UI with both ONNX and non-ONNX models.

* Test image-to-image linear UI with both ONNX and non-ONNX models.

* Test text-to-image linear UI with ONNX model and non-ONNX LoRA.

Performance:

* UNet denoising:
  
  * ONNX v1.5 model denoising steps: `50/50 [00:11<00:00,  4.45it/s]`
  * Torch v1.5 model denoising steps: `50/50 [00:05<00:00,  8.86it/s]`

* I don't have timing logs to quantify it, but the final VAE decoding step is visibly very slow with ONNX due to the known first-load issue. (The blurred latent preview stops changing and then it takes another ~5 seconds for the VAE to decode the final image.)

Notes:

* I tried to download an HF ONNX model via the UI. It downloaded the entire model and then failed. Was this expected to work?

* After installing with `--extra-index-url https://download.pytorch.org/whl/cu117` (as recommended in our docs), I get `Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements to ensure all dependencies are met.`.
  
  * This will cause some models to run very slowly on the CPU, and other models to crash if they contain ONNX ops without CPU support (e.g. GroupNorm).

* I get this warning message in the logs: `Failed to create TensorrtExecutionProvider. Please reference https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html#requirements to ensure all dependencies are met.`
  
  * Do you see this too @brandonrising ? I suspect that me might see better performance if we can get the TRT execution provider working. Then again, it doesn't really matter if it doesn't solve the first-load slowness.

* The ONNX model is not listed as an option in the image-to-image linear UI.

* LoRAs do not seem to be applied correctly to the ONNX model
  "yoda" prompt with pokemon LoRA on **non-ONNX** v1.5 model:
  ![image](https://user-images.githubusercontent.com/14897797/256591976-ebb3fbae-111d-43b8-adcd-6ac78151ff7d.png)
  "yoda" prompt with pokemon LoRA on **ONNX** v1.5 model:
  ![image](https://user-images.githubusercontent.com/14897797/256592194-2418404d-868c-4b36-b791-42da056586f5.png)

I'll take a look into why the model would have failed to download. It could have been related to merging in main.

I went ahead and disabled TensorRT for now. I tried to get it running on my local and it seems we need to do some massaging of the tensors before running the session.

Yes, onnx models are going to be blocked from image2image for now. It will be a text2image and node editor only feature in the short term.

Was that yoda image done on an olive model per chance? Olive models don't support applying loras I believe. If it's just a standard onnx model, could you send me the model and lora you used?

Thanks for testing it out @RyanJDick

Jul 27 '23 18:07 brandonrising

Was that yoda image done on an olive model per chance? Olive models don't support applying loras I believe. If it's just a standard onnx model, could you send me the model and lora you used?

Correct. It was an olive-optimized model.

Jul 27 '23 19:07 RyanJDick

Comments:

I had to pip install onnxruntime.
Ran invokeai-model-install --add axodoxian/stable_diffusion_onnx from the command line -- worked
Confirmed that model appears in the Web Model Manager. It doesn't appear in the model manager TUI however; I can help with this.
Tried the text2image linear workflow and got this:

[2023-07-29 20:51:55,415]::[InvokeAI]::ERROR --> Traceback (most recent call last):
  File "/home/lstein/Projects/InvokeAI/invokeai/app/services/processor.py", line 86, in __process
    outputs = invocation.invoke(
  File "/home/lstein/Projects/InvokeAI/invokeai/app/invocations/onnx.py", line 261, in invoke
    unet.create_session(h, w)
  File "/home/lstein/Projects/InvokeAI/invokeai/backend/model_management/models/base.py", line 625, in create_session
    raise e
  File "/home/lstein/Projects/InvokeAI/invokeai/backend/model_management/models/base.py", line 623, in create_session
    self.session = InferenceSession(self.proto.SerializeToString(), providers=providers, sess_options=sess)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 383, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 435, in _create_inference_session
    sess.initialize_session(providers, provider_options, disabled_optimizers)
onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Failed to find kernel for GroupNorm(1) (node GroupNorm_0). Kernel not found

[2023-07-29 20:51:55,419]::[InvokeAI]::ERROR --> Error while invoking:
[ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Failed to find kernel for GroupNorm(1) (node GroupNorm_0). Kernel not found

Searched the web and found onnxruntime-gpu. Installed that.
Text2Image works, getting ~9 it/s for a 512x512 image. Stock SD-1.5 is giving ~8 it/s at same resolution.
Img2Img not working for me. The model doesn't appear in the model selection menu.
Tried the node editor. Graphs made with ONNXSD1ModelLoader did not work (validation errors), but Onnx Model Loader program loaded and rendered an image.

Overall very positive experience. Just need a better step-by-step guide for people like me who are coming to the ONNX architecture from basically zero.

Jul 30 '23 01:07 lstein

Comments:

I had to pip install onnxruntime.

Ran invokeai-model-install --add axodoxian/stable_diffusion_onnx from the command line -- worked

Confirmed that model appears in the Web Model Manager. It doesn't appear in the model manager TUI however; I can help with this.

Tried the text2image linear workflow and got this:
[2023-07-29 20:51:55,415]::[InvokeAI]::ERROR --> Traceback (most recent call last):
  File "/home/lstein/Projects/InvokeAI/invokeai/app/services/processor.py", line 86, in __process
    outputs = invocation.invoke(
  File "/home/lstein/Projects/InvokeAI/invokeai/app/invocations/onnx.py", line 261, in invoke
    unet.create_session(h, w)
  File "/home/lstein/Projects/InvokeAI/invokeai/backend/model_management/models/base.py", line 625, in create_session
    raise e
  File "/home/lstein/Projects/InvokeAI/invokeai/backend/model_management/models/base.py", line 623, in create_session
    self.session = InferenceSession(self.proto.SerializeToString(), providers=providers, sess_options=sess)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 383, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 435, in _create_inference_session
    sess.initialize_session(providers, provider_options, disabled_optimizers)
onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Failed to find kernel for GroupNorm(1) (node GroupNorm_0). Kernel not found

[2023-07-29 20:51:55,419]::[InvokeAI]::ERROR --> Error while invoking:
[ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Failed to find kernel for GroupNorm(1) (node GroupNorm_0). Kernel not found
Searched the web and found onnxruntime-gpu. Installed that.

Text2Image works, getting ~9 it/s for a 512x512 image. Stock SD-1.5 is giving ~8 it/s at same resolution.

Img2Img not working for me. The model doesn't appear in the model selection menu.

Tried the node editor. Graphs made with ONNXSD1ModelLoader did not work (validation errors), but Onnx Model Loader program loaded and rendered an image.

Overall very positive experience. Just need a better step-by-step guide for people like me who are coming to the ONNX architecture from basically zero.

Thanks for checking it out! I updated the installer to install onnxruntime/onnxruntime-gpu/onnxruntime-directml through optional dependencies based on their elections for different combinations of GPUs and Driver installations. Do you think updating the Readme would suffice telling people they need to install the correct onnxruntime for their environment?

Also yeah, I currently have it setup to only allow onnxruntime for linear text to image and node-editor. It's disabled on all other screens. I was thinking it would be good to go ahead and get it out with minimal functionality and slowly roll out more features as we move forward in separate PRs rather than maintaining this as it gets bigger. Definitely open to a discussion around this though.

I'll definitely work on better documentation for how to use ONNX in workflows, definitely a weak spot in this PR!

Jul 31 '23 13:07 brandonrising

Merged after discussion w/ LStein & Brandon in discord.

Jul 31 '23 21:07 hipsterusername

[2023-07-29 20:51:55,415]::[InvokeAI]::ERROR --> Traceback (most recent call last): File "/home/lstein/Projects/InvokeAI/invokeai/app/services/processor.py", line 86, in __process outputs = invocation.invoke( File "/home/lstein/Projects/InvokeAI/invokeai/app/invocations/onnx.py", line 261, in invoke unet.create_session(h, w) File "/home/lstein/Projects/InvokeAI/invokeai/backend/model_management/models/base.py", line 625, in create_session raise e File "/home/lstein/Projects/InvokeAI/invokeai/backend/model_management/models/base.py", line 623, in create_session self.session = InferenceSession(self.proto.SerializeToString(), providers=providers, sess_options=sess) File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 383, in init self._create_inference_session(providers, provider_options, disabled_optimizers) File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 435, in _create_inference_session sess.initialize_session(providers, provider_options, disabled_optimizers) onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Failed to find kernel for GroupNorm(1) (node GroupNorm_0). Kernel not found

[2023-07-29 20:51:55,419]::[InvokeAI]::ERROR --> Error while invoking: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Failed to find kernel for GroupNorm(1) (node GroupNorm_0). Kernel not found

This error shouldn't be there if I am trying to run the CPU version. When I try to install the CPU specific toolsets still the error is seen

Aug 02 '23 08:08 lalith-mcw

InvokeAI InvokeAI copied to clipboard

[WIP] ONNX Support

Test Environment:

Test Steps:

Performance:

Notes:

Test Environment:

Test Steps:

Performance:

Notes:

InvokeAI
InvokeAI copied to clipboard