InvokeAI
InvokeAI copied to clipboard
[bug]: The accelerate package has deprecated `logging_dir`, removed in `v0.18.0`,replaced with `project_dir`, placing logs in `project_dir/[name]`.
Is there an existing issue for this?
- [X] I have searched the existing issues
OS
macOS
GPU
mps
VRAM
16GB
What version did you experience this issue on?
2.3.2
What happened?
Running Textual Inversion training with accelerate 0.16.0
yields the following:
.../.venv/lib/python3.10/site-packages/accelerate/accelerator.py:243: FutureWarning: `logging_dir` is deprecated and will be removed in version 0.18.0 of 🤗 Accelerate. Use `project_dir` instead.```
### Screenshots
_No response_
### Additional context
Reference https://github.com/huggingface/accelerate/pull/916 for discussion and details/implementation, particularly https://github.com/huggingface/accelerate/pull/916#issuecomment-1346954827 and following comments.
### Contact Details
_No response_
For more context here, this is now breaking in a fresh installation of InvokeAI version 2.3.5.post2
.
From what I can tell, it's this line that is causing the issue: https://github.com/invoke-ai/InvokeAI/blob/5c740452f687430e377f3f819c7e86b231e1ca30/invokeai/backend/training/textual_inversion_training.py#L591
It should instead pass in the argument project_dir=logging_dir
or project_dir=output_dir
(not sure if Accelerate will create a logs dir in there for you).
same bug here MacOS M1
For more context here, this is now breaking in a fresh installation of InvokeAI version
2.3.5.post2
.From what I can tell, it's this line that is causing the issue:
https://github.com/invoke-ai/InvokeAI/blob/5c740452f687430e377f3f819c7e86b231e1ca30/invokeai/backend/training/textual_inversion_training.py#L591
It should instead pass in the argument
project_dir=logging_dir
orproject_dir=output_dir
(not sure if Accelerate will create a logs dir in there for you).
Did this fix the problem for you? I had a python crash with both fixes. Mac M1
Can this be replicated on latest version (3.0 beta)?
I'm using the latest, from yesterday, 'stable' version. I should give the 3.0 beta a try.
Hmm, installed 3.0-b3 OK. I was able to generate some images from text. Text inversion prodcued the same result as before, immediate crash. Tried the two fixes above with the same results. The error message from the last attempt. Mac M1.
O Textual inversion training /Volumes/easystore-1/StableDiffusion/InvokeAI/.venv/lib/python3.10/site-packages/torch/cuda/amp/grad_scaler.py:120:UserWarnin g: torch.cuda.amp.GradScaler is enabled, but CUDA is not available. Disabling. warnings.warn("torch.cuda.amp. GradScaler is enabled, but CUDA is not available. Disabling.") 07/08/2023 13:32:08 - INFO - invokeai.backend.training.textual_inversion_training - Distributed environment: NO g Num processes: 1 Process index: 0 Local process index: 0 Device: mps Mixed precision type: fp16 [2023-07-08 13:32:08,492]::[InvokeAI]::ERROR --> An exception occurred during training. The exception was: 07/08/2023 13:32:08 - ERROR - InvokeAI - An exception occurred during training. The exception was: [2023-07-08 13:32:08,492]::[InvokeAI]: :ERROR --> 'InvokeAIAppConfig' object has no attribute 'cache_dir' 07/08/2023 13:32:08 - ERROR - InvokeAI - 'InvokeAlAppConfig' object has no attribute 'cache_dir' [2023-07-08 13:32:08,492]:: [InvokeAI]::ERROR --> DETAILS: 07/08/2023 13:32:08 - ERROR - InvokeAI - DETAILS: [2023-07-08 13:32:08,493]:: [InvokeAI]::ERROR --> Traceback (most recent call last): File "/Volumes/easystore-1/StableDiffusion/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/frontend/training/textual_in version.py", line 426, in do_front_end do_textual_inversion_training(InvokeAIAppConfig.get_config(),*xargs) File "/Volumes/easvstore-1/StableDiffusion/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/backend/training/textualinv ersion_training.py", line 645, in do_textual_inversion_training pipeline_args = dict (cache_dir=config.cache_dir) AttributeError: 'InvokeAIAppConfig' object has no attribute 'cache dir' 07/08/2023 13:32:08 - ERROR - InvokeAI - Traceback (most recent call last): File "/Volumes/easystore-1/StableDiffusion/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/frontend/training/textual_in version.py", line 426, in do_front_end do_textual_inversion_training(InvokeAIAppConfig.get_config(),*xargs) File "/Volumes/easystore-1/StableDiffusion/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/backend/training/textual_inv ersion_training.py", line 645, in do_textual_inversion_training pipeline_args = dict (cache_dir=config.cache_dir) AttributeError: 'InvokeAIAppConfig' object has no attribute 'cache_dir'