InvokeAI icon indicating copy to clipboard operation
InvokeAI copied to clipboard

[bug]: The accelerate package has deprecated `logging_dir`, removed in `v0.18.0`,replaced with `project_dir`, placing logs in `project_dir/[name]`.

Open changeling opened this issue 1 year ago • 2 comments

Is there an existing issue for this?

  • [X] I have searched the existing issues

OS

macOS

GPU

mps

VRAM

16GB

What version did you experience this issue on?

2.3.2

What happened?

Running Textual Inversion training with accelerate 0.16.0 yields the following:

.../.venv/lib/python3.10/site-packages/accelerate/accelerator.py:243: FutureWarning: `logging_dir` is deprecated and will be removed in version 0.18.0 of 🤗 Accelerate. Use `project_dir` instead.```

### Screenshots

_No response_

### Additional context

Reference https://github.com/huggingface/accelerate/pull/916 for discussion and details/implementation, particularly https://github.com/huggingface/accelerate/pull/916#issuecomment-1346954827 and following comments.

### Contact Details

_No response_

changeling avatar Mar 28 '23 15:03 changeling

For more context here, this is now breaking in a fresh installation of InvokeAI version 2.3.5.post2.

From what I can tell, it's this line that is causing the issue: https://github.com/invoke-ai/InvokeAI/blob/5c740452f687430e377f3f819c7e86b231e1ca30/invokeai/backend/training/textual_inversion_training.py#L591

It should instead pass in the argument project_dir=logging_dir or project_dir=output_dir (not sure if Accelerate will create a logs dir in there for you).

felixnorden avatar Jun 15 '23 14:06 felixnorden

same bug here MacOS M1

mightynico avatar Jun 20 '23 17:06 mightynico

For more context here, this is now breaking in a fresh installation of InvokeAI version 2.3.5.post2.

From what I can tell, it's this line that is causing the issue:

https://github.com/invoke-ai/InvokeAI/blob/5c740452f687430e377f3f819c7e86b231e1ca30/invokeai/backend/training/textual_inversion_training.py#L591

It should instead pass in the argument project_dir=logging_dir or project_dir=output_dir (not sure if Accelerate will create a logs dir in there for you).

Did this fix the problem for you? I had a python crash with both fixes. Mac M1

pharmpk avatar Jul 08 '23 04:07 pharmpk

Can this be replicated on latest version (3.0 beta)?

hipsterusername avatar Jul 08 '23 18:07 hipsterusername

I'm using the latest, from yesterday, 'stable' version. I should give the 3.0 beta a try.

pharmpk avatar Jul 08 '23 18:07 pharmpk

Hmm, installed 3.0-b3 OK. I was able to generate some images from text. Text inversion prodcued the same result as before, immediate crash. Tried the two fixes above with the same results. The error message from the last attempt. Mac M1.

O Textual inversion training /Volumes/easystore-1/StableDiffusion/InvokeAI/.venv/lib/python3.10/site-packages/torch/cuda/amp/grad_scaler.py:120:UserWarnin g: torch.cuda.amp.GradScaler is enabled, but CUDA is not available. Disabling. warnings.warn("torch.cuda.amp. GradScaler is enabled, but CUDA is not available. Disabling.") 07/08/2023 13:32:08 - INFO - invokeai.backend.training.textual_inversion_training - Distributed environment: NO g Num processes: 1 Process index: 0 Local process index: 0 Device: mps Mixed precision type: fp16 [2023-07-08 13:32:08,492]::[InvokeAI]::ERROR --> An exception occurred during training. The exception was: 07/08/2023 13:32:08 - ERROR - InvokeAI - An exception occurred during training. The exception was: [2023-07-08 13:32:08,492]::[InvokeAI]: :ERROR --> 'InvokeAIAppConfig' object has no attribute 'cache_dir' 07/08/2023 13:32:08 - ERROR - InvokeAI - 'InvokeAlAppConfig' object has no attribute 'cache_dir' [2023-07-08 13:32:08,492]:: [InvokeAI]::ERROR --> DETAILS: 07/08/2023 13:32:08 - ERROR - InvokeAI - DETAILS: [2023-07-08 13:32:08,493]:: [InvokeAI]::ERROR --> Traceback (most recent call last): File "/Volumes/easystore-1/StableDiffusion/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/frontend/training/textual_in version.py", line 426, in do_front_end do_textual_inversion_training(InvokeAIAppConfig.get_config(),*xargs) File "/Volumes/easvstore-1/StableDiffusion/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/backend/training/textualinv ersion_training.py", line 645, in do_textual_inversion_training pipeline_args = dict (cache_dir=config.cache_dir) AttributeError: 'InvokeAIAppConfig' object has no attribute 'cache dir' 07/08/2023 13:32:08 - ERROR - InvokeAI - Traceback (most recent call last): File "/Volumes/easystore-1/StableDiffusion/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/frontend/training/textual_in version.py", line 426, in do_front_end do_textual_inversion_training(InvokeAIAppConfig.get_config(),*xargs) File "/Volumes/easystore-1/StableDiffusion/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/backend/training/textual_inv ersion_training.py", line 645, in do_textual_inversion_training pipeline_args = dict (cache_dir=config.cache_dir) AttributeError: 'InvokeAIAppConfig' object has no attribute 'cache_dir'

pharmpk avatar Jul 08 '23 19:07 pharmpk