fast-stable-diffusion icon indicating copy to clipboard operation
fast-stable-diffusion copied to clipboard

Published 20 hours ago • verified publisher icon TheLastBen

Training has broken

Open Daniel-Kelvich opened this issue 3 years ago • 91 comments

Training the unet... Traceback (most recent call last): File "/content/diffusers/examples/dreambooth/train_dreambooth.py", line 18, in from diffusers import AutoencoderKL, DDPMScheduler, StableDiffusionPipeline, UNet2DConditionModel File "/usr/local/lib/python3.7/dist-packages/diffusers/init.py", line 21, in from .models import AutoencoderKL, UNet2DConditionModel, UNet2DModel, VQModel File "/usr/local/lib/python3.7/dist-packages/diffusers/models/init.py", line 19, in from .unet_2d import UNet2DModel File "/usr/local/lib/python3.7/dist-packages/diffusers/models/unet_2d.py", line 11, in from .unet_blocks import UNetMidBlock2D, get_down_block, get_up_block File "/usr/local/lib/python3.7/dist-packages/diffusers/models/unet_blocks.py", line 20, in from .attention import AttentionBlock, SpatialTransformer File "/usr/local/lib/python3.7/dist-packages/diffusers/models/attention.py", line 24, in from ..models.embeddings import ImagePositionalEmbeddings ImportError: cannot import name 'ImagePositionalEmbeddings' from 'diffusers.models.embeddings' (/usr/local/lib/python3.7/dist-packages/diffusers/models/embeddings.py) Traceback (most recent call last): File "/usr/local/bin/accelerate", line 8, in sys.exit(main()) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/accelerate_cli.py", line 43, in main args.func(args) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 837, in launch_command simple_launcher(args) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 354, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/usr/bin/python3', '/content/diffusers/examples/dreambooth/train_dreambooth.py', '--image_captions_filename', '--train_only_unet', '--Session_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/AnnabBaseline3A100', '--save_starting_step=500', '--save_n_steps=0', '--pretrained_model_name_or_path=/content/stable-diffusion-v1-5', '--instance_data_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/AnnabBaseline3A100/instance_images', '--output_dir=/content/models/AnnabBaseline3A100', '--instance_prompt=', '--seed=96576', '--resolution=512', '--mixed_precision=no', '--train_batch_size=1', '--gradient_accumulation_steps=1', '--use_8bit_adam', '--learning_rate=2e-6', '--lr_scheduler=polynomial', '--center_crop', '--lr_warmup_steps=0', '--max_train_steps=2000']' returned non-zero exit status 1. Something went wrong

Daniel-Kelvich avatar Nov 03 '22 16:11 Daniel-Kelvich

I am also experiencing this issue.

fredrickflower avatar Nov 03 '22 16:11 fredrickflower

Just posting here since it's a bit hard to read and to find the root error: It's ImportError, The first 15 lines.

askiiart avatar Nov 03 '22 16:11 askiiart

Also have this issue

Jongulo avatar Nov 03 '22 16:11 Jongulo

same

corbettaluigi avatar Nov 03 '22 16:11 corbettaluigi

I'm running it right now, no issue

TheLastBen avatar Nov 03 '22 16:11 TheLastBen

update to the latest colab

TheLastBen avatar Nov 03 '22 16:11 TheLastBen

Just ran again same issue:

Training the unet... Traceback (most recent call last): File "/content/diffusers/examples/dreambooth/train_dreambooth.py", line 18, in from diffusers import AutoencoderKL, DDPMScheduler, StableDiffusionPipeline, UNet2DConditionModel File "/usr/local/lib/python3.7/dist-packages/diffusers/init.py", line 21, in from .models import AutoencoderKL, UNet2DConditionModel, UNet2DModel, VQModel File "/usr/local/lib/python3.7/dist-packages/diffusers/models/init.py", line 19, in from .unet_2d import UNet2DModel File "/usr/local/lib/python3.7/dist-packages/diffusers/models/unet_2d.py", line 11, in from .unet_blocks import UNetMidBlock2D, get_down_block, get_up_block File "/usr/local/lib/python3.7/dist-packages/diffusers/models/unet_blocks.py", line 20, in from .attention import AttentionBlock, SpatialTransformer File "/usr/local/lib/python3.7/dist-packages/diffusers/models/attention.py", line 24, in from ..models.embeddings import ImagePositionalEmbeddings ImportError: cannot import name 'ImagePositionalEmbeddings' from 'diffusers.models.embeddings' (/usr/local/lib/python3.7/dist-packages/diffusers/models/embeddings.py) Traceback (most recent call last): File "/usr/local/bin/accelerate", line 8, in sys.exit(main()) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/accelerate_cli.py", line 43, in main args.func(args) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 837, in launch_command simple_launcher(args) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 354, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/usr/bin/python3', '/content/diffusers/examples/dreambooth/train_dreambooth.py', '--image_captions_filename', '--train_only_unet', '--Session_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/erindreambooth', '--save_starting_step=500', '--save_n_steps=0', '--pretrained_model_name_or_path=/content/stable-diffusion-v1-5', '--instance_data_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/erindreambooth/instance_images', '--output_dir=/content/models/erindreambooth', '--instance_prompt=', '--seed=96576', '--resolution=512', '--mixed_precision=no', '--train_batch_size=1', '--gradient_accumulation_steps=1', '--use_8bit_adam', '--learning_rate=2e-6', '--lr_scheduler=polynomial', '--center_crop', '--lr_warmup_steps=0', '--max_train_steps=3000']' returned non-zero exit status 1. Something went wrong

fredrickflower avatar Nov 03 '22 16:11 fredrickflower

what settings, is it the default model ?

TheLastBen avatar Nov 03 '22 16:11 TheLastBen

Still same error. Default model (1.5), new method.

Daniel-Kelvich avatar Nov 03 '22 17:11 Daniel-Kelvich

Error here too cannot use it. was fine yesterday

Did normal hugging face token, no special model

Traceback (most recent call last): File "/content/diffusers/examples/dreambooth/train_dreambooth.py", line 18, in from diffusers import AutoencoderKL, DDPMScheduler, StableDiffusionPipeline, UNet2DConditionModel File "/usr/local/lib/python3.7/dist-packages/diffusers/init.py", line 21, in from .models import AutoencoderKL, UNet2DConditionModel, UNet2DModel, VQModel File "/usr/local/lib/python3.7/dist-packages/diffusers/models/init.py", line 19, in from .unet_2d import UNet2DModel File "/usr/local/lib/python3.7/dist-packages/diffusers/models/unet_2d.py", line 11, in from .unet_blocks import UNetMidBlock2D, get_down_block, get_up_block File "/usr/local/lib/python3.7/dist-packages/diffusers/models/unet_blocks.py", line 20, in from .attention import AttentionBlock, SpatialTransformer File "/usr/local/lib/python3.7/dist-packages/diffusers/models/attention.py", line 24, in from ..models.embeddings import ImagePositionalEmbeddings ImportError: cannot import name 'ImagePositionalEmbeddings' from 'diffusers.models.embeddings' (/usr/local/lib/python3.7/dist-packages/diffusers/models/embeddings.py) Traceback (most recent call last): File "/usr/local/bin/accelerate", line 8, in sys.exit(main()) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/accelerate_cli.py", line 43, in main args.func(args) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 837, in launch_command simple_launcher(args) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 354, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/usr/bin/python3', '/content/diffusers/examples/dreambooth/train_dreambooth.py', '--image_captions_filename', '--train_text_encoder', '--save_starting_step=3000', '--stop_text_encoder_training=3000', '--save_n_steps=6000', '--Session_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/MJBEK', '--pretrained_model_name_or_path=/content/stable-diffusion-v1-5', '--instance_data_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/MJBEK/instance_images', '--output_dir=/content/models/MJBEK', '--instance_prompt=', '--seed=96576', '--resolution=512', '--mixed_precision=no', '--train_batch_size=1', '--gradient_accumulation_steps=1', '--use_8bit_adam', '--learning_rate=2e-6', '--lr_scheduler=polynomial', '--center_crop', '--lr_warmup_steps=0', '--max_train_steps=30000']' returned non-zero exit status 1. Something went wrong

Is it the universal pull"Make notebooks universal #150"? Maybe gdrive project issue now because of that?

rasamaya avatar Nov 03 '22 17:11 rasamaya

what settings, is it the default model ?

Default model, 3000 steps, 100% text encoding, images uploaded through g drive, no checkpoint saving, female face, google colab pro so faster gpu.

fredrickflower avatar Nov 03 '22 17:11 fredrickflower

nope, not the pull request, it was closed, maybe the update of this morning use this old version (yesterday) and see if it works : https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/dd79c4f4fd89d55ffa405a372742338404b3cdcd/fast-DreamBooth.ipynb

in the new notebook try setting contains_faces to "No"

TheLastBen avatar Nov 03 '22 17:11 TheLastBen

nope, not the pull request, it was closed, maybe the update of this morning use this old version (yesterday) and see if it works : https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/dd79c4f4fd89d55ffa405a372742338404b3cdcd/fast-DreamBooth.ipynb

in the new notebook try setting contains_faces to "No"

Tried the old dreambooth notebook and got error:

Traceback (most recent call last): File "/content/diffusers/examples/dreambooth/train_dreambooth.py", line 18, in from diffusers import AutoencoderKL, DDPMScheduler, StableDiffusionPipeline, UNet2DConditionModel File "/usr/local/lib/python3.7/dist-packages/diffusers/init.py", line 21, in from .models import AutoencoderKL, UNet2DConditionModel, UNet2DModel, VQModel File "/usr/local/lib/python3.7/dist-packages/diffusers/models/init.py", line 19, in from .unet_2d import UNet2DModel File "/usr/local/lib/python3.7/dist-packages/diffusers/models/unet_2d.py", line 11, in from .unet_blocks import UNetMidBlock2D, get_down_block, get_up_block File "/usr/local/lib/python3.7/dist-packages/diffusers/models/unet_blocks.py", line 20, in from .attention import AttentionBlock, SpatialTransformer File "/usr/local/lib/python3.7/dist-packages/diffusers/models/attention.py", line 24, in from ..models.embeddings import ImagePositionalEmbeddings ImportError: cannot import name 'ImagePositionalEmbeddings' from 'diffusers.models.embeddings' (/usr/local/lib/python3.7/dist-packages/diffusers/models/embeddings.py) Traceback (most recent call last): File "/usr/local/bin/accelerate", line 8, in sys.exit(main()) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/accelerate_cli.py", line 43, in main args.func(args) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 837, in launch_command simple_launcher(args) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 354, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/usr/bin/python3', '/content/diffusers/examples/dreambooth/train_dreambooth.py', '--image_captions_filename', '--train_text_encoder', '--save_starting_step=500', '--stop_text_encoder_training=3010', '--save_n_steps=0', '--pretrained_model_name_or_path=/content/stable-diffusion-v1-5', '--instance_data_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/erindreamboothmodel/instance_images', '--output_dir=/content/models/erindreamboothmodel', '--instance_prompt=', '--seed=96576', '--resolution=512', '--mixed_precision=no', '--train_batch_size=1', '--gradient_accumulation_steps=1', '--use_8bit_adam', '--learning_rate=2e-6', '--lr_scheduler=polynomial', '--center_crop', '--lr_warmup_steps=0', '--max_train_steps=3000']' returned non-zero exit status 1. Something went wrong

fredrickflower avatar Nov 03 '22 17:11 fredrickflower

Im getting this on training:

Traceback (most recent call last):
  File "/content/diffusers/examples/dreambooth/train_dreambooth.py", line 18, in <module>
    from diffusers import AutoencoderKL, DDPMScheduler, StableDiffusionPipeline, UNet2DConditionModel
  File "/usr/local/lib/python3.7/dist-packages/diffusers/__init__.py", line 21, in <module>
    from .models import AutoencoderKL, UNet2DConditionModel, UNet2DModel, VQModel
  File "/usr/local/lib/python3.7/dist-packages/diffusers/models/__init__.py", line 19, in <module>
    from .unet_2d import UNet2DModel
  File "/usr/local/lib/python3.7/dist-packages/diffusers/models/unet_2d.py", line 11, in <module>
    from .unet_blocks import UNetMidBlock2D, get_down_block, get_up_block
  File "/usr/local/lib/python3.7/dist-packages/diffusers/models/unet_blocks.py", line 20, in <module>
    from .attention import AttentionBlock, SpatialTransformer
  File "/usr/local/lib/python3.7/dist-packages/diffusers/models/attention.py", line 24, in <module>
    from ..models.embeddings import ImagePositionalEmbeddings
ImportError: cannot import name 'ImagePositionalEmbeddings' from 'diffusers.models.embeddings' (/usr/local/lib/python3.7/dist-packages/diffusers/models/embeddings.py)
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/accelerate_cli.py", line 43, in main
    args.func(args)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 837, in launch_command
    simple_launcher(args)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 354, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '/content/diffusers/examples/dreambooth/train_dreambooth.py', '--image_captions_filename', '--train_text_encoder', '--save_starting_step=500', '--stop_text_encoder_training=6160', '--save_n_steps=0', '--Session_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/jtoytest3', '--pretrained_model_name_or_path=/content/stable-diffusion-v1-5', '--instance_data_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/jtoytest3/instance_images', '--output_dir=/content/models/jtoytest3', '--instance_prompt=', '--seed=96576', '--resolution=512', '--mixed_precision=no', '--train_batch_size=1', '--gradient_accumulation_steps=1', '--use_8bit_adam', '--learning_rate=2e-6', '--lr_scheduler=polynomial', '--center_crop', '--lr_warmup_steps=0', '--max_train_steps=17600']' returned non-zero exit status 1.
Something went wrong

jtoy avatar Nov 03 '22 17:11 jtoy

@TheLastBen This is weird. I've checked the original diffusers and your fork and it seems that colab code does not resemble either of them.

Daniel-Kelvich avatar Nov 03 '22 17:11 Daniel-Kelvich

Also of note, I trained a model this AM and itworked, then an hour later, I tried 5 more models in a row and they all died.

jtoy avatar Nov 03 '22 17:11 jtoy

You didn't update the diffusers ? which GPU do you have now ?

TheLastBen avatar Nov 03 '22 17:11 TheLastBen

You didn't update the diffusers ? which GPU do you have now ?

Ahhhh, I've only just got Colab Pro so I'm using a better GPU, how can I change the diffusers and how do I check which GPU I'm using?

fredrickflower avatar Nov 03 '22 17:11 fredrickflower

run !nvidia-smi in a new cell

TheLastBen avatar Nov 03 '22 17:11 TheLastBen

run !nvidia-smi in a new cell

I'm running an A100

fredrickflower avatar Nov 03 '22 17:11 fredrickflower

OK, I'll try to fix that

TheLastBen avatar Nov 03 '22 17:11 TheLastBen

OK, I'll try to fix that

Cheers man

fredrickflower avatar Nov 03 '22 17:11 fredrickflower

I tried A100 and T4 same error.

Daniel-Kelvich avatar Nov 03 '22 17:11 Daniel-Kelvich

@fredrickflower try now with the new fix : https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast-DreamBooth.ipynb

TheLastBen avatar Nov 03 '22 18:11 TheLastBen

I tried A100 and T4 same error.

I'm running it on a T4 at the moment, I can't reproduce the error

TheLastBen avatar Nov 03 '22 18:11 TheLastBen

@TheLastBen I'm still seeing the same error on https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast-DreamBooth.ipynb (also running an A100)

sphuff avatar Nov 03 '22 18:11 sphuff

@fredrickflower try now with the new fix : https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast-DreamBooth.ipynb

Same error unfortunately:

Training the unet... Traceback (most recent call last): File "/content/diffusers/examples/dreambooth/train_dreambooth.py", line 18, in from diffusers import AutoencoderKL, DDPMScheduler, StableDiffusionPipeline, UNet2DConditionModel File "/usr/local/lib/python3.7/dist-packages/diffusers/init.py", line 21, in from .models import AutoencoderKL, UNet2DConditionModel, UNet2DModel, VQModel File "/usr/local/lib/python3.7/dist-packages/diffusers/models/init.py", line 19, in from .unet_2d import UNet2DModel File "/usr/local/lib/python3.7/dist-packages/diffusers/models/unet_2d.py", line 11, in from .unet_blocks import UNetMidBlock2D, get_down_block, get_up_block File "/usr/local/lib/python3.7/dist-packages/diffusers/models/unet_blocks.py", line 20, in from .attention import AttentionBlock, SpatialTransformer File "/usr/local/lib/python3.7/dist-packages/diffusers/models/attention.py", line 24, in from ..models.embeddings import ImagePositionalEmbeddings ImportError: cannot import name 'ImagePositionalEmbeddings' from 'diffusers.models.embeddings' (/usr/local/lib/python3.7/dist-packages/diffusers/models/embeddings.py) Traceback (most recent call last): File "/usr/local/bin/accelerate", line 8, in sys.exit(main()) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/accelerate_cli.py", line 43, in main args.func(args) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 837, in launch_command simple_launcher(args) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 354, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/usr/bin/python3', '/content/diffusers/examples/dreambooth/train_dreambooth.py', '--image_captions_filename', '--train_only_unet', '--Session_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/errortest', '--save_starting_step=2000', '--save_n_steps=0', '--pretrained_model_name_or_path=/content/stable-diffusion-v1-5', '--instance_data_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/errortest/instance_images', '--output_dir=/content/models/errortest', '--instance_prompt=', '--seed=96576', '--resolution=512', '--mixed_precision=no', '--train_batch_size=1', '--gradient_accumulation_steps=1', '--use_8bit_adam', '--learning_rate=2e-6', '--lr_scheduler=polynomial', '--center_crop', '--lr_warmup_steps=0', '--max_train_steps=3000']' returned non-zero exit status 1. Something went wrong

fredrickflower avatar Nov 03 '22 18:11 fredrickflower

I think it's fixed now, disconnect from the colab and reconnect

TheLastBen avatar Nov 03 '22 18:11 TheLastBen

Error with T4.

UnboundLocalError: local variable 'save_dir' referenced before assignment Progress:|████ | 15% 300/2000 [05:01<28:29, 1.01s/it, loss=0.0318, lr=1.72e-6] Traceback (most recent call last): File "/usr/local/bin/accelerate", line 8, in sys.exit(main()) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/accelerate_cli.py", line 43, in main args.func(args) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 837, in launch_command simple_launcher(args) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 354, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/usr/bin/python3', '/content/diffusers/examples/dreambooth/train_dreambooth.py', '--image_captions_filename', '--train_text_encoder', '--save_starting_step=500', '--stop_text_encoder_training=300', '--save_n_steps=0', '--Session_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/Test', '--pretrained_model_name_or_path=/content/stable-diffusion-v1-5', '--instance_data_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/Test/instance_images', '--output_dir=/content/models/Test', '--instance_prompt=', '--seed=96576', '--resolution=512', '--mixed_precision=fp16', '--train_batch_size=1', '--gradient_accumulation_steps=1', '--use_8bit_adam', '--learning_rate=2e-6', '--lr_scheduler=polynomial', '--center_crop', '--lr_warmup_steps=0', '--max_train_steps=2000']' returned non-zero exit status 1. Something went wrong

Daniel-Kelvich avatar Nov 03 '22 18:11 Daniel-Kelvich

that's not related to the GPU, that's a different error EDIT : fixed, a mistake I made with earlier update

TheLastBen avatar Nov 03 '22 18:11 TheLastBen