num_samples not provided

[Miosame]

I've put in a gdrive path to the images and I believe the new method is not setting the num_samples like the old one used to, so it's throwing a:

ValueError: num_samples should be a positive integer value, but got num_samples=0

once you start the training

[TheLastBen]

copy the full error log so I can see where is the problem, it seems that the path specified didn't contain images

[Miosame]

Traceback (most recent call last):
  File "/content/diffusers/examples/dreambooth/train_dreambooth.py", line 735, in <module>
    main()
  File "/content/diffusers/examples/dreambooth/train_dreambooth.py", line 528, in main
    train_dataset, batch_size=args.train_batch_size, shuffle=True, collate_fn=collate_fn
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 353, in __init__
    sampler = RandomSampler(dataset, generator=generator)  # type: ignore[arg-type]
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/sampler.py", line 108, in __init__
    "value, but got num_samples={}".format(self.num_samples))
ValueError: num_samples should be a positive integer value, but got num_samples=0
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/accelerate_cli.py", line 43, in main
    args.func(args)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 837, in launch_command
    simple_launcher(args)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 354, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '/content/diffusers/examples/dreambooth/train_dreambooth.py', '--image_captions_filename', '--train_text_encoder', '--save_starting_step=500', '--stop_text_encoder_training=300', '--save_n_steps=500', '--pretrained_model_name_or_path=/content/stable-diffusion-v1-5', '--instance_data_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/xxx/instance_images', '--output_dir=/content/models/xxx', '--instance_prompt=', '--seed=93356', '--resolution=512', '--mixed_precision=fp16', '--train_batch_size=1', '--gradient_accumulation_steps=1', '--use_8bit_adam', '--learning_rate=2e-6', '--lr_scheduler=polynomial', '--center_crop', '--lr_warmup_steps=0', '--max_train_steps=3000']' returned non-zero exit status 1.
Something went wrong

The gdrive path I supplied: /content/gdrive/MyDrive/xxx

[Miosame]

regarding your edit: " it seems that the path specified didn't contain images"

it contains 5.1k images, cutting that down to 2k seems to have made the previous cell load much longer now (probably processing?) - is there a limit of how much images a folder can contain?

[Miosame]

yeah that did it, now it started working, so I assume there's something wrong if there's that many images?

[TheLastBen]

you won't get any noticeable results with 2k images, unless you push the steps to unreasonable numbers

[Miosame]

I've reran this time with 10k images and the error didn't happen, so maybe the other half of the images had one that was corrupted, if I come across which image triggers this, I'll reopen, until then, thanks for the quick reply!