ai-toolkit
ai-toolkit copied to clipboard
Wandb Report Fail during sampling step
Hello, I'm training a LoRA for Wan2.1. During the sampling of the videos I get this error. I'm reporting it to wandb, but I believe there is probably a bug, as it seems to be trying to upload an image, even though the sampler generates videos of 91 frames, not images.
This is the exception for more context:
2025-10-28 01:35:30 2025-10-28 01:35:30,610 - job_training_api - ERROR - [Job: 3307c995-a102-4d8e-b9cb-8f754e03ae36] - An unexpected error occurred in the job runner: 'list' object has no attribute 'ndim'
2025-10-28 01:35:30 Traceback (most recent call last):
2025-10-28 01:35:30 File "/app/src/lib/runner.py", line 53, in wrapper
2025-10-28 01:35:30 func(job, repo, log_and_store, *args, **kwargs)
2025-10-28 01:35:30 File "/app/src/lib/runner.py", line 138, in run_job_in_background
2025-10-28 01:35:30 worker.run() # Any exception here will be caught by the @job_runner
2025-10-28 01:35:30 ^^^^^^^^^^^^
2025-10-28 01:35:30 File "/app/src/lib/workers/toolkit.py", line 30, in run
2025-10-28 01:35:30 trainer.run()
2025-10-28 01:35:30 File "/app/src/lib/trainers/diffusion/base.py", line 19, in run
2025-10-28 01:35:30 return job.run_job(job_config_dict)
2025-10-28 01:35:30 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-28 01:35:30 File "/app/ai-toolkit/toolkit/job.py", line 43, in run_job
2025-10-28 01:35:30 job.run()
2025-10-28 01:35:30 File "/app/ai-toolkit/jobs/ExtensionJob.py", line 22, in run
2025-10-28 01:35:30 process.run()
2025-10-28 01:35:30 File "/app/ai-toolkit/jobs/process/BaseSDTrainProcess.py", line 2218, in run
2025-10-28 01:35:30 self.sample(self.step_num)
2025-10-28 01:35:30 File "/app/ai-toolkit/jobs/process/BaseSDTrainProcess.py", line 366, in sample
2025-10-28 01:35:30 self.sd.generate_images(gen_img_config_list, sampler=sample_config.sampler)
2025-10-28 01:35:30 File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
2025-10-28 01:35:30 return func(*args, **kwargs)
2025-10-28 01:35:30 ^^^^^^^^^^^^^^^^^^^^^
2025-10-28 01:35:30 File "/app/ai-toolkit/toolkit/models/base_model.py", line 611, in generate_images
2025-10-28 01:35:30 gen_config.log_image(img, i)
2025-10-28 01:35:30 File "/app/ai-toolkit/toolkit/config_modules.py", line 1227, in log_image
2025-10-28 01:35:30 self.logger.log_image(image, count, self.prompt)
2025-10-28 01:35:30 File "/app/ai-toolkit/toolkit/logging_aitk.py", line 71, in log_image
2025-10-28 01:35:30 image = self._image(image, caption=caption, *args, **kwargs)
2025-10-28 01:35:30 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-28 01:35:30 File "/usr/local/lib/python3.11/dist-packages/wandb/sdk/data_types/image.py", line 277, in __init__
2025-10-28 01:35:30 self._initialize_from_data(data_or_path, mode, file_type, normalize)
2025-10-28 01:35:30 File "/usr/local/lib/python3.11/dist-packages/wandb/sdk/data_types/image.py", line 434, in _initialize_from_data
2025-10-28 01:35:30 if data.ndim > 2: # type: ignore [union-attr]
2025-10-28 01:35:30 ^^^^^^^^^
2025-10-28 01:35:30 AttributeError: 'list' object has no attribute 'ndim'
Has anyone had these issues too?