NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.

Open peepeepeepoopoopoo opened this issue 3 weeks ago • 0 comments

i try train chroma lora and not works, i have clean installation

(venv) root@2b0d3ea49364:/workspace/ai-toolkit# python run.py 0.dataset.chroma/yaml/chroma_config.yaml Running 1 job { "type": "sd_trainer", "training_folder": "/workspace/ai-toolkit/0.dataset.chroma", "trigger_word": "@dimakava14", "device": "cuda:0", "network": { "type": "lora", "linear": 256, "linear_alpha": 256 }, "save": { "dtype": "bf16", "save_every": 500, "max_step_saves_to_keep": 10 }, "model": { "name_or_path": "/workspace/ai-toolkit/0.dataset.chroma/diffusers", "is_flux": true, "quantize": false, "low_cpu_mem_usage": false }, "sample": { "sampler": "FlowMatchEulerDiscreteScheduler", "sample_every": 500, "width": 1024, "height": 1024, "prompts": [ "@dimakava14 portrait, professional lighting", "@dimakava14 casual photo" ], "seed": 42, "walk_seed": true, "guidance_scale": 4, "sample_steps": 25 }, "datasets": [ { "folder_path": "/workspace/ai-toolkit/0.dataset.chroma", "caption_ext": "txt", "caption_dropout_rate": 0.05, "resolution": [ 1024 ], "cache_latents": true, "cache_latents_to_disk": true, "random_crop": false } ], "train": { "batch_size": 16, "steps": 5000, "gradient_accumulation_steps": 4, "train_unet": true, "train_text_encoder": true, "optimizer": "AdamW", "lr": 0.0001, "optimizer_params": { "weight_decay": 0.01, "betas": [ 0.9, 0.999 ] }, "ema_config": { "use_ema": true, "ema_decay": 0.9999 }, "dtype": "bf16", "gradient_checkpointing": false, "mixed_precision": "bf16", "lr_scheduler": "constant_with_warmup", "lr_warmup_steps": 100 } } Using EMA

#############################################

Running job: chroma_lora_dimakava14

#############################################

Running 1 process Loading Flux model Loading transformer Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00, 1.23s/it] Error running job: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.

======================================== Result:

0 completed jobs
1 failure ======================================== Traceback (most recent call last): File "/workspace/ai-toolkit/run.py", line 120, in main() File "/workspace/ai-toolkit/run.py", line 108, in main raise e File "/workspace/ai-toolkit/run.py", line 96, in main job.run() File "/workspace/ai-toolkit/jobs/ExtensionJob.py", line 22, in run process.run() File "/workspace/ai-toolkit/jobs/process/BaseSDTrainProcess.py", line 1565, in run self.sd.load_model() File "/workspace/ai-toolkit/toolkit/stable_diffusion_model.py", line 649, in load_model transformer.to(self.quantize_device, dtype=dtype) File "/workspace/ai-toolkit/venv/lib/python3.12/site-packages/diffusers/models/modeling_utils.py", line 1430, in to return super().to(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/ai-toolkit/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1369, in to return self._apply(convert) ^^^^^^^^^^^^^^^^^^^^ File "/workspace/ai-toolkit/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 928, in _apply module._apply(fn) File "/workspace/ai-toolkit/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 928, in _apply module._apply(fn) File "/workspace/ai-toolkit/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 928, in _apply module._apply(fn) File "/workspace/ai-toolkit/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 955, in _apply param_applied = fn(param) ^^^^^^^^^ File "/workspace/ai-toolkit/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1362, in convert raise NotImplementedError( NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device. (venv) root@2b0d3ea49364:/workspace/ai-toolkit#

**absl-py 2.3.1 accelerate 1.12.0 aiofiles 23.2.1 albucore 0.0.16 albumentations 1.4.15 annotated-doc 0.0.4 annotated-types 0.7.0 antlr4-python3-runtime 4.9.3 anyio 4.12.0 attrs 25.4.0 bitsandbytes 0.48.2 brotli 1.2.0 certifi 2025.11.12 charset-normalizer 3.4.4 clean-fid 0.1.35 click 8.3.1 clip-anytorch 2.6.0 coloredlogs 15.0.1 contourpy 1.3.3 controlnet_aux 0.0.10 cycler 0.12.1 dctorch 0.1.2 diffusers 0.36.0.dev0 easy_dwpose 1.0.3 einops 0.8.1 eval_type_backport 0.3.1 fastapi 0.123.8 ffmpy 1.0.0 filelock 3.20.0 flatbuffers 25.9.23 flatten-json 0.1.14 fonttools 4.61.0 fsspec 2025.12.0 ftfy 6.3.1 gitdb 4.0.12 GitPython 3.1.45 gradio 4.42.0 gradio_client 1.3.0 groovy 0.1.2 grpcio 1.76.0 h11 0.16.0 hf_transfer 0.1.9 hf-xet 1.2.0 httpcore 1.0.9 httpx 0.28.1 huggingface-hub 0.36.0 humanfriendly 10.0 idna 3.11 ImageIO 2.37.2 importlib_metadata 8.7.0 importlib_resources 6.5.2 invisible-watermark 0.2.0 Jinja2 3.1.6 jsonmerge 1.9.2 jsonschema 4.25.1 jsonschema-specifications 2025.9.1 k-diffusion 0.1.1.post1 kiwisolver 1.4.9 kornia 0.8.2 kornia_rs 0.1.10 lazy_loader 0.4 loguru 0.7.3 lpips 0.1.4 lycoris_lora 1.8.3 Markdown 3.10 markdown-it-py 4.0.0 MarkupSafe 2.1.5 matplotlib 3.10.1 mdurl 0.1.2 mpmath 1.3.0 networkx 3.6 ninja 1.13.0 numpy 1.26.4 nvidia-cublas-cu12 12.8.4.1 nvidia-cuda-cupti-cu12 12.8.90 nvidia-cuda-nvrtc-cu12 12.8.93 nvidia-cuda-runtime-cu12 12.8.90 nvidia-cudnn-cu12 9.10.2.21 nvidia-cufft-cu12 11.3.3.83 nvidia-cufile-cu12 1.13.1.3 nvidia-curand-cu12 10.3.9.90 nvidia-cusolver-cu12 11.7.3.90 nvidia-cusparse-cu12 12.5.8.93 nvidia-cusparselt-cu12 0.7.1 nvidia-nccl-cu12 2.27.3 nvidia-nvjitlink-cu12 12.8.93 nvidia-nvshmem-cu12 3.3.20 nvidia-nvtx-cu12 12.8.90 omegaconf 2.3.0 onnxruntime-gpu 1.21.1 open_clip_torch 3.2.0 opencv-python 4.11.0.86 opencv-python-headless 4.11.0.86 optimum-quanto 0.2.4 orjson 3.11.4 oyaml 1.0 packaging 25.0 pandas 2.3.3 peft 0.17.0 pillow 10.4.0 pip 24.0 platformdirs 4.5.0 prodigyopt 1.1.2 protobuf 6.33.1 psutil 7.1.3 pydantic 2.12.4 pydantic_core 2.41.5 pydub 0.25.1 Pygments 2.19.2 pyparsing 3.2.5 python-dateutil 2.9.0.post0 python-dotenv 1.2.1 python-multipart 0.0.20 python-slugify 8.0.4 pytorch-fid 0.3.0 pytorch-wavelets 1.3.0 pytz 2025.2 PyWavelets 1.9.0 PyYAML 6.0.3 referencing 0.37.0 regex 2025.11.3 requests 2.32.5 rich 14.2.0 rpds-py 0.30.0 ruff 0.14.8 safehttpx 0.1.7 safetensors 0.7.0 scikit-image 0.25.2 scipy 1.16.3 semantic-version 2.10.0 sentencepiece 0.2.1 sentry-sdk 2.47.0 setuptools 69.5.1 shellingham 1.5.4 six 1.17.0 smmap 5.0.2 starlette 0.50.0 sympy 1.14.0 tensorboard 2.20.0 tensorboard-data-server 0.7.2 text-unidecode 1.3 tifffile 2025.10.16 timm 1.0.22 tokenizers 0.22.1 toml 0.10.2 tomlkit 0.12.0 torch 2.8.0+cu128 torchao 0.10.0 torchaudio 2.8.0+cu128 torchdiffeq 0.2.5 torchsde 0.2.6 torchvision 0.23.0+cu128 tqdm 4.67.1 trampoline 0.1.2 transformers 4.57.3 triton 3.4.0 typer 0.20.0 typer-slim 0.20.0 typing_extensions 4.15.0 typing-inspection 0.4.2 tzdata 2025.2 urllib3 2.5.0 uvicorn 0.38.0 wandb 0.23.1 wcwidth 0.2.14 websockets 12.0 Werkzeug 3.1.4 zipp 3.23.0 (venv) root@2b0d3ea49364:/workspace/ai-toolkit# **

Dec 04 '25 18:12 peepeepeepoopoopoo