ColossalAI
ColossalAI copied to clipboard
[BUG]: diffusion train question
🐛 Describe the bug
/home/chenhq/anaconda3/envs/ldm/lib/python3.9/site-packages/lightning-1.9.0-py3.9.egg/lightning/pytorch/loggers/tensorboard.py:188: UserWarning: Could not log computational graph to TensorBoard: The model.example_input_array attribute is not set or input_array was not given.
rank_zero_warn(
/home/chenhq/anaconda3/envs/ldm/lib/python3.9/site-packages/lightning-1.9.0-py3.9.egg/lightning/pytorch/strategies/ddp.py:437: UserWarning: Error handling mechanism for deadlock detection is uninitialized. Skipping check.
rank_zero_warn("Error handling mechanism for deadlock detection is uninitialized. Skipping check.")
Summoning checkpoint.
run python main.py --logdir ./tmp/ --train --base configs/Teyvat/train_colossalai_teyvat.yaml --ckpt 512-base-ema.ckpt , No output but gpu 100%, Can you help me what the problem is?
Environment
cuda11.2 python3.9 pytorch 1.10.0
Hi, what does it mean by no output? Can you show me the full execution message log? Thanks
logs.txt Wait here, wait for an hour or so, no more output
@Fazziekey , can you take a look at this issue? Thanks
@Fazziekey , can you take a look at this issue? Thanks
ok
@Fazziekey @JThh sorry, my dataset is error, is runing now. I will close this issue, Thanks
@Fazziekey @JThh sorry, my dataset is error, is runing now. I will close this issue, Thanks
Thanks