diffusion

diffusion copied to clipboard

Forward order differs across ranks: rank 0 is all-gathering 1 parameters while rank 4 is all-gathering -2142209024 parameters

2

Hi, I am trying to use `autoresume` to continue train my failed jobs, but get the following error: ``` File "/opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/_exec_order_utils.py", line 243, in _check_order RuntimeError: Forward order differs across...

viyjy

FID from the mainline code is different from https://github.com/mosaicml/diffusion/tree/ejyuen-patch-1

1

Hi, I found that the current mainline code can generate reasonable FID score for pre-trained models, but generate very high FID score for the model that is pre-trained using this...

viyjy

Error when training with local precomputed features

3

## Bug when local training with LocalDataset Here is my config (without some personal paths), run for mosaicml's diffusion: ``` algorithms: low_precision_groupnorm: attribute: unet precision: amp_fp16 low_precision_layernorm: attribute: unet precision:...

tungdq212

FID score changes a lot during the model training

2

Hi, I found that FID score becomes larger and larger during the Stage 2 (512x512) training of SD-2.0-base, but the loss keeps at the same level, roughly about 0.12~0.13. Any...

viyjy

Hi, after installing everything by following these commands ``` git clone https://github.com/mosaicml/diffusion.git cd diffusion pip install -e . ``` I was trying to run [fid-clip-evaluation.py](https://github.com/mosaicml/diffusion/blob/main/scripts/fid-clip-evaluation.py) but got the following error:...

viyjy

train error during evaluation with 1 GPU and train with multi GPU

2

Hi thanks for this contribution as a small exercise I am training SD2 on the pokemon dataset I precomputed the latents and it starts training on one gpu However at...

segalinc

Grad become NaN

3

When training on my local machine (3090 24Gb) with batch size 12, grad value become NaN after few steps But I don't meet this when training on Google Cloud A100...

tungdq212

fp32

1

Hi, thanks for this great work. I have a question regarding `encode_latents_in_fp16`. If we set `encode_latents_in_fp16=False` to use fp32, do we expect lower performance compared to fp16? I have tried...

viyjy

Error doing composer run.py --config-path yamls/hydra-yamls --config-name SD-2-base-512.yaml

4

composer run.py --config-path yamls/hydra-yamls --config-name SD-2-base-512.yaml [2023-06-13 20:29:52,077][composer.utils.reproducibility][INFO] - Setting seed to 17 Error executing job with overrides: [] Error in call to target 'diffusion.models.models.stable_diffusion_2': TypeError("UNet2DConditionModel.__init__() got an unexpected keyword...

wangmiaowei

ValueError('cannot mmap an empty file')

1

FileExistsError: [Errno 17] File exists: '/000000_shard_access_times' During handling of the above exception, another exception occurred: InstantiationException: Error in call to target 'diffusion.datasets.laion.laion.build_streaming_laion_dataloader': ValueError('cannot mmap an empty file') full_key: dataset.train_dataset ERROR:composer.cli.launcher:Rank...

s5248