diffusion icon indicating copy to clipboard operation
diffusion copied to clipboard
verified publisher icon mosaicml

Training verbose logs

Open xenia-kra opened this issue 2 years ago • 3 comments

I'm trying to execute a training process with composer run.py --config-path yamls/hydra-yamls --config-name SD-2-base-256.yaml, after changing the configuration to use a custom data loader. Im getting some generic error AttributeError("'IterableDatasetDict' object has no attribute '_distributed'") from unspecified source. How can I get more details?

xenia-kra avatar Jun 27 '23 10:06 xenia-kra

Can you provide some more information on what your custom dataloader is? It looks like you are having some trouble running in a distributed setting with your dataloader. Torch dataloaders (and our streaming dataset) handle all the distributed magic for you -- I'm not super sure what you are running in this case

mvpatel2000 avatar Jun 28 '23 22:06 mvpatel2000

I was hoping, in general, to solve the problem when I have an ability to debug. Specifically, how do I know what file/line does this error come from?

xenia-kra avatar Jun 29 '23 06:06 xenia-kra

Can you please provide a full trace / logs?

mvpatel2000 avatar Jun 29 '23 17:06 mvpatel2000