Clay Mullis

Edit: Here, I used Weights & Biases to create a report. This link has all the images generated (every 100th iteration) for 27,831 total iterations Edit: this one should work...

Hm, does this work? @tommy19970714 ? https://wandb.ai/afiaka87/dalle-pytorch-openai-samples/reports/Training-on-OpenAI-DALL-E-Generated-Images--Vmlldzo1MTk2MjQ?accessToken=89u5e10c2oag5mlv46xm2sz6orkyqdlwjrsj8vd95oz8ke3ez6v8v2fh07klk6j1

> Hi @afiaka87, Amazing results! Can you share more details about your configurations? such as the dataset, learning rate, lr scheduler, number of text and image (8192, right?) tokens? Thanks....

@tommy19970714 I did a hyperparameter sweep with weights and biases. Forty Eight 1200 iteration runs of dalle-pytorch while varying Learning Rate, Depth and Heads, (minimizing the total loss at the...

Most important thing to note here is that the learning rate actually needs to go up to about 0.0005 when dealing with ~26-32 depth

I've done a much longer training session on that same dataset here: https://github.com/lucidrains/DALLE-pytorch/issues/86

@aviallon You will need to train dalle-pytorch on a large dataset yourself, or wait until a pretrained model is released. Edit: There are two projects you can use for now...

@lucidrains is this supported? I never train my own VAE but it shouldn't be too tough to allow for VQGANs trained via the taming-transformers method, right?

Ive been working on this a bit - kobiso and others are fond of using hugging_faces tokenizers library. I think drop in support for the parameter `--bpe_path=""` would be nice...

> @afiaka87 yup I can work on that too! About to push a wip up actually