Pretrained text encoder

[ethancohen123]

Is it possible to use and train dalle with an external ( frozen) text encoder ( as those available in hugging face) ?

[ethancohen123]

Anyone has an idea about this ? @lucidrains

[kingnobro]

Hi. If you want to use pretrained language model, you are actually using the text embedding of that model.

1. At first, you can load and save the text embedding layer weight of pretrained models like CLIP and BERT.
2. Then, you need to replace the text_emb in DALLE __init__ function. Now, instead of using nn.Embedding to create new text embedding, you can use torch.load to load pretrained weight saved in step 1.

Example: link