Enrico Shippole

Hi @HoxtonR , I just pushed an update to fix the import errors you had listed. One of the last updates had changed the file directory structure. The repository initially...

@Bachstelze I am curious as to where you are seeing that the Flan-PaLM architecture switched to encoder-decoder? This is taken from the paper Scaling Instruction-Finetuned Language Models. 

> @conceptofmind Sorry, I got confused by this figure from [UL2](https://ai.googleblog.com/2022/10/ul2-20b-open-source-unified-language.html) and concluded that they switched completely to encode-decoder models:  Description: In both decoder-only and encoder-decoder setups, UL2 strikes...

@lucidrains How would you feel about adding Mixture-of-denoisers (ul2 objective) for initially pre-training the added encoder-decoder model? Would this be too off-topic?

@lucidrains , Ok. Always looking forward to your implementations! If anyone deserves a Holiday break it is definitely you! I made an attempt at an encoder-decoder T5 architecture implementation. I...

@lucidrains Oh! I realized after I had left for a walk that I did not connect the output of the encoder-decoder! ```python3 def forward(self, src, tgt, mask = None, context_mask...

@lucidrains I hope you have a great New Year as well!

@lucidrains Sent a small, Thank you / Holiday, gift since I appreciate you answering my questions :smile:. I know you are away now but whenever you are back and free,...

@lucidrains I figured that would be the best way to show gratitude. I appreciate you reviewing the code. I will create a repository with the final T5 code above. @Bachstelze...

@lucidrains I like Nightwalk by Spencer Brown. Made an attempt at PaLM encoder-decoder. I did not think I should open a PR for this so I am just going to...