Connor Leahy
Connor Leahy
Hi there. These models were trained primarily on English text, so I have no idea how good or bad it is for other languages. By default, it should be able...
Thanks for the interesting comments everyone. I think under the line what it comes down to is that the model was trained primarily on english text, so it naturally struggles...
Google actually has a system you can use to build a BPE encoder: https://github.com/google/sentencepiece It's not exactly the same as OpenAI's, so you'd need to adapt encoder.py to use the...
I'm currently investigating these quirks in fact! I'll talk about this more if my hunches are confirmed, but it might take a while.
Unfortunately not much interesting to report so far. I've tried several tweaks, to no avail. I'll continue experimenting for a while before I compile my results.
One of the main suspects for my model's worse performance is weight initialization. I just pushed some new code that should allow for different kinds of weight initialization and make...
This repo is pretty sparse and I don't have any plans currently on working on it more, so I don't have any kind of fancy support for custom encoders. If...
I'm not familiar with the absl library, but it seems the flags are not passed to the main function, no?
Hi there! There are scripts in the dataset directory that roughly show what needs to be done, but it's all super rough I'm sorry, I still haven't found the time...
Thanks, I'm hoping to find some time this week to polish some code and write a better tutorial, we'll see. Looking forward to your PR! 1. To train the model,...