mint
mint copied to clipboard
WIP: Feature/OPT
Add support for OPT. It is:
- a decoder-only model with learned-positional embeddings up to 2k
- same checkpoint structure as BART without encoder
- GPT2 byte-level tokenizer with a different vocabulary
- ReLU activations instead of GeLU
The HuggingFace Tokenizers library cannot use the tokenizer_config.json provided in the repo, so I created a tokenizers.json using the GPT2 one as an example and adding a post-processor to the tokenizer. My tokenizer.json is available from https://www.dropbox.com/s/ut8qj4nynhkq4cd/tokenizer.json?dl=1 which, once saved as tokenizer.json locally, can be used with the opt_completer example.