mint icon indicating copy to clipboard operation
mint copied to clipboard

WIP: Feature/OPT

Open dpressel opened this issue 3 years ago • 0 comments

Add support for OPT. It is:

  • a decoder-only model with learned-positional embeddings up to 2k
  • same checkpoint structure as BART without encoder
  • GPT2 byte-level tokenizer with a different vocabulary
  • ReLU activations instead of GeLU

The HuggingFace Tokenizers library cannot use the tokenizer_config.json provided in the repo, so I created a tokenizers.json using the GPT2 one as an example and adding a post-processor to the tokenizer. My tokenizer.json is available from https://www.dropbox.com/s/ut8qj4nynhkq4cd/tokenizer.json?dl=1 which, once saved as tokenizer.json locally, can be used with the opt_completer example.

dpressel avatar Jul 26 '22 20:07 dpressel