python-bpe icon indicating copy to clipboard operation
python-bpe copied to clipboard

Published 20 hours ago • verified publisher icon soaxelbrooke

How to generate a GPT-2 (openAI) encoder.json ?

Open Skylixia opened this issue 3 years ago • 1 comments

How to use this library to generate an encoder.json file such as the ones used for GPT-2 model ?

Skylixia avatar Jul 30 '20 18:07 Skylixia

I'm not knowledgeable of the encoder.json that GPT-2 uses, or the tokenization it uses, could you link me to relevant documentation?

soaxelbrooke avatar Aug 07 '20 00:08 soaxelbrooke