aitextgen icon indicating copy to clipboard operation
aitextgen copied to clipboard
verified publisher icon minimaxir

Add support for loading serialized custom tokenizers

Open minimaxir opened this issue 5 years ago • 0 comments

New feature in tokenizers and transformers. This is the default state for new tokenizers, so support must be added.

Will likely include in #40

minimaxir avatar Jul 05 '20 18:07 minimaxir