aitextgen
aitextgen copied to clipboard
Add support for loading serialized custom tokenizers
New feature in tokenizers and transformers. This is the default state for new tokenizers, so support must be added.
Will likely include in #40