YouTokenToMe icon indicating copy to clipboard operation
YouTokenToMe copied to clipboard

Published 20 hours ago • verified publisher icon VKCOM

Support custom tokens

Open 9173860 opened this issue 1 year ago • 0 comments

This PR resolves #65 #44, which implemented the custom tokens feature.

Training BPE is intact, and custom tokens are just added to the model file. The tokens are used during encode/decode phase.

Encoding speed is not affected if custom tokens are not provided. Providing custom tokens will make encoding time about 10% longer, which should be acceptable.

9173860 avatar Sep 15 '22 13:09 9173860