gpt-fast icon indicating copy to clipboard operation
gpt-fast copied to clipboard

Published 20 hours ago • verified publisher icon pytorch-labs

Making TokenizerInterface more usable for the user's code.

Open Artyom17 opened this issue 1 year ago • 0 comments

Adding id_to_piece, piece_to_id and is_special_token functionality to TokenizerInterface and the corresponding implementations. Thus, the interface can be used by user's code to encode/decode single tokens. These new functions are not directly used by the gpt-fast code.

Artyom17 avatar Apr 30 '24 21:04 Artyom17