rutransform
rutransform copied to clipboard
Sberbank 🤗 repository name changed & skip_special_tokens removal
- On project set-up, when trying to load models from sberbank huggingface repos, transformer lib throws unknown repo name error. It happens because of sberbank repo name has been changed from
sberbank-ai
toai-forever
. For future-proof it's preferable to store model checkpoints in author's own 🤗 repository. - When using
add-sent
transformation, which hasgtp-3
model by default, code throws error, that indicatesskip_special_tokens
is not in the parameters list ofgtp-3
model. So it is suggested to be removed orfor
-loop to be refactored