Could we update Tokenizers version dependency to ^0.19 from ^0.15?

[rishabgit]

What happened?

Not a bug but rather a request -

Currently, fastembed requires an old version (>=0.15,<0.16) which also conflicts with other libraries such as cohere's python SDK

Latest tokenizers version is 0.19.1

What Python version are you on? e.g. python --version

Python 3.10.14, Poetry

Version

0.2.7 (Latest)

What os are you seeing the problem on?

Linux

Relevant stack traces and/or logs

No response

[joein]

Hey @rishabgit , yeah, sure, #244

[rishabgit]

Thanks! @joein