Please replace the Tiktoken dependency with a library in pure python (not Rust)

[Emasoft]

Please replace the Tiktoken dependency with a tokenizer library in pure python (not Rust dependencies) for people like me that cannot compile and run Rust binaries on their system (for various reasons: package managers support, company policy, intranet or local machine security, docking containers limitations, vm restrictions, environment virtualization, lack of Rust support in jupyter notebooks remote hosting, etc).

[fredzannarbor]

I would not say "replace" but I endorse providing an alternate pure-python path. at some places it's very difficult to install anything unfamiliar.

[Emasoft]

Yes, you can add the pure python version as a fallback. @jerryjliu ?

[jerryjliu]

@Emasoft thanks for flagging, hopefully will take a look today or tomorrow

[Emasoft]

@jerryjliu Any update on this?

[logan-markewich]

If you are running python3.8, the tokenizer from transformers will be used instead of tiktoken

[dosubot[bot]]

Hi, @Emasoft. I'm Dosu, and I'm helping the LlamaIndex team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, you opened this issue requesting the replacement of the Tiktoken dependency with a pure Python tokenizer library. Some users were unable to compile and run Rust binaries on their systems, so there was a need for an alternate solution. Fredzannarbor supported the idea of providing a pure-python path, and Jerryjliu has been asked to look into it. Logan-markewich mentioned that if running python3.8, the tokenizer from transformers will be used instead of tiktoken.

I wanted to check with you if this issue is still relevant to the latest version of the LlamaIndex repository. If it is, please let the LlamaIndex team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your contribution to the LlamaIndex project! Let us know if you have any further questions or concerns.

[fredzannarbor]

Wow, this is way better than a human would have produced, if at all.

Fred Zimmerman, Publisher Nimble Books LLC The AI Lab for Book-Lovers http://NimbleBooks.com

On Sep 4, 2023 at 12:39:25 PM, dosu-beta[bot] @.***> wrote:

Hi, @Emasoft https://github.com/Emasoft. I'm Dosu, and I'm helping the LlamaIndex team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, you opened this issue requesting the replacement of the Tiktoken dependency with a pure Python tokenizer library. Some users were unable to compile and run Rust binaries on their systems, so there was a need for an alternate solution. Fredzannarbor supported the idea of providing a pure-python path, and Jerryjliu has been asked to look into it. Logan-markewich mentioned that if running python3.8, the tokenizer from transformers will be used instead of tiktoken.

I wanted to check with you if this issue is still relevant to the latest version of the LlamaIndex repository. If it is, please let the LlamaIndex team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your contribution to the LlamaIndex project! Let us know if you have any further questions or concerns.

— Reply to this email directly, view it on GitHub https://github.com/jerryjliu/llama_index/issues/567#issuecomment-1705524576, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAI4TS5VBPFMBQHCZJQNMVTXYX733ANCNFSM6AAAAAAVJYZF2Q . You are receiving this because you commented.Message ID: @.***>

[dosubot[bot]]

Thank you for your response, @fredzannarbor! We appreciate your kind words. We will be closing this issue now. If you have any further questions or concerns, please let us know.