tiktoken python package

[jibinmathew69]

Even though the tiktoken python package is shown in the pyproject.toml, it doesnt seem to have been installed, and I got the error on summarization with map-reduce,

ValueError: Could not import tiktoken python package. This is needed in order to calculate get_num_tokens. Please it install it with pip install tiktoken.

Upon pip installation of the package it worked.

[outday29]

Looking at the [tool.poetry.extras] section in the pyproject.toml file, it seems tiktoken is part of the all additional packages, so you will have to do pip install "langchain[all]" to install it. Note that doing so will install all other addtional packages as well.

[joybro]

Shouldn't tool.poetry.extras.open include tiktoken then?

-openai = ["openai"]
+openai = ["openai", "tiktoken"]

[markthepixel]

same issue ValueError: Could not import tiktoken python package. This is needed in order to calculate max_tokens_for_prompt. Please install it with 'pip install tiktoken'.

[fxrcode]

still failing for summarize chain:

File "/opt/managed-artifacts/python/3.8/lib/python3.8/site-packages/langchain/chains/base.py", line 140, in __call__
 raise e
 File "/opt/managed-artifacts/python/3.8/lib/python3.8/site-packages/langchain/chains/base.py", line 134, in __call__
 self._call(inputs, run_manager=run_manager)
 File "/opt/managed-artifacts/python/3.8/lib/python3.8/site-packages/langchain/chains/combine_documents/base.py", line 84, in _call
 output, extra_return_dict = self.combine_docs(
 File "/opt/managed-artifacts/python/3.8/lib/python3.8/site-packages/langchain/chains/combine_documents/map_reduce.py", line 149, in combine_docs
 return self._process_results(
 File "/opt/managed-artifacts/python/3.8/lib/python3.8/site-packages/langchain/chains/combine_documents/map_reduce.py", line 183, in _process_results
 num_tokens = length_func(result_docs, **kwargs)
 File "/opt/managed-artifacts/python/3.8/lib/python3.8/site-packages/langchain/chains/combine_documents/stuff.py", line 79, in prompt_length
 return self.llm_chain.llm.get_num_tokens(prompt)
 File "/opt/managed-artifacts/python/3.8/lib/python3.8/site-packages/langchain/chat_models/openai.py", line 366, in get_num_tokens
 raise ValueError(
ValueError: Could not import tiktoken python package. This is needed in order to calculate get_num_tokens. Please install it with `pip install tiktoken`.

[Telsho]

I ran into the same issue, the solution was to change the python interpreter in VSCode to the poetry one.

https://code.visualstudio.com/docs/python/environments#_select-and-activate-an-environment

ref: https://stackoverflow.com/questions/71229685/packages-installed-with-poetry-fail-to-import

[jayarerita]

I ran into the exact same situation as @jibinmathew69 using summarization. Solved with a separate call to pip install tiktoken. Given the wide usage of the load_summarize_chain utility it would be nice if it were installed without the [all] option.

[dosubot[bot]]

Hi, @jibinmathew69,

I'm helping the LangChain team manage their backlog and am marking this issue as stale. From what I understand, you raised this issue due to a ValueError during a map-reduce operation caused by the tiktoken python package not being installed. The problem was resolved by manually installing the package using pip install tiktoken. There were also other proposed solutions from different users, such as modifying the pyproject.toml file, changing the Python interpreter in VSCode, and suggesting to install tiktoken without the [all] option.

Could you please confirm if this issue is still relevant to the latest version of the LangChain repository? If it is, please let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you for your understanding and cooperation.