[pyproject.toml] add `tiktoken` to `tool.poetry.extras.openai`

[ofey404]

System Info

langchain[openai]==0.0.165 Ubuntu 22.04.2 LTS (Jammy Jellyfish) python 3.10.6

Who can help?

@vowelparrot

Information

• [ ] The official example notebooks/scripts
• [X] My own modified scripts

Related Components

• [ ] LLMs/Chat Models
• [ ] Embedding Models
• [ ] Prompts / Prompt Templates / Prompt Selectors
• [ ] Output Parsers
• [ ] Document Loaders
• [ ] Vector Stores / Retrievers
• [ ] Memory
• [ ] Agents / Agent Executors
• [X] Tools / Toolkits
• [ ] Chains
• [ ] Callbacks/Tracing
• [ ] Async

Reproduction

The OpenAI component requires tiktoken package, but if we install like below, the tiktoken package is not found.

langchain[openai]==0.0.165

It's natural to add tiktoken, since there is a dependency in the pyproject.toml file.

https://github.com/hwchase17/langchain/blob/46b100ea630b5d1d7fedd6a32d5eb9ecbadeb401/pyproject.toml#L35-L36

Besides, the missing of tiktoken would cause an issue under some dependency pinning tool, like bazel or jazzband/pip-tools

Traceback (most recent call last):
  File "/home/ofey/.cache/bazel/_bazel_ofey/90bb890b04415910673f256b166d6c9b/sandbox/linux-sandbox/15/execroot/walking_shadows/bazel-out/k8-fastbuild/bin/src/backend/services/world/internal/memory/test/test_test.runfiles/pip_langchain/site-packages/langchain/embeddings/openai.py", line 186, in _get_len_safe_embeddings
    import tiktoken
ModuleNotFoundError: No module named 'tiktoken'

...

  File "/home/ofey/.cache/bazel/_bazel_ofey/90bb890b04415910673f256b166d6c9b/sandbox/linux-sandbox/15/execroot/walking_shadows/bazel-out/k8-fastbuild/bin/src/backend/services/world/internal/memory/test/test_test.runfiles/pip_langchain/site-packages/langchain/embeddings/openai.py", line 240, in _get_len_safe_embeddings
    raise ValueError(
ValueError: Could not import tiktoken python package. This is needed in order to for OpenAIEmbeddings. Please install it with `pip install tiktoken`.

Expected behavior

Add a dependency in pyproject.toml

[tool.poetry.extras]
...
openai = ["openai", "tiktoken"]

Actually I'm using langchain with bazel, this is my project: ofey404/WalkingShadows