Langchain-Chatchat icon indicating copy to clipboard operation
Langchain-Chatchat copied to clipboard

Published 20 hours ago • verified publisher icon chatchat-space

Could not import sentence_transformers python package. Please install it with `pip install sentence_transformers`.

Open wengieeee opened this issue 1 year ago • 4 comments

问题描述 / Problem Description 执行报错，无法上传知识库 ERROR 2023-05-18 14:08:12,342-1d: Could not import sentence_transformers python package. Please install it with pip install sentence_transformers.

复现问题的步骤 / Steps to Reproduce python webui.py

实际结果 / Actual Result (/mnt/data/deploy/venv_lc2) [root@OneThingEdgeCloud langchain-ChatGLM]# python webui.py INFO 2023-05-18 14:07:41,687-1d: loading model config llm device: cuda embedding device: cuda dir: /mnt/data/code/langchain-ChatGLM flagging username: 054012f1800641b593a81535730d2626

/mnt/data/deploy/venv_lc2/lib/python3.10/site-packages/paddle/fluid/core.py:219: UserWarning: Load /lib64/libgomp.so.1 failed warnings.warn("Load {} failed".format(dso_absolute_path)) Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a revision is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:07<00:00, 1.05it/s] ERROR 2023-05-18 14:08:12,342-1d: Could not import sentence_transformers python package. Please install it with pip install sentence_transformers. Running on local URL: http://0.0.0.0:7860 Running on public URL: https://7e8ac414467778dcf4.gradio.live

This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces

环境信息 / Environment Information

  • langchain-ChatGLM 版本/commit 号：master
  • 是否使用 Docker 部署（是/否）：否
  • 使用的模型（ChatGLM-6B / ClueAI/ChatYuan-large-v2 等）：ChatGLM-6B
  • 使用的 Embedding 模型（GanymedeNil/text2vec-large-chinese 等）：GanymedeNil/text2vec-large-chinese
  • 操作系统及版本 / Operating system and version: centOS 7.9
  • Python 版本 / Python version: 3.10
  • 其他相关环境信息 / Other relevant environment information: gcc 4.8 ,如果使用gcc 10 会报段错误

附加信息 / Additional Information gradio 可以正常启动，但是上传的上传知识库并不生效，没有生成 vector_store 文件夹

wengieeee avatar May 18 '23 06:05 wengieeee

问题里的pip install就是解决办法

jixi_chen @.***>于2023年5月18日 周四14:14写道：

问题描述 / Problem Description 执行报错，无法上传知识库 ERROR 2023-05-18 14:08:12,342-1d: Could not import sentence_transformers python package. Please install it with pip install sentence_transformers.

复现问题的步骤 / Steps to Reproduce python webui.py

实际结果 / Actual Result (/mnt/data/deploy/venv_lc2) @.*** langchain-ChatGLM]# python webui.py INFO 2023-05-18 14:07:41,687-1d: loading model config llm device: cuda embedding device: cuda dir: /mnt/data/code/langchain-ChatGLM flagging username: 054012f1800641b593a81535730d2626

/mnt/data/deploy/venv_lc2/lib/python3.10/site-packages/paddle/fluid/core.py:219: UserWarning: Load /lib64/libgomp.so.1 failed warnings.warn("Load {} failed".format(dso_absolute_path)) Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a revision is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:07<00:00, 1.05it/s] ERROR 2023-05-18 14:08:12,342-1d: Could not import sentence_transformers python package. Please install it with pip install sentence_transformers. Running on local URL: http://0.0.0.0:7860 Running on public URL: https://7e8ac414467778dcf4.gradio.live

This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces

环境信息 / Environment Information

  • langchain-ChatGLM 版本/commit 号：master
  • 是否使用 Docker 部署（是/否）：否
  • 使用的模型（ChatGLM-6B / ClueAI/ChatYuan-large-v2 等）：ChatGLM-6B
  • 使用的 Embedding 模型（GanymedeNil/text2vec-large-chinese 等）：GanymedeNil/text2vec-large-chinese
  • 操作系统及版本 / Operating system and version: centOS 7.9
  • Python 版本 / Python version: 3.10
  • 其他相关环境信息 / Other relevant environment information: gcc 4.8 ,如果使用gcc 10 会报段错误

附加信息 / Additional Information gradio 可以正常启动，但是上传的上传知识库并不生效，没有生成 vector_store 文件夹

— Reply to this email directly, view it on GitHub https://github.com/imClumsyPanda/langchain-ChatGLM/issues/400, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLH5ET655V5UI44MGMNPODXGW433ANCNFSM6AAAAAAYGA3MAM . You are receiving this because you are subscribed to this thread.Message ID: @.***>

imClumsyPanda avatar May 18 '23 06:05 imClumsyPanda

好像不行的。 实际执行 pip install sentence_transformers 后 pip list |grep sentence_transformers 还是找不到这包，但是有sentence-transformers 这个包。


(/mnt/data/deploy/venv_lc2) [root@OneThingEdgeCloud f1479560d1648f0b26a798e569408b6788dcc4cf]# pip install sentence_transformers
Requirement already satisfied: sentence_transformers in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (2.2.2)
Requirement already satisfied: sentencepiece in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from sentence_transformers) (0.1.99)
Requirement already satisfied: numpy in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from sentence_transformers) (1.23.5)
Requirement already satisfied: huggingface-hub>=0.4.0 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from sentence_transformers) (0.14.1)
Requirement already satisfied: tqdm in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from sentence_transformers) (4.65.0)
Requirement already satisfied: torchvision in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from sentence_transformers) (0.15.2)
Requirement already satisfied: torch>=1.6.0 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from sentence_transformers) (2.0.1)
Requirement already satisfied: transformers<5.0.0,>=4.6.0 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from sentence_transformers) (4.27.1)
Requirement already satisfied: nltk in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from sentence_transformers) (3.8.1)
Requirement already satisfied: scikit-learn in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from sentence_transformers) (1.2.2)
Requirement already satisfied: scipy in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from sentence_transformers) (1.10.1)
Requirement already satisfied: packaging>=20.9 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence_transformers) (23.1)
Requirement already satisfied: filelock in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence_transformers) (3.12.0)
Requirement already satisfied: fsspec in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence_transformers) (2023.5.0)
Requirement already satisfied: requests in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence_transformers) (2.30.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence_transformers) (4.5.0)
Requirement already satisfied: pyyaml>=5.1 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence_transformers) (6.0)
Requirement already satisfied: nvidia-cusolver-cu11==11.4.0.1 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from torch>=1.6.0->sentence_transformers) (11.4.0.1)
Requirement already satisfied: nvidia-nccl-cu11==2.14.3 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from torch>=1.6.0->sentence_transformers) (2.14.3)
Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from torch>=1.6.0->sentence_transformers) (8.5.0.96)
Requirement already satisfied: nvidia-nvtx-cu11==11.7.91 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from torch>=1.6.0->sentence_transformers) (11.7.91)
Requirement already satisfied: networkx in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from torch>=1.6.0->sentence_transformers) (3.1)
Requirement already satisfied: sympy in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from torch>=1.6.0->sentence_transformers) (1.12)
Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from torch>=1.6.0->sentence_transformers) (11.7.99)
Requirement already satisfied: nvidia-cufft-cu11==10.9.0.58 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from torch>=1.6.0->sentence_transformers) (10.9.0.58)
Requirement already satisfied: nvidia-cuda-cupti-cu11==11.7.101 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from torch>=1.6.0->sentence_transformers) (11.7.101)
Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from torch>=1.6.0->sentence_transformers) (11.7.99)
Requirement already satisfied: jinja2 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from torch>=1.6.0->sentence_transformers) (3.1.2)
Requirement already satisfied: nvidia-curand-cu11==10.2.10.91 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from torch>=1.6.0->sentence_transformers) (10.2.10.91)
Requirement already satisfied: triton==2.0.0 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from torch>=1.6.0->sentence_transformers) (2.0.0)
Requirement already satisfied: nvidia-cusparse-cu11==11.7.4.91 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from torch>=1.6.0->sentence_transformers) (11.7.4.91)
Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from torch>=1.6.0->sentence_transformers) (11.10.3.66)
Requirement already satisfied: setuptools in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.6.0->sentence_transformers) (66.0.0)
Requirement already satisfied: wheel in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.6.0->sentence_transformers) (0.38.4)
Requirement already satisfied: cmake in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from triton==2.0.0->torch>=1.6.0->sentence_transformers) (3.26.3)
Requirement already satisfied: lit in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from triton==2.0.0->torch>=1.6.0->sentence_transformers) (16.0.3)
Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from transformers<5.0.0,>=4.6.0->sentence_transformers) (0.13.3)
Requirement already satisfied: regex!=2019.12.17 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from transformers<5.0.0,>=4.6.0->sentence_transformers) (2023.5.5)
Requirement already satisfied: joblib in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from nltk->sentence_transformers) (1.2.0)
Requirement already satisfied: click in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from nltk->sentence_transformers) (8.1.3)
Requirement already satisfied: threadpoolctl>=2.0.0 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from scikit-learn->sentence_transformers) (3.1.0)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from torchvision->sentence_transformers) (9.5.0)
Requirement already satisfied: MarkupSafe>=2.0 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from jinja2->torch>=1.6.0->sentence_transformers) (2.1.2)
Requirement already satisfied: urllib3<3,>=1.21.1 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from requests->huggingface-hub>=0.4.0->sentence_transformers) (2.0.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from requests->huggingface-hub>=0.4.0->sentence_transformers) (3.1.0)
Requirement already satisfied: idna<4,>=2.5 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from requests->huggingface-hub>=0.4.0->sentence_transformers) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from requests->huggingface-hub>=0.4.0->sentence_transformers) (2023.5.7)
Requirement already satisfied: mpmath>=0.19 in /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages (from sympy->torch>=1.6.0->sentence_transformers) (1.3.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
(/mnt/data/deploy/venv_lc2) [root@OneThingEdgeCloud f1479560d1648f0b26a798e569408b6788dcc4cf]# pip list |grep sentence_transformers
(/mnt/data/deploy/venv_lc2) [root@OneThingEdgeCloud f1479560d1648f0b26a798e569408b6788dcc4cf]#

问题里的pip install就是解决办法 jixi_chen @.>于2023年5月18日 周四14:14写道： … 问题描述 / Problem Description 执行报错，无法上传知识库 ERROR 2023-05-18 14:08:12,342-1d: Could not import sentence_transformers python package. Please install it with pip install sentence_transformers. 复现问题的步骤 / Steps to Reproduce python webui.py 实际结果 / Actual Result (/mnt/data/deploy/venv_lc2) @. langchain-ChatGLM]# python webui.py INFO 2023-05-18 14:07:41,687-1d: loading model config llm device: cuda embedding device: cuda dir: /mnt/data/code/langchain-ChatGLM flagging username: 054012f1800641b593a81535730d2626 /mnt/data/deploy/venv_lc2/lib/python3.10/site-packages/paddle/fluid/core.py:219: UserWarning: Load /lib64/libgomp.so.1 failed warnings.warn("Load {} failed".format(dso_absolute_path)) Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a revision is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:07<00:00, 1.05it/s] ERROR 2023-05-18 14:08:12,342-1d: Could not import sentence_transformers python package. Please install it with pip install sentence_transformers. Running on local URL: http://0.0.0.0:7860 Running on public URL: https://7e8ac414467778dcf4.gradio.live This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces 环境信息 / Environment Information - langchain-ChatGLM 版本/commit 号：master - 是否使用 Docker 部署（是/否）：否 - 使用的模型（ChatGLM-6B / ClueAI/ChatYuan-large-v2 等）：ChatGLM-6B - 使用的 Embedding 模型（GanymedeNil/text2vec-large-chinese 等）：GanymedeNil/text2vec-large-chinese - 操作系统及版本 / Operating system and version: centOS 7.9 - Python 版本 / Python version: 3.10 - 其他相关环境信息 / Other relevant environment information: gcc 4.8 ,如果使用gcc 10 会报段错误 附加信息 / Additional Information gradio 可以正常启动，但是上传的上传知识库并不生效，没有生成 vector_store 文件夹 — Reply to this email directly, view it on GitHub <#400>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLH5ET655V5UI44MGMNPODXGW433ANCNFSM6AAAAAAYGA3MAM . You are receiving this because you are subscribed to this thread.Message ID: @.***>

wengieeee avatar May 18 '23 06:05 wengieeee

Having the same issue.

alelom avatar May 18 '23 14:05 alelom

Having the same issue.

I found after troubleshooting that the reason was that importing the Embeddings model from HuggingFaceEmbeddings failed. After further investigation, I found that my actual error was: OSError: dlopen: cannot load any more object with static TLS.

This problem was solved by upgrading glibc to 2.31.

wengieeee avatar May 18 '23 16:05 wengieeee

我也存在相同的问题 运行webui.py Loading /data/chatglm2-6b... Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:14<00:00, 2.04s/it] Loaded the model in 42.32 seconds. ERROR 2023-07-19 02:01:37,935-1d: Could not import sentence_transformers python package. Please install it with pip install sentence_transformers. INFO 2023-07-19 02:01:37,935-1d: 模型未成功加载，请到页面左上角"模型配置"选项卡中重新选择后点击"加载模型"按钮 Running on local URL: http://0.0.0.0:7860 运行cli_demo.py时报出来这个 INFO 2023-07-19 02:18:38,896-1d: Note: detected 128 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable. INFO 2023-07-19 02:18:38,896-1d: Note: NumExpr detected 128 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. INFO 2023-07-19 02:18:38,896-1d: NumExpr defaulting to 8 threads. Loading /data/chatglm2-6b... Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:13<00:00, 1.94s/it] Loaded the model in 42.01 seconds. INFO 2023-07-19 02:19:24,923-1d: Load pretrained SentenceTransformer: /data/textbase WARNING 2023-07-19 02:19:24,923-1d: No sentence-transformers model found with name /data/textbase. Creating a new one with MEAN pooling. Input your local knowledge file path 请输入本地知识文件路径：/data/textbase 加载文件: 0it [00:00, ?it/s]INFO 2023-07-19 02:20:08,247-1d: Reading document from string ... INFO 2023-07-19 02:20:08,248-1d: Reading document ... 加载文件: 1it [00:02, 2.16s/it]ERROR 2023-07-19 02:20:08,404-1d: Json schema does not match the Unstructured schema WARNING 2023-07-19 02:20:08,418-1d: Could not detect the filetype from application/octet-stream MIME type. ERROR 2023-07-19 02:20:08,419-1d: Invalid file /data/textbase/pytorch_model.bin. The FileType.ZIP file type is not supported in partition. ERROR 2023-07-19 02:20:08,419-1d: Json schema does not match the Unstructured schema ERROR 2023-07-19 02:20:08,427-1d: Json schema does not match the Unstructured schema ERROR 2023-07-19 02:20:08,428-1d: Json schema does not match the Unstructured schema 加载文件: 9it [00:03, 2.47it/s] INFO 2023-07-19 02:20:09,890-1d: 以下文件未能成功加载： INFO 2023-07-19 02:20:09,890-1d: config.json

INFO 2023-07-19 02:20:09,890-1d: pytorch_model.bin

INFO 2023-07-19 02:20:09,890-1d: special_tokens_map.json

INFO 2023-07-19 02:20:09,890-1d: tokenizer.json

INFO 2023-07-19 02:20:09,890-1d: tokenizer_config.json

INFO 2023-07-19 02:20:09,891-1d: 文件加载完毕，正在生成向量库 请问怎么解决呀

czq001 avatar Jul 19 '23 02:07 czq001

Could not import sentence_transformers python package. Please install it with pip install sentence_transformers.这个问题有解决了吗？我是Windows系统使用conda部署的环境

jszhangyili avatar Jul 19 '23 05:07 jszhangyili

@czq001 @jszhangyili 按照报错信息 执行pip install sentence-transformers

imClumsyPanda avatar Jul 19 '23 05:07 imClumsyPanda

执行pip install sentence-transformers后也没有解决这个问题。在读embedding模型那句代码前，无法import sentence-transformers

VSRacer avatar Jul 19 '23 06:07 VSRacer

root@localhost:/workspace/langchain-ChatGLM-master# pip install sentence-transformers Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple/, https://pypi.ngc.nvidia.com Requirement already satisfied: sentence-transformers in /usr/local/lib/python3.8/dist-packages (2.2.2) Requirement already satisfied: transformers<5.0.0,>=4.6.0 in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (4.29.1) Requirement already satisfied: tqdm in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (4.65.0) Requirement already satisfied: torch>=1.6.0 in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (2.0.0a0+1767026) Requirement already satisfied: torchvision in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (0.15.0a0) Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (1.23.5) Requirement already satisfied: scikit-learn in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (1.2.0) Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (1.10.1) Requirement already satisfied: nltk in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (3.8.1) Requirement already satisfied: sentencepiece in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (0.1.99) Requirement already satisfied: huggingface-hub>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (0.16.4) Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (3.10.0) Requirement already satisfied: fsspec in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (2023.1.0) Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (2.28.2) Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (6.0) Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (4.5.0) Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (23.0) Requirement already satisfied: sympy in /usr/local/lib/python3.8/dist-packages (from torch>=1.6.0->sentence-transformers) (1.11.1) Requirement already satisfied: networkx in /usr/local/lib/python3.8/dist-packages (from torch>=1.6.0->sentence-transformers) (3.1) Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers) (2022.10.31) Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.8/dist-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers) (0.13.3) Requirement already satisfied: click in /usr/local/lib/python3.8/dist-packages (from nltk->sentence-transformers) (8.1.3) Requirement already satisfied: joblib in /usr/local/lib/python3.8/dist-packages (from nltk->sentence-transformers) (1.2.0) Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from scikit-learn->sentence-transformers) (3.1.0) Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.8/dist-packages (from torchvision->sentence-transformers) (9.2.0) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers) (2.1.0) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers) (3.4) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers) (1.26.14) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers) (2022.12.7) Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.8/dist-packages (from sympy->torch>=1.6.0->sentence-transformers) (1.3.0) sentence-transformers是安装了的

czq001 avatar Jul 19 '23 06:07 czq001

报错是卡在了local_doc_qa.py中的 self.embeddings = HuggingFaceEmbeddings(model_name=embedding_model_dict[embedding_model], model_kwargs={'device': embedding_device}) 这句代码上面

VSRacer avatar Jul 19 '23 06:07 VSRacer

这个问题我解决的方式是删除你电脑力pip 缓存中的sentence-transformers，然后再安装就可以了

| | 张宜力 | | @.*** | ---- Replied Message ---- | From | @.> | | Date | 7/19/2023 14:31 | | To | @.> | | Cc | @.> , @.> | | Subject | Re: [imClumsyPanda/langchain-ChatGLM] Could not import sentence_transformers python package. Please install it with pip install sentence_transformers. (Issue #400) |

执行pip install sentence-transformers后也没有解决这个问题。在读embedding模型那句代码前，无法import sentence-transformers

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

jszhangyili avatar Jul 20 '23 03:07 jszhangyili

这个问题我解决的方式是删除电脑中pip 缓存中的sentence-transformers，然后再安装就可以了

| | 张宜力 | | @.*** | ---- Replied Message ---- | From | @.> | | Date | 7/19/2023 14:32 | | To | @.> | | Cc | @.> , @.> | | Subject | Re: [imClumsyPanda/langchain-ChatGLM] Could not import sentence_transformers python package. Please install it with pip install sentence_transformers. (Issue #400) |

@.**:/workspace/langchain-ChatGLM-master# pip install sentence-transformers Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple/, https://pypi.ngc.nvidia.com Requirement already satisfied: sentence-transformers in /usr/local/lib/python3.8/dist-packages (2.2.2) Requirement already satisfied: transformers<5.0.0,>=4.6.0 in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (4.29.1) Requirement already satisfied: tqdm in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (4.65.0) Requirement already satisfied: torch>=1.6.0 in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (2.0.0a0+1767026) Requirement already satisfied: torchvision in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (0.15.0a0) Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (1.23.5) Requirement already satisfied: scikit-learn in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (1.2.0) Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (1.10.1) Requirement already satisfied: nltk in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (3.8.1) Requirement already satisfied: sentencepiece in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (0.1.99) Requirement already satisfied: huggingface-hub>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (0.16.4) Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (3.10.0) Requirement already satisfied: fsspec in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (2023.1.0) Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (2.28.2) Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (6.0) Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (4.5.0) Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (23.0) Requirement already satisfied: sympy in /usr/local/lib/python3.8/dist-packages (from torch>=1.6.0->sentence-transformers) (1.11.1) Requirement already satisfied: networkx in /usr/local/lib/python3.8/dist-packages (from torch>=1.6.0->sentence-transformers) (3.1) Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers) (2022.10.31) Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.8/dist-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers) (0.13.3) Requirement already satisfied: click in /usr/local/lib/python3.8/dist-packages (from nltk->sentence-transformers) (8.1.3) Requirement already satisfied: joblib in /usr/local/lib/python3.8/dist-packages (from nltk->sentence-transformers) (1.2.0) Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from scikit-learn->sentence-transformers) (3.1.0) Requirement already satisfied: pillow!=8.3.,>=5.3.0 in /usr/local/lib/python3.8/dist-packages (from torchvision->sentence-transformers) (9.2.0) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers) (2.1.0) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers) (3.4) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers) (1.26.14) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers) (2022.12.7) Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.8/dist-packages (from sympy->torch>=1.6.0->sentence-transformers) (1.3.0) sentence-transformers是安装了的

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

jszhangyili avatar Jul 20 '23 03:07 jszhangyili

同遇到了类似的问题，并且执行pip install sentence-transformers以及sklearn后也没有解决这个问题 Traceback (most recent call last): File "/home/users/xxxx/anaconda3/envs/chatglm/lib/python3.8/site-packages/sklearn/__check_build/init.py", line 45, in from ._check_build import check_build # noqa ImportError: dlopen: cannot load any more object with static TLS

youxiho1 avatar Jul 21 '23 09:07 youxiho1

遇到同样问题，还没有解决

SnoopyGeeek avatar Aug 17 '23 16:08 SnoopyGeeek

我看到有这个错误 ImportError: accelerate>=0.20.3 is required for a normal functioning of this module, but found accelerate==0.18.0.

但是pip install accelerate>=0.20.3不管用 先pip uninstall accelerate再pip install accelerate>=0.20.3可以

没再报找不到sentence_transformers

Valdanitooooo avatar Aug 24 '23 08:08 Valdanitooooo

I fixed the problem by updating torch to version 2.1.1 P P4SN$A%54Y70ACTR9 PQI

948726484 avatar Dec 01 '23 06:12 948726484

在自定义大模型以及使用HuggingFaceEmbedding时，使用了transformers库以及sentence-transformers库，如果安装了这两个库还分别报错：AttributeError: 'ChatGLMTokenizer' object has no attribute 'sp_tokenizer'或者“could not import sentence_transformers python package. pip intall sentence-transformers”。第一个报错往往是transformers版本过高。使用pip install transformers==4.27.1亲测可以解决第一个问题。第二个问题可能是两个包版本冲突了。可以降低一下sentence-transformers版本，pip install sentence-transformers==2.2.2可以解决上面问题。

123sghn avatar Jul 13 '24 02:07 123sghn