ChatGLM2-6B
ChatGLM2-6B copied to clipboard
THUDM
[BUG/Help] openai_api.py中缺少了usage，导致langchain执行失败
Is there an existing issue for this?
- [X] I have searched the existing issues
Current Behavior
openai 的api中没有封装usage，导致langchain执行chat模型失败
raise e
File "/home/langchain-trilium/.venv/lib/python3.10/site-packages/langchain/llms/base.py", line 163, in _generate_helper
self._generate(
File "/home/langchain-trilium/.venv/lib/python3.10/site-packages/langchain/llms/openai.py", line 824, in _generate
"token_usage": full_response["usage"],
KeyError: 'usage'
Expected Behavior
No response
Steps To Reproduce
增加了一行后修复了 usage: Optional[int] = 0 class ChatCompletionResponse(BaseModel): model: str object: Literal["chat.completion", "chat.completion.chunk"] choices: List[Union[ChatCompletionResponseChoice, ChatCompletionResponseStreamChoice]] created: Optional[int] = Field(default_factory=lambda: int(time.time())) usage: Optional[int] = 0
Environment
- OS:
- Python:
- Transformers:
- PyTorch:
- CUDA Support (`python -c "import torch; print(torch.cuda.is_available())"`) :
Anything else?
No response
已经解决了，运行openai_api.py之后，采用langchain的ChatOpenAI调用：
llm_chatglm2 = ChatOpenAI( openai_api_base="http://localhost:8000/v1", openai_api_key="none", streaming=True, temperature=0.7 )
能让Chatglm无缝对接langchain的ChatModel，可以配合ChatPromptTemplate一起使用。
language model直接from langchain.llms import ChatGLM即可
已经解决了，运行openai_api.py之后，采用langchain的ChatOpenAI调用：
llm_chatglm2 = ChatOpenAI( openai_api_base="http://localhost:8000/v1", openai_api_key="none", streaming=True, temperature=0.7 )能让Chatglm无缝对接langchain的ChatModel，可以配合ChatPromptTemplate一起使用。language model直接from langchain.llms import ChatGLM即可
弱弱的问一下，这个的实现方式就是url和出参入参和openai一致，然后只是openai_api_base指向自己部署的chatglm服务，实际上通过http通信实现的吧？
@WilliamChen-luckbob 非常正确，就是用OpenAI的Message封装框架发送一个POST请求，到本地部署的API服务，能无缝对接使用ChatGPT的所有场景
@WilliamChen-luckbob 非常正确，就是用OpenAI的Message封装框架发送一个POST请求，到本地部署的API服务，能无缝对接使用ChatGPT的所有场景
用于agent好用吗，我用from langchain.llms import ChatGLM封装llm，然后使用agent = initialize_agent(llm=llm)，怎么调prompt都没法达到text-davinci-003的效果:(
@mortalGeorge 可以参考这篇文章： 定制基于 ChatGLM-6B 的本地知识库+联网检索能力的 LangChain Agent - 拔赤的文章 - 知乎 https://zhuanlan.zhihu.com/p/635724707 Langchain内置的agent prompt基本是为chatgpt设计的，不太适合chatglm
已经解决了，运行openai_api.py之后，采用langchain的ChatOpenAI调用：
llm_chatglm2 = ChatOpenAI( openai_api_base="http://localhost:8000/v1", openai_api_key="none", streaming=True, temperature=0.7 )能让Chatglm无缝对接langchain的ChatModel，可以配合ChatPromptTemplate一起使用。language model直接from langchain.llms import ChatGLM即可
想请教一下用from langchain.llms import ChatGLM之后还需要配置什么吗？
我用命令行POST测试有正确回应
curl -X POST "http://127.0.0.1:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{\"model\": \"chatglm2-6b\", \"messages\": [{\"role\": \"system\", \"content\": \"You are ChatGLM2-6B, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.\"}, {\"role\": \"user\", \"content\": \"你好，给我讲一个故事，大概100字\"}], \"stream\": false, \"max_tokens\": 100, \"temperature\": 0.8, \"top_p\": 0.8}"
{"id":"chatcmpl-3nF7JbjqCtGaC9VBc8gVzo","object":"chat.completion","created":1701090590,"model":"chatglm2-6b","choices":[{"index":0,"message":{"role":"assistant","content":"这是一个古老的寓言，叫做《乌龟和兔子》。\n\n从前有一只兔子，他非常快，经常骄傲地说自己是最快的动物。一天，他遇到了一只缓慢而稳定的乌龟。他们互相挑衅，兔子嘲笑乌龟说他从来没有机会赢得比赛。\n\n乌龟感到受到了冒犯，于是他向兔子提出了一个挑战：在一场比赛中，看看谁能先到达终点。兔子非常高兴地接受了挑战，并开玩笑说他不需要全力以赴就能轻易赢得比赛"},"finish_reason":"stop"}],"usage":{"prompt_tokens":65,"total_tokens":165,"completion_tokens":100}}
然后看Langchain的文档想实现下面的代码，报错显示404
from langchain.chains import LLMChain
from langchain.llms import ChatGLM
from langchain.prompts import PromptTemplate
llm = ChatGLM(endpoint_url="http://127.0.0.1:8000/v1")
template = """{question}"""
prompt = PromptTemplate(template=template, input_variables=["question"])
llm_chain = LLMChain(prompt=prompt, llm=llm)
llm_chain.run("北京和上海两座城市有什么不同？")
Traceback (most recent call last):
File "/home/dxxm/langchain-sql-test/langchain_sql_test.py", line 11, in <module>
llm_chain.run("北京和上海两座城市有什么不同？")
File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/chains/base.py", line 505, in run
return self(args[0], callbacks=callbacks, tags=tags, metadata=metadata)[
File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/chains/base.py", line 310, in __call__
raise e
File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/chains/base.py", line 304, in __call__
self._call(inputs, run_manager=run_manager)
File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/chains/llm.py", line 108, in _call
response = self.generate([inputs], run_manager=run_manager)
File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/chains/llm.py", line 120, in generate
return self.llm.generate_prompt(
File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/llms/base.py", line 507, in generate_prompt
return self.generate(prompt_strings, stop=stop, callbacks=callbacks, **kwargs)
File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/llms/base.py", line 656, in generate
output = self._generate_helper(
File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/llms/base.py", line 544, in _generate_helper
raise e
File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/llms/base.py", line 531, in _generate_helper
self._generate(
File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/llms/base.py", line 1053, in _generate
self._call(prompt, stop=stop, run_manager=run_manager, **kwargs)
File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/llms/chatglm.py", line 104, in _call
raise ValueError(f"Failed with response: {response}")
ValueError: Failed with response: <Response [404]>
@dbtzy 如果http://127.0.0.1:8000/v1/chat/completions正确运行了，那可以这样试下 from langchain.chat_models import ChatOpenAI from langchain.schema import ( HumanMessage, ) from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler chat = ChatOpenAI( openai_api_base="http://localhost:8000/v1", openai_api_key="none", streaming=True, callbacks=[StreamingStdOutCallbackHandler()], temperature=0 )
resp = chat([HumanMessage(content="写一首诗歌。")])