petals-infra
ValueError: extra_stop_sequences require stop_sequence length to be exactly 1 token
Hi @borzunov,
I started a private swarm with two GPU servers and built this web chat on another CPU node.
When I input something, it raises errors like this
Oct 03 18:31:31.046 [INFO] ws.generate.open(), model_name='meta-llama/Meta-Llama-3.1-8B-Instruct', max_length=512, http_request.origin='http://172.28.177.63:5001' Oct 03 18:31:31.105 [INFO] ws.generate.step(), inputs="A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.###Assistant: Hi! How can I help you?###Human: hi###Assistant:" Oct 03 18:31:31.116 [INFO] stop_seq='###' Oct 03 18:31:31.116 [INFO] extra_stop_seq=['</s>'] Oct 03 18:31:31.118 [INFO] cont_token=tensor([[128000, 14711]]) Oct 03 18:31:31.119 [WARN] [/home/zhangsq/chat.petals.dev/websocket_api.py.ws_api_generate:95] ws.generate failed: Traceback (most recent call last): File "/home/zhangsq/chat.petals.dev/websocket_api.py", line 51, in ws_api_generate raise ValueError("extra_stop_sequences require stop_sequence length to be exactly 1 token") ValueError: extra_stop_sequences require stop_sequence length to be exactly 1 token
If I directly commented on these two lines, then the output of this chatbot would look like this
Assistant:|begin_of_text|> Hi|begin_of_text|>!|begin_of_text|> How|begin_of_text|> can|begin_of_text|> I|begin_of_text|> assist|begin_of_text|> you|begin_of_text|> today|begin_of_text|>?|begin_of_text|>.
Could you help me out? Thanks in advance.
Best, Shiqi
