llama-cpp-python
llama-cpp-python copied to clipboard
abetlen
Bugfix: Fix broken: UnicodeDecodeError: 'utf-8' codec can't decode
Doesn't this just remove the error? Emoji's still don't work
Test passed in Chinese, not test Emoji.
Very funny indeed. My vicuna prefers to answer me in Chinese. With this fix at least it can without erroring out.
Was this resolved Upstream? https://github.com/ggerganov/llama.cpp/commit/aaf3b23debc1fe1a06733c8c6468fb84233cc44f
@MillionthOdin16 I don't think so because I've had this issue on linux as well. I believe the issue is that utf-8 encodng is variable length and certain tokens are not valid utf-8 because they're just returned as bytes which may include partial utf-8 code points.
I think this needs some tests to ensure we're properly keeping track of the number of returned bytes.
@Niek can you confirm that this fixes the bug and gives the same result in streaming vs. regular mode. For example, compare streaming and regular mode for a completion that breaks in streaming mode with a fixed seed and temperature=0.
I just tested, for my own reference:
docker run --rm -it -v /path/to/models:/models -p8000:8000 python:3-buster bash
git clone -b river https://github.com/riverzhou/llama-cpp-python.git /app
cd /app
sed -i -e 's/[email protected]:/https:\/\/github.com\//' -e 's/.git$//' .gitmodules
git submodule update --init --recursive
python -m pip install --upgrade pip pytest cmake scikit-build setuptools fastapi sse_starlette uvicorn
python setup.py develop
HOST=0.0.0.0 MODEL=/models/ggml-vicuna-7b-4bit.bin python3 -m llama_cpp.server
With a prompt like show me 3 emojis, I no longer get an error, but it seems the message returned is empty instead. So doesn't look like a full fix.
@Niek Can you try chaging for i in range(1,4): to for i in reversed(range(1,4)): so instead we're decoding the longest posible sequence? Not sure this would fix it either but worth a try.
I change the code a littie bit, and it works: _text = '' try: _text = text[start:].decode("utf-8") except UnicodeDecodeError: for i in range(-2,2): # changed to (-2,2) try: _text = text[start+i:].decode("utf-8") # changed to [start+i:] break except UnicodeDecodeError: continue yield { "id": completion_id, "object": "text_completion", "created": created, "model": self.model_path, "choices": [ { "text": _text, "index": 0, "logprobs": None, "finish_reason": None,
@riverzhou can you check if this bug is still occurs since #118 was merged?
@riverzhou can you check if this bug is still occurs since #118 was merged?
Fine. It's OK now. Thanks.