Gibberish generated with deepseek-ai/deepseek-coder-6.7b-base

[zch-cc]

System Info

TGI Version: Tried 2.0.3, 2.0.4, 2.1.1 all does not work but 2.0.2 works

Information

• [X] Docker
• [ ] The CLI directly

Tasks

• [X] An officially supported command
• [ ] My own modifications

Reproduction

def main():
    client = Client(base_url="http://127.0.0.1:8080")
    response = client.generate(
        prompt="def hel",
    )
    print(response.generated_text)


if __name__ == "__main__":
    main()

outputs

metryryryryryryryryryryryryryryryryryryry 

Expected behavior

should be helloworld or something make more sense

[zch-cc]

find similar issue here https://github.com/huggingface/text-generation-inference/issues/1957 and it looks like versions after 2.0.2 is not working. I tried all versions after 2.0.2, all fails

[ErikKaum]

Hi!

Thanks for reporting the issue 👍 could you share a bit on how to reproduce this? E.g. which model are you using, what's the command to launch the docker container?

[zch-cc]

Hi @ErikKaum, Thanks for responding. The model is

deepseek-ai/deepseek-coder-6.7b-base

and you can use the official docker container command to reproduce this

docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data \
    ghcr.io/huggingface/text-generation-inference:2.1.1 --model-id $model

[ErikKaum]

Thank you.

Yeah, I'm able to reproduce this on my machine. Also quickly checking seems that deepseek-ai/deepseek-coder-6.7b-base is working with the transformers library. So most likely a bug in our end.

At the moment, I unfortunately don't have bandwidth to start debugging.

I'm seeing a lot of warnings like this:

2024-07-16T15:00:17.309591Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token 'õ' was expected to have ID '32000' but was given ID 'None'
2024-07-16T15:00:17.309615Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '÷' was expected to have ID '32001' but was given ID 'None'
2024-07-16T15:00:17.309618Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token 'Á' was expected to have ID '32002' but was given ID 'None'
2024-07-16T15:00:17.309621Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token 'ý' was expected to have ID '32003' but was given ID 'None'
2024-07-16T15:00:17.309624Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token 'À' was expected to have ID '32004' but was given ID 'None'
2024-07-16T15:00:17.309626Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token 'ÿ' was expected to have ID '32005' but was given ID 'None'
2024-07-16T15:00:17.309629Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token 'ø' was expected to have ID '32006' but was given ID 'None'
2024-07-16T15:00:17.309631Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token 'ú' was expected to have ID '32007' but was given ID 'None'
2024-07-16T15:00:17.309641Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token 'þ' was expected to have ID '32008' but was given ID 'None'
2024-07-16T15:00:17.309643Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token 'ü' was expected to have ID '32009' but was given ID 'None'
2024-07-16T15:00:17.309646Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token 'ù' was expected to have ID '32010' but was given ID 'None'
2024-07-16T15:00:17.309648Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token 'ö' was expected to have ID '32011' but was given ID 'None'
2024-07-16T15:00:17.309651Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token 'û' was expected to have ID '32012' but was given ID 'None'
2024-07-16T15:00:17.309653Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<｜begin▁of▁sentence｜>' was expected to have ID '32013' but was given ID 'None'
2024-07-16T15:00:17.309656Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<｜end▁of▁sentence｜>' was expected to have ID '32014' but was given ID 'None'
2024-07-16T15:00:17.309658Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<｜fim▁hole｜>' was expected to have ID '32015' but was given ID 'None'
2024-07-16T15:00:17.309661Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<｜fim▁begin｜>' was expected to have ID '32016' but was given ID 'None'
2024-07-16T15:00:17.309664Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<｜fim▁end｜>' was expected to have ID '32017' but was given ID 'None'
2024-07-16T15:00:17.309666Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<pad>' was expected to have ID '32018' but was given ID 'None'
2024-07-16T15:00:17.309669Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|User|>' was expected to have ID '32019' but was given ID 'None'
2024-07-16T15:00:17.309672Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|Assistant|>' was expected to have ID '32020' but was given ID 'None'
2024-07-16T15:00:17.309674Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|EOT|>' was expected to have ID '32021' but was given ID 'None'
2024-07-16T15:00:17.310063Z  INFO text_generation_router: router/src/main.rs:330: Overriding LlamaTokenizer with TemplateProcessing to follow python override defined in https://github.com/huggingface/transformers/blob/4aa17d00690b7f82c95bb2949ea57e22c35b4336/src/transformers/models/llama/tokenization_llama_fast.py#L203-L205

which makes me think it might be a tokenization issue.

[lambert0312]

How to fix this?