text-generation-launcher failed to load model.

[fkcheung]

To reproduce this problem, use the following Commit ID (0f2daad8b959361aa41d5500d3778e23b1118bdc) and run the command:

CUDA_VISIBLE_DEVICES=0,1,2,3 text-generation-launcher --model-id deepseek-ai/deepseek-coder-33b-instruct --dtype float16 --max-input-length 3072 --max-total-tokens 8192 --hostname 0.0.0.0 --port 8080 &

Afterward, the following error message will be displayed after the "connected" log message:

thread 'main' panicked at router/src/infer.rs:324:14: called Result::unwrap() on an Err value: Error { kind: SyntaxError, detail: "unexpected ., expected assignment operator", name: "", line: 7 }

---------------------------------- ----------------------------------- 4 | {%- set ns = namespace(found=false) -%} 5 | {%- for message in messages -%} 6 | {%- if message['role'] == 'system' -%} 7 > {%- set ns.found = true -%} i ^ syntax error 8 | {%- endif -%} 9 | {%- endfor -%} 10 | {{bos_token}}{%- if not ns.found -%}

No referenced variables
-------------------------------------------------------------------------------

The problem can be fixed by checking out the previous commit ID (532146338bfcc6af86efbe61825206a9e913f37f).

[heibaidaolx123]

same error

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.