Tensor Size Mismatch and `setup_caches` Assertion Error in `generate.py`

[ppmzhang2]

Self Checks

• [X] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• [X] Please do not modify this template :) and fill in all the required fields.

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

python tools/llama/generate.py
--text "要转换的文本"
--prompt-text "你的参考文本"
--prompt-tokens "fake.npy"
--checkpoint-path "checkpoints/fish-speech-1.4"
--num-samples 1
--compile

✔️ Expected Behavior

codes_0.npy

❌ Actual Behavior

Traceback (most recent call last):
  File "/fish-speech/tools/llama/generate.py", line 703, in <module>
    main()
  File "/home/user/.pyenv/versions/miniconda3-latest/envs/py310-nix-fishspeech/lib/python3.10/site-packages/click/core.py", line 1157, in
__call__
    return self.main(*args, **kwargs)
  File "/home/user/.pyenv/versions/miniconda3-latest/envs/py310-nix-fishspeech/lib/python3.10/site-packages/click/core.py", line 1078, in
main
    rv = self.invoke(ctx)
  File "/home/user/.pyenv/versions/miniconda3-latest/envs/py310-nix-fishspeech/lib/python3.10/site-packages/click/core.py", line 1434, in
invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user/.pyenv/versions/miniconda3-latest/envs/py310-nix-fishspeech/lib/python3.10/site-packages/click/core.py", line 783, in i
nvoke
    return __callback(*args, **kwargs)
  File "/fish-speech/tools/llama/generate.py", line 687, in main
    for response in generator:
  File "/fish-speech/tools/llama/generate.py", line 488, in generate_long
    y = generate(
  File "/home/user/.pyenv/versions/miniconda3-latest/envs/py310-nix-fishspeech/lib/python3.10/site-packages/torch/utils/_contextlib.py", l
ine 116, in decorate_context
    return func(*args, **kwargs)
  File "/home/user/.pyenv/versions/miniconda3-latest/envs/py310-nix-fishspeech/lib/python3.10/site-packages/torch/utils/_contextlib.py", l
ine 116, in decorate_context
    return func(*args, **kwargs)
  File "/fish-speech/tools/llama/generate.py", line 245, in generate
    empty[:, :T] = prompt
RuntimeError: The expanded size of the tensor (0) must match the existing size (275) at non-singleton dimension 1.  Target sizes: [9, 0].
 Tensor sizes: [9, 275]