FastChat icon indicating copy to clipboard operation
FastChat copied to clipboard

Reduce the peak memory requirement when applying delta

Open zhisbug opened this issue 1 year ago • 1 comments

Seeing many complaints about the required peak memory being too high. We would like to keep the peak memory of the following command be less than 8GB.

python3 -m fastchat.model.apply_delta \
    --base /path/to/llama-13b \
    --target /output/path/to/vicuna-13b \
    --delta lmsys/vicuna-13b-delta-v1.1

zhisbug avatar Apr 08 '23 02:04 zhisbug

I am working on reducing the peak memory, and the peak memory now is 11372.2 MiB. See #402 and we can further improve it.

andy-yang-1 avatar Apr 13 '23 06:04 andy-yang-1

@andy-yang-1 Great work, thanks!

zhisbug avatar Apr 21 '23 02:04 zhisbug