姜永久

icon company University of Science and Technical of China icon location Code to make a better world

Results 2 issues of 姜永久

decoding_mode top_k_top_p does not take effect for llama2 not same with huggingface

3 comment

### System Info CPU Architecture: AMD EPYC 7V13 64-Core Processor CPU/Host memory size: 440 GPU properties: A800 80GB GPU name: NVIDIA A800 80GB x2 GPU mem size: 80Gb x 2...

triaged

[BUG/Help] 用vllm 起INT4量化版本的模型报错 类型不匹配 self_attention.dense.weight int4 shape [4096,2048] mismatch fp16 shape [4096, 4096]

### Is there an existing issue for this? - [X] I have searched the existing issues ### Current Behavior self_attention.dense.weight int4 shape [4096,2048] mismatch fp16 shape [4096, 4096] which results...