Yanli comments

Repositories
Issues
Comments

Results 3 comments of


                                            Yanli

[Max1100/bigdl-llm]Met OOM easily when running Deci/DeciLM-7B int4/fp8 multi-batch(bs=8) based on bigdl-llm=2.5.0b20240124 while it could support up to bs=150 based on bigdl-llm=2.5.0b20240118

[run.txt](https://github.com/intel-analytics/BigDL/files/14048561/run.txt) [benchmark_hf_model_bigdl.txt](https://github.com/intel-analytics/BigDL/files/14048564/benchmark_hf_model_bigdl.txt)

[Max1100/bigdl-llm]Met OOM easily when running llama2-7b/Mistral-7B-v0.1 int4/fp8 multi-batch

[log.txt](https://github.com/intel-analytics/BigDL/files/14025950/log.txt)

[Max1100/bigdl-llm]Met OOM easily when running llama2-7b/Mistral-7B-v0.1 int4/fp8 multi-batch

[benchmark_hf_model_bigdl.txt](https://github.com/intel-analytics/BigDL/files/14026139/benchmark_hf_model_bigdl.txt) [run.txt](https://github.com/intel-analytics/BigDL/files/14026142/run.txt)