rsong0606

gpu memory size recommended for pruning the llama2-7b-chat-hf model

4

Great work team! Currently, I am pruning on the llama2-7b-chat-hf model from hugging face. python main.py > --model NousResearch/Llama-2-7b-chat-hf > --prune_method wanda > --sparsity_ratio 0.5 > --sparsity_type 2:4 > --save...

[Inference Speed] How we can shorten the inference time per generation?

3

Hey Team, good work overall! I am using this sample code, played a bit with different descriptions. Overall, this is great. However, it took **9** seconds to generate a **20...