How to set a number of cpu thereds to use?

[HCBlackFox]

I am using cpu for LLm and how predictable it is slow, but also it uses only 15-20% of cpu power. Whats may be wrong?

[Ph0rk0z]

Transformers/Accelerate does this in CPU mode too? Ouch.

edit: Hey.. so a shot in the dark, did you try with https://github.com/zphang/transformers.git@68d640f7c368bcaaaecfc678f11908ebbd3d6176

That transformer would use multiple cores for me for GPTQ and RWKV loading.

pip uninstall transformers
pip install git+https://github.com/zphang/transformers.git@3884da12ce327667d4df5101aef3533cc32be61f 

To go back:

pip uninstall transformers
pip install git+https://github.com/huggingface/transformers

[HCBlackFox]

Transformers/Accelerate does this in CPU mode too? Ouch.

edit: Hey.. so a shot in the dark, did you try with https://github.com/zphang/transformers.git@68d640f7c368bcaaaecfc678f11908ebbd3d6176

That transformer would use multiple cores for me for GPTQ and RWKV loading.

pip uninstall transformers
pip install git+https://github.com/zphang/transformers.git@3884da12ce327667d4df5101aef3533cc32be61f 

To go back:

pip uninstall transformers
pip install git+https://github.com/huggingface/transformers

Oh, thank u, i will try)

[github-actions[bot]]

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.