text-generation-webui icon indicating copy to clipboard operation
text-generation-webui copied to clipboard

Published 20 hours ago • verified publisher icon oobabooga

Increase the max threads from 32 to 256

Open chraac opened this issue 10 months ago • 5 comments

In #5292, we've requested to increase the threads_batch in ui to allow > 32 threads, and here's the PR, to enlarge it.

Minor changes, tested on my rig with llama.cpp loader, work as expected. 947559cc1db69b66a2950aefcb6f8a9

Checklist:

chraac avatar Apr 20 '24 03:04 chraac

I think sometimes using more threads than physical cores is desirable. For some intel CPUs. Depends on your setup.

Ph0rk0z avatar Apr 20 '24 19:04 Ph0rk0z

I think sometimes using more threads than physical cores is desirable. For some intel CPUs. Depends on your setup.

hmm, then I'll leave the maximum to 256, let the user decide how many threads they want to use

chraac avatar Apr 21 '24 13:04 chraac

@Ph0rk0z do you mind to merge this PR regarding its a trival change, verified on my machine, and also the linter show no errors.

chraac avatar Apr 25 '24 02:04 chraac

Heh, I can't merge. Its on oobabooga.

Ph0rk0z avatar Apr 26 '24 23:04 Ph0rk0z

Heh, I can't merge. Its on oobabooga.

Ahh, got it, anyway, thanks for the comments

chraac avatar Apr 27 '24 10:04 chraac

I'd also like to see this merged

end-me-please avatar May 15 '24 20:05 end-me-please

@oobabooga , sorry for bothering you, could you please have a look on this PR when you have a monent? I'm open to any feedback you might have and am ready to make any necessary adjustments.. Thanks!

chraac avatar May 18 '24 03:05 chraac

Thanks for the PR -- it's a fair change, I just didn't have the time to review it before. Merging now.

oobabooga avatar May 19 '24 23:05 oobabooga