Possible methods to eliminate GPU usage

[benaclejames]

Hi there, I'd first like to thank you for writing this code. It's been a godsend to use it as a baseplate, and implementing loopback support via WASAPI was certainly not as bad as it would've been with other codebases.

However, I'm trying to get the model to use less GPU. I understand it's using compute shaders but this method causes noticeable lag when using the GPU for other things such as games. Is there any method for reducing load on the GPU short of fully switching to CPU?

Thanks

[Const-me]

@benaclejames If you’re integrating into your own game, a proper solution is D3D12 (I never tested but I would expect my code to be compatible with D3D11On12 emulation layer), use D3D12_COMMAND_QUEUE_PRIORITY_HIGH for rendering, and another queue for Whisper, with D3D12_COMMAND_QUEUE_PRIORITY_NORMAL

If you run Whisper in different process, I’m not sure it’s possible, but here’s a few things to try. All these changes gonna decrease performance of Whisper, but they might improve performance of other processes which use GPU.

1. Low the OS priority for the Whisper process, see SetPriorityClass WinAPI, the default is NORMAL_PRIORITY_CLASS, set to below normal or idle.

2. Reduce queueLength constant, there, from 32 to a smaller value, like 8 or even 4. That constant defines how many pending compute tasks are allowed to run pipelined on GPU before my code starts to wait for some of them to complete.

3. Set useHighRezTimer constant to true, there. That flag defines how specifically Whisper waits for GPU, and the default value causes it to busy wait.