mistral.rs icon indicating copy to clipboard operation
mistral.rs copied to clipboard

Reduce CPU usage when idle

Open scottwey opened this issue 5 months ago • 4 comments

Currently, the tight loop in Engine causes very high single core CPU usage when idle. This is also not great because this is long-running blocking code running inside of an async task, blocking an async worker entirely. On systems with lower core count, this will probably impact performance fairly negatively.

From my testing, this change dramatically drops CPU usage with minimal impact to performance, although I have not had a chance to benchmark properly.

scottwey avatar Sep 15 '24 01:09 scottwey