mistral.rs
mistral.rs copied to clipboard
Reduce CPU usage when idle
Currently, the tight loop in Engine
causes very high single core CPU usage when idle. This is also not great because this is long-running blocking code running inside of an async task, blocking an async worker entirely. On systems with lower core count, this will probably impact performance fairly negatively.
From my testing, this change dramatically drops CPU usage with minimal impact to performance, although I have not had a chance to benchmark properly.