feat: support dynamic configuration for dynamic batching without service restart

[wenzhaojie]

Describe the feature

We can set parameters such as num=1, max_batch_size=4, max_wait_time=10 within append_worker(). However, we cannot modify these parameters during the service's runtime. Is it possible to provide a corresponding API to adjust the service parameters?

Why do you need this feature?

If we can implement this control API, we can adjust resource configurations without restarting the service, achieving dynamic changes in QoS (Quality of Service).

Additional context

No response

[kemingy]

Thanks for your feedback.

num is used by Python side to control how many multiprocesses to run. max_batch_size and max_wait_time are used by Rust side to route the requests. Also, only rust side has the web interface. If we need to change all of them, then Python main process might need to register a SIGUSR1 handler. This also requires a lock to sync the configuration between tokio threads. Restarting the container with a different arguments might be easier for both developers and users. (But still welcome PR)

I think it's related to https://github.com/mosecorg/mosec/issues/382 though that one is fully automatic.