Custom uvicorn configuration

[pablobgar]

We are having some difficulties with Uvicorn configuration. We need to apply some custom settings on the rest server like using custom timeouts or applying a custom ssl certificate. This configuration is exposed by Uvicorn and can be set ussing uvicorn.Config method, but it is not exposed by MLServer configuration file.

To support these new configurations, we could add new fields in the settings file and then add them to the Uvicorn configuration input, but we believe this is an inflexible and not scalable solution.

So could it make sense to allow users to directly give a kwargs dictionary in Uvicorn format with their own settings? By doing this, we would have an easier way to apply this configuration and users would have more flexibility to give a complete and customized Uvicorn configuration. Allowing them to use any configuration, even if it is not supported in the MLServer settings file.

Maybe a similar model could be applied to the GRPC server and the new metrics server too.

What do you think?

[adriangonz]

Hey @pablobgar,

Most of these things apply to both gRPC and REST, right? As a user, I would expect to set a SSL cert once and then have that applied to both the gRPC and REST servers (i.e. instead of tinkering with Uvicorn's and grpcio config separately). Similarly, as a user I would prefer to set the timeout once and get that applied to both gRPC and the REST server rather than setting that value twice.

Therefore, would it make sense to just add an ssl_key / ssl_cert pair of config fields to set up SSL servers? Regarding the timeout, I can't seem to find many flags to handle that. FastAPI / Uvicorn seems to rely on gUnicorn or whatever else is upstream (e.g. Istio) to handle timeouts. Is there any particular setting that you're looking at?

[pablobgar]

Hi @adriangonz ,

Thanks for your response!! I understand your point, and I agree that the one you describe is going to be the most common use case. However, I think it would be interesting to have the possibility to directly modify the properties of the gRPC and REST servers.

Although for most users the current configuration is sufficient, there may be use cases where it is necessary to apply a specific configuration for each server, as is ours.

In addition, we would like to have more flexibility to be able to modify the configuration of the servers without having such a large dependency on MLServer.

We could add the ssl_key and ssl_cert options now, but what if we need to modify another parameter in a few days? I think we need a more flexible way to solve this than having to create an issue and a PR, and wait for a release to come out with these new settings.

I agree that the normal execution flow should still be to apply all settings using the settings class, and the settings applied on this class should have priority. But I think it would be a good idea to have a more flexible option for more advanced use cases.

[pablobgar]

I link this related issue in which @erik-hasse raised a similar concern about having to open an issue and wait for new configs to be added whenever someone wants to use a new Uvicorn config.

[adriangonz]

Leaving aside the usability aspects (i.e. making it easier to use for the user), my main concern is that giving full access to the underlying Uvicorn / FastAPI / gRPC settings would make it really hard to ensure that all settings are aligned and compatible with what MLServer expects.

Based on that, we could expose a sort of low-level _uvicorn_settings field (or allow for env vars to be read as suggested in #571 ). However, this would be provided with no support and it would be up to the user to ensure that the settings are compatible with what MLServer expects. Likewise, any setting provided here would override whatever is provided in the MLServer settings file as it would otherwise be a massive effort to reconcile both sources of configuration.

[pablobgar]

Hi @adriangonz!!

I understand that this has to be provided without support. Anyway it would be nice to have the possibility to do so assuming the consequences.

In addition, it would be great to allow this configuration also on the new metrics server 😀