Package up environment as part of MLServer CLI

[adriangonz]

MLServer now has built-in support to unpack and activate a conda-pack tarball. This feature could be leveraged to run the custom environment defined in theconda.yaml file usually present in MLflow model artifacts. Since MLServer expects a tarball, this issue should explore best practices on going from a conda.yaml file to a conda-pack tarball, reducing the potential friction for the user.

One potential solution is to include a utility on the mlserver-mlflow package which bridges this gap for the users.

[kyuwoo-choi]

It is a great improvement for mlflow models

[axsaucedo]

That's great feedback @kyuwoo-choi thanks for the comment - we're still exploring the most effective ways of providing workflows for this as well as tradeoffs (eg image building vs tarball env download), will be great to get further feedback as these workflows develop 🙂

[kyuwoo-choi]

Cheers. Shouldn't building the image or making the tarball requires engineers to do the extra steps for it? It will be greater if we can let MLServer create conda env according to the conda.yaml included in the MLFlow model. Similar to the Seldon Core, With it, I don't know how, the dependencies are downloaded while initialing the inference server respecting the MLServer model conda.yaml.

[adriangonz]

Hey @kyuwoo-choi ,

The previous behavior in Seldon Core (i.e. building the environment on-the-fly) had a few issues - the main one being that generating the environment takes a while, resulting in the pod taking a long time to come up. That is, each time your model comes up (i.e. bc of a new version, a restart, etc), it can take a few good minutes to start.

For MLServer, we decided to try out a different approach, which involves pre-generating this environment in advance (i.e. through conda-pack and the environment tarballs). I agree that this is an extra step compared to the previous behavior in SC, however it should still improve on other aspects of the overall infrastructure.

This issue aims to introduce some utilities to streamline the environment creation (i.e. so that you can do something like mlserver pack -o env.tar.gz). This won't remove the extra step, but should help for CI pipelines.