[KED-1511] Model Deployment: Flask/Plumber/Seldon

[yarncraft]

Description

Many ML researchers use pipelines in order to conduct structured hyperparameter tuning and save the tuned model as a pickle (in Python) or RDS (in R). Next, they deploy the saved model as a prediction service. With Docker we have the possibility at hand to do this by extending the template with Flask (for Python) or Plumber (for R).

Context

Business could benefit from Kedro as an end-to-end pipeline whereby ETL, hyperparameter tuning and model training could take place in parallel pipeline-runners, and in a next step, having the best performing model deployed as a prediction service through the use of the mentioned tools.

Possible Implementation

As for now, only Spark and Hadoop are integrated in the template. In order to be able to use the docker container as a prediction service, we must have a flask service running on a port (which can be port forwarded through -p XXXX:XXXX). We could spin up a flask API in a pipeline node (but the libraries are needed in the docker template or similar to the following example: https://www.thegoldfish.org/2018/04/running-flask-on-spark2/).

Possible Alternatives

More advanced deployment strategies could be adopted through the use of Seldon Core, which is integrated in some of the alternative pipeline frameworks. This seems to me as a must-have feature in cases where large amounts of prediction requests are expected and thus scalability is required. However, the complexity of implementation is more advanced.

[idanov]

@yarncraft Thank you very much for adding this issue. We've been asked for providing some sort of an API for running Kedro pipelines with an HTTP request and we have been thinking how to solve it in a more general way. You suggestion will be taken into consideration when we are designing the solution and it will most likely involve exposing an API endpoint through Flask. Please stay tuned, we'll post updates on our progress.

[yarncraft]

Great, thanks for the heads up! It would also make sense to have the option to persist trained models outside the container (for example in blob storage), it is good practice to use bind mounts (or volumes). More info can be found here: https://docs.docker.com/v17.09/engine/admin/volumes/

[yetudada]

Hi @yarncraft! I wanted to check in on this ticket? Did you manage to find ways to deploy Kedro with a Flask API endpoint?

[yarncraft]

Hi @yarncraft! I wanted to check in on this ticket? Did you manage to find ways to deploy Kedro with a Flask API endpoint?

Hello again @yetudada,

I submitted this request cause it seemed to me to be of importance for future clients. I am, however, not actively working on a solution to this feature request at the moment. Since this has never been picked up I can add this to my todo list and start working on a solution. There have been some recent developments in the Data Engineering world that could also be beneficial for Kedro Deployments, more specifically it would be nice to review if something like https://github.com/microsoft/onnxruntime could be leveraged to instantiate a serving layer that works both effectively and efficiently on a platitude of runtimes (take a look here: https://github.com/microsoft/onnxruntime/blob/master/docs/ONNX_Runtime_Server_Usage.md).

[yarncraft]

Having worked with Kubernetes and Cloud Native tech, it also seems promising to take the following approach:

If I would draft an architectural design for the Kedro Server I would go for a workflow whereby models are stored in a blob storage, this could be a Cloud Managed one or a Kubernetes-Native one (f.e. https://rook.io/), and leverage ONNX or Kubernetes-native serverless functions (f.e. https://docs.fission.io/docs/languages/python/) as a serving layer of these stored models. The extra dependency may not be desired but the benefits of serverless functions are:

• Cost effectiveness
• Scalability
• Security
• Support for Enterprise Integration Patterns (Nats triggers, Kafka, REST...)

Hence, given the success of Kubernetes in the Enterprise, such a solution for serving ML models seems like the goto approach to me.