metaflow
metaflow copied to clipboard
Is it possible to run metaflow steps in custom docker containers on local?
What I am trying to do:
I am trying to create a metaflow pipeline. I have several steps one of those steps, I want to run in a custom docker container. The ways I found was using either @kuberenetes decorator & specifying the image name or using @batch decorator. Now, since I want to test it first.
Approach I followed:
I thought of setting up minikube on my local, so for a particular step metaflow could spin up the custom container in minikube & do the process.
What went wrong:
but, while running the command.
python metaflow_test.py run --with kubernetes:image=hub.docker.com/custom_container
I get error:
Kubernetes error:
The @kubernetes decorator requires --datastore=s3 or --datastore=azure or --datastore=gs at the moment.
Conclusion:
looking at this error what I am understading is I can only spawn containers on remote k8s clusters running on cloud not on minikube.
Question:
is there a way to test locally? or should I just do the setup on cloud.