metaflow icon indicating copy to clipboard operation
metaflow copied to clipboard

Is it possible to run metaflow steps in custom docker containers on local?

Open AyushUnleashed opened this issue 1 year ago • 0 comments

What I am trying to do:

I am trying to create a metaflow pipeline. I have several steps one of those steps, I want to run in a custom docker container. The ways I found was using either @kuberenetes decorator & specifying the image name or using @batch decorator. Now, since I want to test it first.

Approach I followed:

I thought of setting up minikube on my local, so for a particular step metaflow could spin up the custom container in minikube & do the process.

What went wrong:

but, while running the command.

 python metaflow_test.py run --with kubernetes:image=hub.docker.com/custom_container

I get error:

Kubernetes error:
The @kubernetes decorator requires --datastore=s3 or --datastore=azure or --datastore=gs at the moment.

Conclusion:

looking at this error what I am understading is I can only spawn containers on remote k8s clusters running on cloud not on minikube.

Question:

is there a way to test locally? or should I just do the setup on cloud.

AyushUnleashed avatar Feb 19 '24 13:02 AyushUnleashed