[BUG]: Import/Export not working with local Kubeflow stacks

[alarivarmann]

Contact Details [Optional]

[email protected]

System Information

ZenML version: 0.11.0 Install path: /media/alaridl/DATA_LINUX1/pypoetry/virtualenvs/zenmlenv-Rn-xthbg-py3.9/lib/python3.9/site-packages/zenml Python version: 3.9.13 Platform information: {'os': 'linux', 'linux_distro': 'ubuntu', 'linux_distro_like': 'debian', 'linux_distro_version': '20.04'} Environment: native Integrations: ['airflow', 'graphviz', 'kubeflow', 'kubernetes', 'mlflow', 'scipy', 'seldon', 'tensorflow']

What happened?

It seems that the import feature is dependent on some dependencies which are not included at import time, therefore it would not work for Kubeflow stacks. Simulating how a stack exported by another user (user B) can be imported, can be accomplished by user A first exporting the stack zenml stack export and then running zenml stack import by user B. However, it does not work with Kubeflow stacks because the user A and B will not have the same Kubernetes dependency setup.

In fact the Kubernetes cluster that was called "remote" here was created on the same machine as where it was exported by ZenML itself, which ZenML does not recognize. I attempted to set the kubecontext to k3d-zenml-kubeflow-85fa0ba0 , but it didn't help. A PR to make the import/export feature available for local Kubeflow stacks would be indicated.

Reproduction steps

1. Create a Kubeflow stack (orchestrator=Kubeflow)
2. zenml stack export stack_name some_name.yaml
3. zenml stack import some_name.yaml This will result in the error given above, so the import will not work due to stack validation error. ...

Relevant log output

StackValidationError: Custom validation function failed to validate stack 'kubeflow-mlflow2': The Kubeflow orchestrator is configured to run pipelines in a remote Kubernetes cluster designated by the 'k3d-zenml-kubeflow-85fa0ba0' configuration context, but the 'default2' metadata_store is a local stack component and will not be available in the Kubeflow pipeline step. Please ensure that you always use non-local stack components with a remote Kubeflow orchestrator, otherwise you may run into pipeline execution problems. You should use a flavor of metadata_store other than 'sqlite'.

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[fa9r]

Hi Alari,

Thanks for opening the issue. Could you try manually removing the following line from your exported YAML file and then retry the import command:

    kubernetes_context: k3d-zenml-kubeflow-85fa0ba0

This should force the Kubeflow orchestrator to regenerate the kubernetes_context attribute when being imported.

Best regards, Felix

[htahir1]

@alarivarmann Did you try this again? Did it work out? Perhaps we can close it if it worked for you

[alarivarmann]

If I remember correctly, I still had some issue with this. I would probably have remembered if it worked right away. I could get back at you earliest some time next week because this week is very packed for me, if you still wish to debug automated imports/exports. Otherwise maybe close the issue. Just a note that if a user cannot be sure that the functionality works as described in the docs, then I wouldn't close the issue until the docs match the functionality. Your call of course :)

[AlexejPenner]

Hey @alarivarmann is this still relevant for you? Zenml has changed a lot over the past few months and by nature of those changes this should not be an issue anymore. I would be closing this issue, but feel free to reopen if this still impacts you and we'll try to help you.