zenml-io
[BUG]: ImportError: cannot import name 'Output' from 'zenml.steps'
System Information
zenml version: 0.82.1 system: Ubuntu 22.04.5 LTS
What happened?
Hi everyone! I'm trying to build a custom orchestrator, exploring an example using docker, but when I try to execute it, I receive an error message. Seems like the code sample is not updated.
Can you help? I try other times with the original code and happens the same.
sample code: https://github.com/zenml-io/zenml-plugins/blob/main/how_to_custom_orchestrator/steps/importer/importer_step.py
[UPDATE]: This sample was created 2 years, when zenml was on 0.43 version. In the actual version the class Output doesn't exist anymore. How can I replace it?
Reproduction steps
- clone the zenml project from github
- follow all the instructions in the README located at https://github.com/zenml-io/zenml-plugins/blob/main/how_to_custom_orchestrator/
- run the run.py file ...
Relevant log output
ImportError: cannot import name 'Output' from 'zenml.steps'
Code of Conduct
- [x] I agree to follow this project's Code of Conduct
@camila-cg That code is indeed outdated! Since then we have migrated to from zenml import step, pipeline.. let me update the code in that repo
@camila-cg Latest version here should work. if you're working locally a zenml clean might be a good practice because you want to start from a fresh slate
Hi @htahir1 , thanks for update! I got the changes but a new error occurred here:
RuntimeError: Unable to connect to the Docker daemon. There are three common
causes for this:
1) The Docker daemon isn't running.
2) The Docker client isn't configured correctly. The client loads its
configuration from the following file: $HOME/.docker/config.json. If your
configuration file is in a different location, set the `DOCKER_CONFIG`
environment variable to the directory that contains your `config.json` file.
3) If your Docker CLI is working fine but you ran into this issue, you might be
using a non-default Docker context which is not supported by the Docker python
library. To verify this, run `docker context ls` and check which context has a
`*` next to it. If this is not the `default` context, copy the `DOCKER ENDPOINT`
value of that context and set the `DOCKER_HOST` environment variable to that
value.
I have the lastest version of docker installed, with no images. My expectation was zenml creating the pipeline image, but it didn't happened.
Can you help with this new scenario?``
@camila-cg yeah sure looks like docker isnt running (as it says on the error :-) ). have you tried doing what it says on the snippet you copy pasted?
Sure, was my env variables, this is working now, but I found another outdated code on file my_docker_orchestrator.py, line 199.
run_models = Client().list_pipeline_runs(
custom_filter={
"environment_attributes": {
"orchestrator_run_id": orchestrator_run_id
}
}
)
I receive the following error:
TypeError: Client.list_pipeline_runs() got an unexpected keyword argument 'custom_filter'
How would be the updated code version to fix it, please?
I saw the code of list_pipeline_runs and made this little adjustment and the sample just worked!
run_models = Client().list_pipeline_runs(
orchestrator_run_id=orchestrator_run_id
)
Thanks for helping!
I have one more question about this code example. Running with zenml 0.82 the example works, but I need to create a custom orchestrator to run on 0.72 zenml version, and on this version this sample crashed.
What is the minimum zenml version to work with custom orchestrators?
@camila-cg I think there is no minimum version but yes as the zenml version changes sometimes sometimes some other stuff changes in the interface .. you can maybe use an older version of this docker orchestrator from git history of the branch I gave you and it might work with 0.72.0
@htahir1 , I'll put here the log I received using the version 0.72, where I did not understand the cause to give you more information about the bug.
Error message:
ValidationError: 1 validation error for SqlZenStore
ssl
Extra inputs are not permitted [type=extra_forbidden, input_value=False, input_type=bool]
For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
Complete log:
Initiating a new run for the pipeline: sklearn_pipeline.
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/camilagusmao/Projetos/how_to_custom_orchestrator/run.py:18 in <module> │
│ │
│ 15 from pipelines import sklearn_pipeline │
│ 16 │
│ 17 if __name__ == "__main__": │
│ ❱ 18 │ sklearn_pipeline() │
│ 19 │
│ │
│ /home/camilagusmao/.pyenv/versions/3.10.12/envs/local_docker_zenml72/lib/python3.10/site-package │
│ s/zenml/pipelines/pipeline_definition.py:1388 in __call__ │
│ │
│ 1385 │ │ │ return self.entrypoint(*args, **kwargs) │
│ 1386 │ │ │
│ 1387 │ │ self.prepare(*args, **kwargs) │
│ ❱ 1388 │ │ return self._run() │
│ 1389 │ │
│ 1390 │ def _call_entrypoint(self, *args: Any, **kwargs: Any) -> None: │
│ 1391 │ │ """Calls the pipeline entrypoint function with the given arguments. │
│ │
│ /home/camilagusmao/.pyenv/versions/3.10.12/envs/local_docker_zenml72/lib/python3.10/site-package │
│ s/zenml/pipelines/pipeline_definition.py:785 in _run │
│ │
│ 782 │ │ logger.info(f"Initiating a new run for the pipeline: `{self.name}`.") │
│ 783 │ │ │
│ 784 │ │ with track_handler(AnalyticsEvent.RUN_PIPELINE) as analytics_handler: │
│ ❱ 785 │ │ │ stack = Client().active_stack │
│ 786 │ │ │ deployment = self._create_deployment(**self._run_args) │
│ 787 │ │ │ │
│ 788 │ │ │ self.log_pipeline_deployment_metadata(deployment) │
│ │
│ /home/camilagusmao/.pyenv/versions/3.10.12/envs/local_docker_zenml72/lib/python3.10/site-package │
│ s/zenml/client.py:309 in __call__ │
│ │
│ 306 │ │ │
│ 307 │ │ if not cls._global_client: │
│ 308 │ │ │ cls._global_client = cast( │
│ ❱ 309 │ │ │ │ "Client", super().__call__(*args, **kwargs) │
│ 310 │ │ │ ) │
│ 311 │ │ │
│ 312 │ │ return cls._global_client │
│ │
│ /home/camilagusmao/.pyenv/versions/3.10.12/envs/local_docker_zenml72/lib/python3.10/site-package │
│ s/zenml/client_lazy_loader.py:211 in _inner │
│ │
│ 208 │ │ │ │ │ with contextlib.suppress(ValueError): │
│ 209 │ │ │ │ │ │ kwargs[k] = ClientLazyLoader(**v).evaluate() │
│ 210 │ │ │ │
│ ❱ 211 │ │ │ return func(*args_, **kwargs) │
│ 212 │ │ │
│ 213 │ │ return _inner │
│ 214 │
│ │
│ /home/camilagusmao/.pyenv/versions/3.10.12/envs/local_docker_zenml72/lib/python3.10/site-package │
│ s/zenml/client.py:382 in __init__ │
│ │
│ 379 │ │ self._root: Optional[Path] = None │
│ 380 │ │ self._config: Optional[ClientConfiguration] = None │
│ 381 │ │ │
│ ❱ 382 │ │ self._set_active_root(root) │
│ 383 │ │
│ 384 │ @classmethod │
│ 385 │ def get_instance(cls) -> Optional["Client"]: │
│ │
│ /home/camilagusmao/.pyenv/versions/3.10.12/envs/local_docker_zenml72/lib/python3.10/site-package │
│ s/zenml/client_lazy_loader.py:211 in _inner │
│ │
│ 208 │ │ │ │ │ with contextlib.suppress(ValueError): │
│ 209 │ │ │ │ │ │ kwargs[k] = ClientLazyLoader(**v).evaluate() │
│ 210 │ │ │ │
│ ❱ 211 │ │ │ return func(*args_, **kwargs) │
│ 212 │ │ │
│ 213 │ │ return _inner │
│ 214 │
│ │
│ /home/camilagusmao/.pyenv/versions/3.10.12/envs/local_docker_zenml72/lib/python3.10/site-package │
│ s/zenml/client.py:439 in _set_active_root │
│ │
│ 436 │ │ │
│ 437 │ │ # Sanitize the client configuration to reflect the current │
│ 438 │ │ # settings │
│ ❱ 439 │ │ self._sanitize_config() │
│ 440 │ │
│ 441 │ def _config_path(self) -> Optional[str]: │
│ 442 │ │ """Path to the client configuration file. │
│ │
│ /home/camilagusmao/.pyenv/versions/3.10.12/envs/local_docker_zenml72/lib/python3.10/site-package │
│ s/zenml/client_lazy_loader.py:211 in _inner │
│ │
│ 208 │ │ │ │ │ with contextlib.suppress(ValueError): │
│ 209 │ │ │ │ │ │ kwargs[k] = ClientLazyLoader(**v).evaluate() │
│ 210 │ │ │ │
│ ❱ 211 │ │ │ return func(*args_, **kwargs) │
│ 212 │ │ │
│ 213 │ │ return _inner │
│ 214 │
│ │
│ /home/camilagusmao/.pyenv/versions/3.10.12/envs/local_docker_zenml72/lib/python3.10/site-package │
│ s/zenml/client.py:462 in _sanitize_config │
│ │
│ 459 │ │ if not self._config: │
│ 460 │ │ │ return │
│ 461 │ │ │
│ ❱ 462 │ │ active_workspace, active_stack = self.zen_store.validate_active_config( │
│ 463 │ │ │ self._config.active_workspace_id, │
│ 464 │ │ │ self._config.active_stack_id, │
│ 465 │ │ │ config_name="repo", │
│ │
│ /home/camilagusmao/.pyenv/versions/3.10.12/envs/local_docker_zenml72/lib/python3.10/site-package │
│ s/zenml/client.py:644 in zen_store │
│ │
│ 641 │ │ Returns: │
│ 642 │ │ │ The global zen store. │
│ 643 │ │ """ │
│ ❱ 644 │ │ return GlobalConfiguration().zen_store │
│ 645 │ │
│ 646 │ @property │
│ 647 │ def root(self) -> Optional[Path]: │
│ │
│ /home/camilagusmao/.pyenv/versions/3.10.12/envs/local_docker_zenml72/lib/python3.10/site-package │
│ s/zenml/config/global_config.py:230 in __custom_getattribute__ │
│ │
│ 227 │ │ Returns: │
│ 228 │ │ │ The attribute value. │
│ 229 │ │ """ │
│ ❱ 230 │ │ value = super().__getattribute__(key) │
│ 231 │ │ if key.startswith("_") or key not in type(self).model_fields: │
│ 232 │ │ │ return value │
│ 233 │
│ │
│ /home/camilagusmao/.pyenv/versions/3.10.12/envs/local_docker_zenml72/lib/python3.10/site-package │
│ s/zenml/config/global_config.py:712 in zen_store │
│ │
│ 709 │ │ │ The current zen store. │
│ 710 │ │ """ │
│ 711 │ │ if self._zen_store is None: │
│ ❱ 712 │ │ │ self._configure_store(self.store_configuration) │
│ 713 │ │ assert self._zen_store is not None │
│ 714 │ │ │
│ 715 │ │ return self._zen_store │
│ │
│ /home/camilagusmao/.pyenv/versions/3.10.12/envs/local_docker_zenml72/lib/python3.10/site-package │
│ s/zenml/config/global_config.py:369 in _configure_store │
│ │
│ 366 │ │ │
│ 367 │ │ # TODO: Revisit the flow regarding the registration of the default │
│ 368 │ │ # entities once the analytics v1 is removed. │
│ ❱ 369 │ │ store = BaseZenStore.create_store(config, True, **kwargs) │
│ 370 │ │ │
│ 371 │ │ logger.debug(f"Configuring the global store to {store.config}") │
│ 372 │ │ self.store = store.config │
│ │
│ /home/camilagusmao/.pyenv/versions/3.10.12/envs/local_docker_zenml72/lib/python3.10/site-package │
│ s/zenml/zen_stores/base_zen_store.py:236 in create_store │
│ │
│ 233 │ │ """ │
│ 234 │ │ logger.debug(f"Creating store with config '{config}'...") │
│ 235 │ │ store_class = BaseZenStore.get_store_class(config.type) │
│ ❱ 236 │ │ store = store_class( │
│ 237 │ │ │ config=config, │
│ 238 │ │ │ skip_default_registrations=skip_default_registrations, │
│ 239 │ │ │ **kwargs, │
│ │
│ /home/camilagusmao/.pyenv/versions/3.10.12/envs/local_docker_zenml72/lib/python3.10/site-package │
│ s/zenml/zen_stores/base_zen_store.py:135 in __init__ │
│ │
│ 132 │ │ │ **kwargs: Additional keyword arguments to pass to the Pydantic │
│ 133 │ │ │ │ constructor. │
│ 134 │ │ """ │
│ ❱ 135 │ │ super().__init__(**kwargs) │
│ 136 │ │ │
│ 137 │ │ self._initialize() │
│ 138 │
│ │
│ /home/camilagusmao/.pyenv/versions/3.10.12/envs/local_docker_zenml72/lib/python3.10/site-package │
│ s/pydantic/main.py:193 in __init__ │
│ │
│ 190 │ │ """ │
│ 191 │ │ # `__tracebackhide__` tells pytest and some other tools to omit this function fr │
│ 192 │ │ __tracebackhide__ = True │
│ ❱ 193 │ │ self.__pydantic_validator__.validate_python(data, self_instance=self) │
│ 194 │ │
│ 195 │ # The following line sets a flag that we use to determine when `__init__` gets overr │
│ 196 │ __init__.__pydantic_base_init__ = True # pyright: ignore[reportFunctionMemberAccess │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValidationError: 1 validation error for SqlZenStore
ssl
Extra inputs are not permitted [type=extra_forbidden, input_value=False, input_type=bool]
For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
I think its hard to debug old versions for me - maybe upgrade to the latest version? :-)
There are a lot of zenml pipelines here running on version 0.72, that is not so old. This is the reason I need to keep this version for the custom orchestrator I'm developing.
Hey @camila-cg, I am closing this issue due to inactivity for now. Feel free to comment again if the issue persists.
