elyra icon indicating copy to clipboard operation
elyra copied to clipboard

Add support to Prefect runtime processor type.

Open jcfaracco opened this issue 2 years ago • 2 comments

Is your feature request related to a problem? Please describe. As mentioned by https://github.com/elyra-ai/elyra/issues/2258, it should be easy now to integrate new processor executor type into Elyra definition. This is just a tracker to support Prefect. Ideally, the implementation should be similar to Airflow considering the similarities between each other.

Describe the solution you'd like A pipeline that supports Prefect runtime processor.

Describe alternatives you've considered There is an alternative to create a generic pipeline with Prefect code, but it would be nice to have it built-in.

jcfaracco avatar Aug 14 '22 14:08 jcfaracco

Is it possible to include runtimes as a plugin? So, we don't need to open requests like these. We could just maintain a separate elyra-<runtime_type> and if it is relevant, it can be incorporated in the future.

jcfaracco avatar Aug 15 '22 12:08 jcfaracco

Is it possible to include runtimes as a plugin? So, we don't need to open requests like these. We could just maintain a separate elyra-<runtime_type> and if it is relevant, it can be incorporated in the future.

unfortunately we dont have the plumbing in place yet for a more modular way of including additional processor runtimes, this includes packaging (see in progress https://github.com/elyra-ai/elyra/pull/2259).

Makes sense to keep this open as an umbrella issue for adding support for prefect in elyra, we would be happy to help answer any questions or issues that come up and look forward to this contribution.

akchinSTC avatar Aug 15 '22 17:08 akchinSTC