Airflow integration in kubeflow pipeline

[goswamig]

Is your feature request related to a problem? Please describe. Currently Kubeflow pipeline uses Argo in backend and frontend, in order to integrate it with AWS airflow, we need to enable the support of airflow.

Describe the solution you'd like Swap out the need of argo in kubeflow to airflow.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

[Harikantipudi]

@goswamig . Did something similar recently may not be exactly , I can share steps but I think we need to have a more strong rationale and justification to have Airflow integrated within Kubeflow, since end of the day orchestration is already taken care

[goswamig]

@Harikantipudi in AWS distro of kubeflow, we are trying to help customers who wants to use some of the managed services like RDS (meta data store), S3 (artifact storage), Cognito (user managements) etc. I believe mapping out Argo to managed services like aws managed airflow does frees up resources usage of argo side car that runs in the data plane of clusters.