airflow-operator icon indicating copy to clipboard operation
airflow-operator copied to clipboard
verified publisher icon GoogleCloudPlatform

git-sync on airflow does not support SSH authentication

Open raam86 opened this issue 7 years ago • 7 comments

The current object only supports the username + password authentication method: https://github.com/GoogleCloudPlatform/airflow-operator/blob/1fbc02dbbaefd70b43f1db5b4a51e3d5ab6cd537/pkg/apis/airflow/v1alpha1/utils.go#L307

in order to fully support git-sync we need to 1. set GIT_SYNC_SSH to true and map a mounted volume that points to the ConfigMap: https://github.com/kubernetes/git-sync/blob/9ceb61f7947fbe463b1cc6e9ae5d719f5d8eebd2/docs/ssh.md#step-3-configure-git-sync-container

raam86 avatar Nov 05 '18 15:11 raam86

Would you have time to take this up ? @raam86 I can help you with the setup and tests

barney-s avatar Nov 20 '18 09:11 barney-s

After some more research I opted for GCS sidecar container, I think it's the best of all worlds. Basically lifted your setup into the helm/charts/incubator

raam86 avatar Nov 21 '18 13:11 raam86

Thanks. What did you mean by lifted you setup into helm/charts ?

If you are using both i would love to hear you feedback on helm/charts vs the operator.

barney-s avatar Nov 30 '18 18:11 barney-s

I initially used the operator and got quite excited but was stuck when trying to customize it. I specifically wanted to have the gcs sync as a sidecar instead of an init container and it was easier to do using the helm chart. Since I realized I am going to do work anyway I wanted it to be less google specific so I opted for the helm chart. the helm chart also has more users, more support and is more standard

raam86 avatar Dec 04 '18 13:12 raam86

Ah i see. GCS sync is a side-car. The default value of .spec.dags.gcs.once is false. Which helm chart are you using ?

barney-s avatar Dec 04 '18 18:12 barney-s

The one that just became stable. You are definitely right about the defaults. I think that what actually happened was that I tried using git sync realized I can’t opted for the helm chart and eventually opted for gcs. Sorry about the confusion. But the underlying reason for continuing using the helm chart instead of the operator is that the helm chart uses standard and available kubernetes concepts while the operator is minting bew ones I don’t have lots of incentive to learn

raam86 avatar Dec 06 '18 21:12 raam86

hi there, since this is the first Google result when looking for SSH auth for git-sync on Airflow just wanted to let you know I've been working on adding this to the Airflow project:

https://issues.apache.org/jira/browse/AIRFLOW-3918 https://github.com/apache/airflow/pull/4777

dmateusp avatar Feb 25 '19 22:02 dmateusp