Check for usage of Git sync of DAGs on staging and astro cloud

[pankajkoti]

Reach out to @MatrixManAtYrService to understand how their team uses Git sync and plan next set of actions so that we can test our example DAGs quickly on staging and Astro cloud rather than having to build and deploy Docker images for development activities.

[MatrixManAtYrService]

I set this up with the OSS Airflow helm chart before it was implemented in astro cloud, so I don't know where all the buttons are to do it with the astrocloud. But I'm thinking that the UI must prompt for most of these values to get the job done.

dags:
    gitSync:
                branch: main
                enabled: true
                repo: [email protected]:astronomer/qa-tb11c-dags.git
                sshKeySecret: input-ssh-extra
                subPath: dags
            env:
            - name: PYTHONPATH
              value: /opt/airflow/dags/repo/plugins
            - name: AIRFLOW__CORE__PLUGINS_FOLDER
              value: /opt/airflow/dags/repo/plugins
            extraSecrets:
              input-ssh-extra:
                data: 'gitSshKey : LS0tLS1C...base64 encoded key goes here...VZLS0tLS0K'

Once you get in syncing at all, you can get a shell on the scheduler container and poke around until you have a feel for where your dags actually are.

Personally, I don't like syncing the entire qa-scenario-dags repo into $AIRFLOW_HOME/dags because if any dag has parse errors, those errors show up all over the place--so if you're failing tests based on errors in logs, you end up with way more broken tests than you actually have.

Instead, I like to sync the main repo somewhere outside of that folder, and then have whatever orchestrates the test copy just the files it needs for that test into $AIRFLOW_HOME/dags. That way you can attach the right error to the right test. In my case that means copying the files while we're building the dockerfile, which happens here.

Other people on my team might have a better idea of how to get this done on astrocloud since I'm just working with OSS airflow (@himabindu07 or @Priyanka13Astro come to mind).

[phanikumv]

Stale request - @pankajkoti is not actively working on this at the moment