Multi-tenant and DAG loading mode recommendations

[tianrobin]

Description

1. I need a "workspace" or "DAG Group" to support multi-tenancy, there will be different projects within the organization that I hope can be isolated from each other

2. I expect Airflow to evolve into a platform rather than a product, so I hope it can load DAG files into SCM like Jenkins

   1. Add a confidential dictionary model to store keys
   2. Load the DAG file from the git repository

Use case/motivation

1. I would like to add "workspace" or "DAG Group" and Airflow permission management to isolate different projects. There will be a lot of business and line of business in an organization and there will be a lot of employees involved, and the same is true for data projects, so it is hoped that DAGs that do not pass through the line of business or between projects can be isolated from each other as "workspace" or "DAG Group"

2. I've been using Airflow for a while now, and I'm also trying to explore best practices, and with my current experience I hope Airflow can be a platform. Currently, we have DAG source control in Gitlab and have built the Jenkins pipeline for CI/CD, and we need to use NFS on the Scheduler node and Worker node to implement the distribution of DAG code. When we deploy a DAG, we need a pipeline to perform a "git pull" under the DAG Folder, so I hope Airflow can provide functionality that simplifies the current DAG onboarding process and Scheduler distribution to Worker's DAG code.

Related issues

No

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct