upgrade infrastructure and airflow for version 2.0

[cicerojmm]

Airflow 2.0 in AWS ECS Fargate

Setup to run Airflow 2.0 in AWS ECS containers

Run Airflow 2.0 in AWS we will use ECS (Elastic Container Service) with components in AWS:

• AWS ECS Fargate: run all Airflow services (Webserver, Flower, Workers and Scheduler);
• ElasticCache (Redis): communication between Airflow Services;
• RDS for Postgres: database MetadataDB for Airflow servies;
• EFS: persistent storage for Airflow dags;
• ELB: Application Load Balance for Airflow WebServer access;
• CloudWatch: logs for container services and Airflow run tasks;
• IAM: communications services permission for ECS containers;
• ECR: image repository Docker for storage Airflow images.

[nicor88]

@cicerojmm thanks for your work, I'm not a big fan of using EFS to mount the dags. Instead, I prefer to bake new images and release the changes. Overall it looks good, I just need to test it out.

[cicerojmm]

@cicerojmm thanks for your work, I'm not a big fan of using EFS to mount the dags. Instead, I prefer to bake new images and release the changes. Overall it looks good, I just need to test it out.

Thanks @nicor88 ! About EFS, do you see any problem using it or do you just prefer it? Because sometimes we have a requirement to synchronize dags directly from S3. In this case would you do otherwise?