dag-factory icon indicating copy to clipboard operation
dag-factory copied to clipboard

Not able to generate dynamic dags more than 200

Open kuldeep27396 opened this issue 2 years ago • 1 comments

I have 20 folders and all have 100+ dags in each one of them. One of the folders have 201 dags, but airflow dagfactory is creating an issue while parsing more than 200 dags. What is the solution for this?

kuldeep27396 avatar Oct 18 '22 12:10 kuldeep27396

@kuldeep27396 i'm using AWS MWAA, I have a similar use case, got a question for you: i have defined the boilerplate python file and the corresponding yml file under s3://bucket-name/dags/folder1/dag.py s3://bucket-name/dags/folder1/dagyaml.yml The dag.py file is recognized and MWAA throws error saying not able to find /usr/local/airflow/dags/gagyam.yml but if i place s3://bucket-name/dags/dagyaml.yml, it works. Any inputs or suggestion is appreciated

satishgvgit avatar Oct 28 '22 13:10 satishgvgit