dag-factory
dag-factory copied to clipboard
Not able to generate dynamic dags more than 200
I have 20 folders and all have 100+ dags in each one of them. One of the folders have 201 dags, but airflow dagfactory is creating an issue while parsing more than 200 dags. What is the solution for this?
@kuldeep27396 i'm using AWS MWAA, I have a similar use case, got a question for you: i have defined the boilerplate python file and the corresponding yml file under s3://bucket-name/dags/folder1/dag.py s3://bucket-name/dags/folder1/dagyaml.yml The dag.py file is recognized and MWAA throws error saying not able to find /usr/local/airflow/dags/gagyam.yml but if i place s3://bucket-name/dags/dagyaml.yml, it works. Any inputs or suggestion is appreciated