Secret caching is skipped when DAGs are parsed in Airflow CLI

[Birne94]

Apache Airflow version

main (development)

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Airflow supports the secrets.use_cache option to speed up DAG file processing. According to the documentation, it "enables local caching of Variables, when parsing DAGs only". After enabling this option we found that the DAG processing is still slow when using the Airflow CLI. Looking at the code, the cache is not used at all because SecretCache.init() is only called as part of DAG processing job, not however when fetching DAGs from the CLI.

What you think should happen instead?

When enabling secret caching, I would expect it to be applied for the CLI as well.

How to reproduce

1. enable secrets.use_cache in airflow config
2. have a DAG that accesses variables in top-level code
3. run airflow dags list-import-errors
4. see that it takes very long to execute (20s in my case)

Operating System

Amazon Linux

Versions of Apache Airflow Providers

No response

Deployment

Amazon (AWS) MWAA

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[eladkal]

As explained in https://github.com/apache/airflow/pull/37551#pullrequestreview-1891141876 its not a bug. It's by design. Since its about experimental feature and requires dev mailing list discussion I am closing the issue as the community have no further task here (we try to keep our Github issuew actionable). I encourgae you to follow the instructions left on the PR comment and start a discussion on the mailing list. If your ideas is accepted you can continue working on the PR (no need to have Github issue along side)