astronomer
Ignore DAGs in the import test when deploying
- [x] I have checked that a similar feature request does not already exist.
✍️ Is your feature request related to a problem? Please describe.
When deploying, the import check for DAG integrity is failing on a file which dynamically generates DAGs. In this file, we use Connection.get_connection_... to get the configuration of the DAG names and schedule intervals that we're generating. When we run the DAG integrity test, it fails because the connection cannot be accessed at this time.
🧩 Describe the solution you'd like
Every other DAG imports OK, and we actually like the have the check for everything except for this single file. We would like to have the ability to ignore a file in the dag_integrity_test.
🤔 Describe alternatives you've considered
We've considered a combination of copying and creating our own version of the test, which mocks this connection for this DAG, and then running that manually with astro dev pytest. Then, if that passes, we can run astro deploy -f which would deploy even with the CLI version of that test failing because we know that our version of that test is passing OK.
Is your feature request specific to a particular Astronomer Platform?
- [x] Astro
- [ ] Software
- [ ] None/Unknown
Couldn't you just add a monkey patch for get_connection? Can you provide an example DAG that we can test?
Hmmm great question. Customer has the DAG, which I can pull if needed. When I'm testing I usually use unittest.mock and related patches from there in context managers. I'll write up a minimal DAG to test with.
from airflow.decorators import dag, task
from pendulum import datetime
from airflow.hooks.base import BaseHook
connection_info = BaseHook().get_connection('my_conn_id')
# Define the DAG
@dag(
description='A sample DAG',
schedule_interval='0 0 * * *',
start_date=datetime(2022, 1, 1),
catchup=False
)
def sample_dag():
@task
def hello_world():
print("Hello World")
hello_world()
sample_dag()
Hi again, I wasn't able to patch. The test_file_imports file is defined as part of the CLI rather than the repo itself. They can test separately and then force deploy, however they like to have the test feature as part of their deploy for the other DAGs in their environment.
I can't seem to find where in the repo the test_file_imports is actually defined. I wrote an update to it that allows users to list files that they want to ignore parse failures on.
@pytest.mark.parametrize(
"rel_path, rv", get_import_errors(), ids=[x[0] for x in get_import_errors()]
)
def test_file_imports(rel_path, rv):
"""Test for import errors on a file"""
if os.path.exists(".astro/dag_integrity_exceptions.txt"):
with open(".astro/dag_integrity_exceptions.txt", "r") as f:
exceptions = f.readlines()
print(f"Exceptions: {exceptions}")
if (rv != "No import errors") and rel_path not in exceptions:
# If rv is not "No import errors," consider it a failed test
raise Exception(f"{rel_path} failed to import with message \n {rv}")
else:
# If rv is "No import errors," consider it a passed test
print(f"{rel_path} passed the import test")
Users should edit their test file located at .astro/test_dag_integrity_default.py to make changes like this