airflow icon indicating copy to clipboard operation
airflow copied to clipboard

Published 20 hours ago • verified publisher icon apache

Add missing test modules in our unit test suite

Open potiuk opened this issue 2 years ago • 16 comments
trafficstars

See test_providers_modules_should_have_tests in tests/always/test_project_structure.py but those tests are missing currently:

Provider amazon

codecov

  • [ ] tests/providers/amazon/aws/executors/ecs/test_boto_schema.py
  • [ ] tests/providers/amazon/aws/executors/ecs/test_ecs_executor_config.py
  • [ ] tests/providers/amazon/aws/executors/ecs/test_utils.py
  • [x] tests/providers/amazon/aws/fs/test_s3.py
  • [x] tests/providers/amazon/aws/hooks/test_dms.py
  • [x] tests/providers/amazon/aws/links/test_base_aws.py
  • [x] tests/providers/amazon/aws/links/test_batch.py
  • [x] tests/providers/amazon/aws/links/test_emr.py
  • [x] tests/providers/amazon/aws/links/test_glue.py
  • [x] tests/providers/amazon/aws/links/test_logs.py
  • [x] tests/providers/amazon/aws/operators/test_dms.py
  • [ ] tests/providers/amazon/aws/operators/test_emr.py
  • [x] tests/providers/amazon/aws/operators/test_s3.py
  • [ ] tests/providers/amazon/aws/operators/test_sagemaker.py
  • [x] tests/providers/amazon/aws/sensors/test_dms.py
  • [ ] tests/providers/amazon/aws/sensors/test_emr.py
  • [x] tests/providers/amazon/aws/sensors/test_s3.py
  • [ ] tests/providers/amazon/aws/sensors/test_sagemaker.py
  • [ ] tests/providers/amazon/aws/test_exceptions.py
  • [x] tests/providers/amazon/aws/triggers/test_athena.py
  • [x] tests/providers/amazon/aws/triggers/test_batch.py
  • [ ] tests/providers/amazon/aws/triggers/test_eks.py
  • [x] tests/providers/amazon/aws/triggers/test_emr.py
  • [x] tests/providers/amazon/aws/triggers/test_glue_crawler.py
  • [x] tests/providers/amazon/aws/triggers/test_lambda_function.py
  • [ ] tests/providers/amazon/aws/triggers/test_rds.py
  • [x] tests/providers/amazon/aws/triggers/test_redshift_cluster.py
  • [ ] tests/providers/amazon/aws/triggers/test_step_function.py
  • [ ] tests/providers/amazon/aws/utils/test_rds.py
  • [ ] tests/providers/amazon/aws/utils/test_sagemaker.py
  • [x] tests/providers/amazon/aws/utils/test_sqs.py
  • [x] tests/providers/amazon/aws/utils/test_tags.py
  • [ ] tests/providers/amazon/aws/waiters/test_base_waiter.py

Provider apache.cassandra

codecov

  • [ ] tests/providers/apache/cassandra/hooks/test_cassandra.py

Provider apache.drill

codecov

  • [ ] tests/providers/apache/drill/operators/test_drill.py

Provider apache.druid

codecov

  • [ ] tests/providers/apache/druid/operators/test_druid_check.py

Provider apache.hdfs

codecov

  • [ ] tests/providers/apache/hdfs/hooks/test_hdfs.py
  • [ ] tests/providers/apache/hdfs/log/test_hdfs_task_handler.py
  • [ ] tests/providers/apache/hdfs/sensors/test_hdfs.py

Provider apache.hive

codecov

  • [ ] tests/providers/apache/hive/plugins/test_hive.py
  • [ ] tests/providers/apache/hive/transfers/test_mssql_to_hive.py

Provider apache.kafka

codecov

  • [ ] tests/providers/apache/kafka/hooks/test_base.py
  • [x] tests/providers/apache/kafka/hooks/test_client.py
  • [x] tests/providers/apache/kafka/hooks/test_consume.py
  • [x] tests/providers/apache/kafka/hooks/test_produce.py

Provider celery

codecov

  • [ ] tests/providers/celery/executors/test_celery_executor_utils.py
  • [ ] tests/providers/celery/executors/test_default_celery.py

Provider cncf.kubernetes

codecov

  • [ ] tests/providers/cncf/kubernetes/backcompat/test_backwards_compat_converters.py
  • [ ] tests/providers/cncf/kubernetes/executors/test_kubernetes_executor_types.py
  • [ ] tests/providers/cncf/kubernetes/executors/test_kubernetes_executor_utils.py
  • [ ] tests/providers/cncf/kubernetes/operators/test_kubernetes_pod.py
  • [ ] tests/providers/cncf/kubernetes/test_k8s_model.py
  • [ ] tests/providers/cncf/kubernetes/test_kube_client.py
  • [ ] tests/providers/cncf/kubernetes/test_kube_config.py
  • [ ] tests/providers/cncf/kubernetes/test_pod_generator_deprecated.py
  • [ ] tests/providers/cncf/kubernetes/test_pod_launcher_deprecated.py
  • [ ] tests/providers/cncf/kubernetes/test_python_kubernetes_script.py
  • [ ] tests/providers/cncf/kubernetes/test_secret.py
  • [ ] tests/providers/cncf/kubernetes/triggers/test_kubernetes_pod.py
  • [ ] tests/providers/cncf/kubernetes/utils/test_delete_from.py
  • [ ] tests/providers/cncf/kubernetes/utils/test_k8s_hashlib_wrapper.py
  • [ ] tests/providers/cncf/kubernetes/utils/test_xcom_sidecar.py

Provider common.io

codecov

  • [x] tests/providers/common/io/operators/test_file_transfer.py

Provider databricks

codecov

  • [ ] tests/providers/databricks/hooks/test_databricks_base.py

Provider dbt.cloud

codecov

  • [x] tests/providers/dbt/cloud/hooks/test_dbt.py
  • [x] tests/providers/dbt/cloud/operators/test_dbt.py
  • [x] tests/providers/dbt/cloud/sensors/test_dbt.py
  • [x] tests/providers/dbt/cloud/triggers/test_dbt.py

Provider docker

codecov

  • [ ] tests/providers/docker/test_exceptions.py

Provider elasticsearch

codecov

  • [x] tests/providers/elasticsearch/log/test_es_json_formatter.py
  • [x] tests/providers/elasticsearch/log/test_es_response.py

Provider google

codecov

  • [ ] tests/providers/google/cloud/fs/test_gcs.py
  • [ ] tests/providers/google/cloud/links/test_automl.py
  • [ ] tests/providers/google/cloud/links/test_base.py
  • [ ] tests/providers/google/cloud/links/test_bigquery.py
  • [ ] tests/providers/google/cloud/links/test_bigquery_dts.py
  • [ ] tests/providers/google/cloud/links/test_bigtable.py
  • [ ] tests/providers/google/cloud/links/test_cloud_build.py
  • [ ] tests/providers/google/cloud/links/test_cloud_functions.py
  • [ ] tests/providers/google/cloud/links/test_cloud_memorystore.py
  • [ ] tests/providers/google/cloud/links/test_cloud_sql.py
  • [ ] tests/providers/google/cloud/links/test_cloud_storage_transfer.py
  • [ ] tests/providers/google/cloud/links/test_cloud_tasks.py
  • [ ] tests/providers/google/cloud/links/test_compute.py
  • [ ] tests/providers/google/cloud/links/test_data_loss_prevention.py
  • [ ] tests/providers/google/cloud/links/test_datacatalog.py
  • [ ] tests/providers/google/cloud/links/test_dataflow.py
  • [ ] tests/providers/google/cloud/links/test_dataform.py
  • [ ] tests/providers/google/cloud/links/test_datafusion.py
  • [ ] tests/providers/google/cloud/links/test_dataplex.py
  • [ ] tests/providers/google/cloud/links/test_dataprep.py
  • [ ] tests/providers/google/cloud/links/test_dataproc.py
  • [ ] tests/providers/google/cloud/links/test_datastore.py
  • [ ] tests/providers/google/cloud/links/test_kubernetes_engine.py
  • [ ] tests/providers/google/cloud/links/test_life_sciences.py
  • [ ] tests/providers/google/cloud/links/test_mlengine.py
  • [ ] tests/providers/google/cloud/links/test_pubsub.py
  • [ ] tests/providers/google/cloud/links/test_spanner.py
  • [ ] tests/providers/google/cloud/links/test_stackdriver.py
  • [ ] tests/providers/google/cloud/links/test_vertex_ai.py
  • [ ] tests/providers/google/cloud/links/test_workflows.py
  • [ ] tests/providers/google/cloud/operators/vertex_ai/test_auto_ml.py
  • [ ] tests/providers/google/cloud/operators/vertex_ai/test_batch_prediction_job.py
  • [ ] tests/providers/google/cloud/operators/vertex_ai/test_custom_job.py
  • [ ] tests/providers/google/cloud/operators/vertex_ai/test_dataset.py
  • [ ] tests/providers/google/cloud/operators/vertex_ai/test_endpoint_service.py
  • [ ] tests/providers/google/cloud/operators/vertex_ai/test_hyperparameter_tuning_job.py
  • [ ] tests/providers/google/cloud/operators/vertex_ai/test_model_service.py
  • [ ] tests/providers/google/cloud/operators/vertex_ai/test_pipeline_job.py
  • [ ] tests/providers/google/cloud/sensors/test_dataform.py
  • [ ] tests/providers/google/cloud/transfers/test_bigquery_to_sql.py
  • [ ] tests/providers/google/cloud/transfers/test_presto_to_gcs.py
  • [ ] tests/providers/google/cloud/transfers/test_trino_to_gcs.py
  • [ ] tests/providers/google/cloud/triggers/test_cloud_composer.py
  • [ ] tests/providers/google/cloud/transfers/test_bigquery_to_mssql.py
  • [ ] tests/providers/google/cloud/transfers/test_mssql_to_gcs.py
  • [ ] tests/providers/google/cloud/utils/test_bigquery.py
  • [ ] tests/providers/google/cloud/utils/test_bigquery_get_data.py
  • [ ] tests/providers/google/cloud/utils/test_dataform.py
  • [ ] tests/providers/google/common/links/test_storage.py
  • [ ] tests/providers/google/common/test_consts.py
  • [ ] tests/providers/google/test_go_module_utils.py

Provider microsoft.azure

codecov

  • [ ] tests/providers/microsoft/azure/fs/test_adls.py
  • [x] tests/providers/microsoft/azure/hooks/test_batch.py
  • [x] tests/providers/microsoft/azure/hooks/test_container_instance.py
  • [x] tests/providers/microsoft/azure/hooks/test_container_registry.py
  • [x] tests/providers/microsoft/azure/hooks/test_container_volume.py
  • [x] tests/providers/microsoft/azure/hooks/test_cosmos.py
  • [x] tests/providers/microsoft/azure/hooks/test_data_factory.py
  • [x] tests/providers/microsoft/azure/hooks/test_data_lake.py
  • [x] tests/providers/microsoft/azure/hooks/test_fileshare.py
  • [x] tests/providers/microsoft/azure/hooks/test_synapse.py
  • [ ] tests/providers/microsoft/azure/operators/test_adls.py
  • [x] tests/providers/microsoft/azure/operators/test_batch.py
  • [x] tests/providers/microsoft/azure/operators/test_container_instances.py
  • [x] tests/providers/microsoft/azure/operators/test_cosmos.py
  • [x] tests/providers/microsoft/azure/operators/test_data_factory.py
  • [x] tests/providers/microsoft/azure/operators/test_synapse.py
  • [x] tests/providers/microsoft/azure/secrets/test_key_vault.py
  • [x] tests/providers/microsoft/azure/sensors/test_cosmos.py
  • [x] tests/providers/microsoft/azure/sensors/test_data_factory.py
  • [ ] tests/providers/microsoft/azure/transfers/test_azure_blob_to_gcs.py
  • [x] tests/providers/microsoft/azure/triggers/test_data_factory.py
  • [ ] tests/providers/microsoft/azure/triggers/test_wasb.py

Provider mongo

codecov

  • [ ] tests/providers/mongo/sensors/test_mongo.py

Provider openlineage

codecov

  • [x] tests/providers/openlineage/extractors/test_base.py
  • [x] tests/providers/openlineage/extractors/test_bash.py
  • [ ] tests/providers/openlineage/extractors/test_manager.py
  • [x] tests/providers/openlineage/extractors/test_python.py
  • [ ] tests/providers/openlineage/plugins/test_adapter.py
  • [ ] tests/providers/openlineage/plugins/test_facets.py
  • [ ] tests/providers/openlineage/plugins/test_macros.py
  • [ ] tests/providers/openlineage/test_sqlparser.py

Provider presto

codecov

  • [x] tests/providers/presto/transfers/test_gcs_to_presto.py

Provider redis

codecov

  • [ ] tests/providers/redis/operators/test_redis_publish.py
  • [ ] tests/providers/redis/sensors/test_redis_key.py

Provider slack

codecov

  • [ ] tests/providers/slack/notifications/test_slack_notifier.py

Provider snowflake

codecov

  • [ ] tests/providers/snowflake/triggers/test_snowflake_trigger.py

Provider trino

codecov

  • [x] tests/providers/trino/transfers/test_gcs_to_trino.py

Committer

  • [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

potiuk avatar Nov 04 '23 19:11 potiuk

cc: @eladkal

potiuk avatar Nov 04 '23 19:11 potiuk

@eladkal -> first test removed from the list with #35457 -> I hope you can bring more people to contribute here :)

potiuk avatar Nov 05 '23 13:11 potiuk

Also related to #35127

potiuk avatar Nov 05 '23 14:11 potiuk

cc: @ephraimbuddy -> this might guide people on where to add tests to cover whole modules that are not "targeted" for tests (they might still be accidentally tested by other modules but they do not have dedicated module tests) - this might help people to make decision on where to add some tests. Also this is a nice list of "todos"

potiuk avatar Nov 05 '23 14:11 potiuk

tests/providers/slack/notifications/test_slack_notifier.py

I think how to exclude case when the original module airflow.providers.slack.notifications.slack_notifier is deprecated

Taragolis avatar Nov 05 '23 16:11 Taragolis

And just add here previous investigation about missing tests https://github.com/apache/airflow/pull/28459#discussion_r1054708003, non all of them actually missing:

  • some of tests moved into tests.integrations
  • Amazon links tested in one go in same module
  • Some of amazon tests split across different test modules, for example: test_s3_bucket.py + test_s3_bucket_tagging.py + test_s3_file_transform.py + test_s3_list.py + test_s3_list_prefixes.py + test_s3_object.py but module named as s3.py (merged version)

Taragolis avatar Nov 05 '23 16:11 Taragolis

And codecov statistic also might help to found untested parts: https://app.codecov.io/gh/apache/airflow/tree/main/airflow%2Fproviders

Taragolis avatar Nov 05 '23 16:11 Taragolis

tests/providers/slack/notifications/test_slack_notifier.py

I think how to exclude case when the original module airflow.providers.slack.notifications.slack_notifier is deprecated

In similar cases we used to have deprecated classes list and we removed it from the output

eladkal avatar Nov 05 '23 17:11 eladkal

I think how to exclude case when the original module airflow.providers.slack.notifications.slack_notifier is deprecated

Absolutely - I think there are many false-negatives there, so removal them by finding and automated way and improving the test case is absolutely legitimate way of cleaning some of those. PRs are most welcome :). I just revived the test really quickly - also in order to not allow more of new ones creep in, but I think we should definitely remove some of those from the list by making the test "smarter"

And codecov statistic also might help to found untested parts: https://app.codecov.io/gh/apache/airflow/tree/main/airflow%2Fproviders

Oh absolutely - this is why I added my comment https://github.com/apache/airflow/issues/35442#issuecomment-1793747405 and updated https://github.com/apache/airflow/issues/35127 with the comment that these two are connected. I see that as two sides of the same coing - eventually they should converge - the missing coverage should quite closely show similar list as the one generated here. Currently they are far-off - for the reasons you explained, but also because some of the module's code are tested by some other tests. - but this is precisely I think what should guide us in making the test smarter - eventually they should show the same result.

And just add here previous investigation about missing tests https://github.com/apache/airflow/pull/28459#discussion_r1054708003, non all of them actually missing:

some of tests moved into tests.integrations Amazon links tested in one go in same module Some of amazon tests split across different test modules, for example: test_s3_bucket.py + test_s3_bucket_tagging.py + test_s3_file_transform.py + test_s3_list.py + test_s3_list_prefixes.py + test_s3_object.py but module named as s3.py (merged version)

Yeah - why don't we automate that in the test then? I think some of them can be easily automatically detected.

potiuk avatar Nov 05 '23 19:11 potiuk

If you don't mind I've split this list by providers

Taragolis avatar Nov 05 '23 20:11 Taragolis

If you don't mind I've split this list by providers

Not at all ! Very helpful I think :)

potiuk avatar Nov 05 '23 20:11 potiuk

Super useful with codecov links BTW! Good idea.

potiuk avatar Nov 05 '23 20:11 potiuk

Hi! I found that the classes present in the following modules are already covered with tests, but they are unchecked in the issue's first comment:

tests/providers/amazon/aws/operators/test_emr.py tests/providers/amazon/aws/operators/test_sagemaker.py tests/providers/amazon/aws/sensors/test_emr.py tests/providers/amazon/aws/triggers/test_athena.py tests/providers/amazon/aws/triggers/test_batch.py tests/providers/amazon/aws/triggers/test_emr.py tests/providers/amazon/aws/triggers/test_glue_crawler.py tests/providers/amazon/aws/triggers/test_lambda_function.py tests/providers/amazon/aws/triggers/test_rds.py tests/providers/amazon/aws/triggers/test_redshift_cluster.py tests/providers/amazon/aws/utils/test_rds.py tests/providers/amazon/aws/utils/test_sagemaker.py tests/providers/amazon/aws/utils/test_sqs.py tests/providers/amazon/aws/utils/test_tags.py tests/providers/amazon/aws/waiters/test_base_waiter.py tests/providers/amazon/aws/triggers/test_step_function.py

If it is possible, it would be nice if anyone could check them, so that who is looking to contribute can find missing tests more easily. Thanks!

CC: @potiuk @Taragolis

mateuslatrova avatar Apr 27 '24 00:04 mateuslatrova

@mateuslatrova Some test modules from your list do not exists or named differently

Exists

I've marked this as completed

Not Exists

Taragolis avatar Apr 30 '24 12:04 Taragolis

@mateuslatrova Some test modules from your list do not exists or named differently

Exists

I've marked this as completed

Not Exists

Hi @Taragolis ! Thanks for your help.

I guess I was not so clear, but the corresponding code of each of that non-existing files is covered in other test files, and that is why I put those ones on the list.

For example, one test file from the above list that does not exist is tests/providers/amazon/aws/operators/test_emr.py. This test file would cover the code present in airflow/providers/amazon/aws/operators/emr.py , right? But this code is being covered in other test files:

  • tests/providers/amazon/aws/operators/test_emr_add_steps.py
  • tests/providers/amazon/aws/operators/test_emr_containers.py
  • tests/providers/amazon/aws/operators/test_emr_create_job_flow.py
  • tests/providers/amazon/aws/operators/test_emr_modify_cluster.py
  • tests/providers/amazon/aws/operators/test_emr_notebook_execution.py
  • tests/providers/amazon/aws/operators/test_emr_serverless.py
  • tests/providers/amazon/aws/operators/test_terminate_job_flow.py

So the tests/providers/amazon/aws/operators/test_emr.py should not be in that list.

mateuslatrova avatar May 01 '24 13:05 mateuslatrova

There is pretty simple rule there if providers module placed into the airflow/providers/amazon/aws/operators/emr.py then tests should be placed into the tests/providers/amazon/aws/operators/test_emr.py.

Original check was broken for a while (1-2 years or so), so we have about 100 modules which do not follow this one.

Taragolis avatar May 01 '24 16:05 Taragolis