kedro-plugins icon indicating copy to clipboard operation
kedro-plugins copied to clipboard
verified publisher icon kedro-org

`kedro-datasets`: Investigate `spark.SparkHiveDataset` flaky unit tests

Open ankatiyar opened this issue 1 year ago • 3 comments

Description

The tests for spark.SparkHiveDataset have been failing intermittently during nightly builds and checks for PRs. Usually re-running them solves the problem but we should investigate the root cause.

Context

Eg for a failed run - https://github.com/kedro-org/kedro-plugins/actions/runs/9807280152/job/27080744636 Seems to be related to the SparkSession setup.

ankatiyar avatar Jul 08 '24 10:07 ankatiyar

Hi @ankatiyar , I have opened a PR with a possible solution. Could you check it please?. I am not sure on how to re-run the actions to test it again.

felipemonroy avatar Jul 20 '24 21:07 felipemonroy

I am not sure how to fix it. I assume some tests use an unintended spark session (like the ones in test_spark_hive_dataset using one without hive support). Do we have the option of running tests/spark without xdist as the ones in tests/tensorflow/test_tensorflow_model_dataset.py?

felipemonroy avatar Jul 22 '24 14:07 felipemonroy

@felipemonroy we can investigate this at our end, if in the meantime your tests are failing on other PRs with this error, tag someone from the team (or me) we can re-run the tests for the PR. Locally, you can run make test-no-spark I think to run all the other unit tests

ankatiyar avatar Jul 22 '24 16:07 ankatiyar