Kubernetes contrib : waiting for cluster scale up

[dav009]

Description

Makes sure Luigi waits for kubernetes to scale up when waiting for a job run.

Motivation and Context

Sometimes scheduling a new job results in the cluster needing to scale up. In these scenarios current luigi throws an exception which exits the pipeline. In the meantime the cluster runs the job after scaling up. Changes introduced in this PR makes it so that Luigi waits for Scale up to take place when waiting for a job to run.

Have you tested this? If so, how?

• unit tests included in this PR
• scheduling a task for my pipepelines

[dav009]

@dlstadther @dataex any hints on why the tests might be failing ? I did not touch anything related to hadoop:

FAILED test/contrib/hadoop_test.py::CreatePackagesArchive::test_create_packages_archive_module
FAILED test/contrib/hadoop_test.py::CreatePackagesArchive::test_create_packages_archive_package
FAILED test/contrib/hadoop_test.py::CreatePackagesArchive::test_create_packages_archive_package_submodule
FAILED test/contrib/hadoop_test.py::CreatePackagesArchive::test_create_packages_archive_package_submodule_with_absolute_import
FAILED test/contrib/hadoop_test.py::CreatePackagesArchive::test_create_packages_archive_package_submodule_without_imports
FAILED test/contrib/hadoop_test.py::CreatePackagesArchive::test_create_packages_archive_package_subpackage
FAILED test/contrib/hadoop_test.py::CreatePackagesArchive::test_create_packages_archive_package_subpackage_submodule

[tnsetting]

It was because of a python version update in github action. From 3.8.9 to 3.8.10. From 3.9.4 to 3.9.5.

[tnsetting]

There is a PR #3093 for change the kube client. Maybe it can be incorporated with that PR #3093

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If closed, you may revisit when your time allows and reopen! Thank you for your contributions.