dbt-databricks icon indicating copy to clipboard operation
dbt-databricks copied to clipboard

Update `_wait_for_cluster_to_start` func to check for libraries statuses

Open psivash opened this issue 1 year ago • 0 comments
trafficstars

Resolves #821

Description

Enhance _wait_for_cluster_to_start to wait for library installation

Previously, the _wait_for_cluster_to_start function only checked if the cluster state was 'running'. This update adds a check to ensure all cluster libraries have been successfully installed (or skipped or restored) before proceeding. This ensures the cluster is fully ready for operations that depend on these libraries.

  • Added get_cluster_libraries_status to fetch the status of cluster libraries.
  • Updated _wait_for_cluster_to_start to wait until all libraries are in valid statuses ('INSTALLED', 'RESTORED', 'SKIPPED').

Checklist

  • [x] I have run this code in development and it appears to resolve the stated issue
  • [x] This PR includes tests, or tests are not required/relevant for this PR
  • [ ] I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

psivash avatar Oct 10 '24 08:10 psivash