ucx
ucx copied to clipboard
databrickslabs
[FEATURE]: DBR upgrade mechanism
There is no systematic way to upgrade DBR. Often, the first step large customers need to take is a DBR upgrade to move to a UC-supported runtime. Note that this feature could (and should) be built for the generic case, independent of UC migrations as historically it has always been a field need.
Related issues:
- [ ] https://github.com/databrickslabs/ucx/issues/1124
- [ ] #1122
Proposed Solution
The feature should:
- Evaluate and display unsupported DBR impact preventing UC migration
- Migrate clusters and associated policies to new target runtime
- Validate successful migration and associated workloads.
This can/should be rolled into #1122, where validation should be done via:
- Successful change of targeted cluster configuration.
- Successful execution of targeted workflows.
and should surface the following information:
- Errors upon failure of DBR upgrade
- Migration rollup success metrics (% clusters migrated successfully, % migrated jobs executed successfully, % migrated workflows executed successfully)
- Tables with item-level success metrics e.g. for clusters (cluster_id | migrated_flag | error_message) and for workflows (job_id, execution_success, error_message)
Additional Context
For one enterprise customer, it took nearly 12 months to move from 7.3LTS to 9.1LTS.
This seems like a HUGE feature request. Like it could be it's own Labs project. I'm not suggesting it's not important, just a large amount of work; and somewhat independent to the goals of UCX.
@pohlposition completely agree. I was hoping to use this as a catalyst, but I suppose as Serverless adoption/usage accelerates there may not be a need in a few years. Maybe a "lighter" version to support UC migrations, specifically?
For ref, we do have this (per @nfx's ask on upgrade documentation): https://dbrmg.databricks.com/?cloud_provider=aws So, it would be more about applying translation of these, or UC-specific changes (not clearly documented from what I can see).