Move to v3.Cluster

[kkaempf]

Description

After adding support for importing CAPI clusters into Rancher via the clusters.management.cattle.io (also known as v3.Cluster) resource (refer to PR), we would like to make this the default solution and deprecate the existing import mechanism, based on clusters.provisioning.cattle.io.

The default configuration of Turtles keeps the v3 functionality behind a feature gate managementv3-cluster and is only activated if explicitly specified, as its default value is false.

To Do

Fully migrating to using v3 clusters means that we must:

• Extend the E2E test cases that cover clusters.management.cattle.io to the existing clusters.provisioning.cattle.io: this means moving existing test suites to be based on the "new" Rancher cluster resource. This will also guarantee that the new Turtles controller works for every environment we're covering in E2E.
• Investigate a migration strategy for those that are already using Turtles and have clusters imported into Rancher: old clusters backed by provisioning.cattle.io should be migrated to management.cattle.io transparently. Propose solution as a Turtles ADR to reach agreement as a team. This ADR should also specify the deprecation procedure for the existing controller.
• Implement migration procedure: after agreeing on a migration strategy, we should create a migration tool/script/process to provide users with a guide on how to upgrade their clusters.
• Remove feature gate and make v3 clusters the default resource for importing CAPI clusters to Rancher.

Child issues

• [ ] https://github.com/rancher/turtles/issues/475
• [ ] https://github.com/rancher/turtles/issues/477
• [ ] https://github.com/rancher/turtles/issues/478
• [ ] https://github.com/rancher/turtles/issues/479