`azurerm_arc_kubernetes_cluster` - support for `aad_profile`, `azure_hybrid_benefit`, `arc_agent_auto_upgrade_enabled`, `arc_agent_desired_version` and `kind` properties

[teowa]

Community Note

• Please vote on this PR by adding a :thumbsup: reaction to the original PR to help the community and maintainers prioritize for review
• Please do not leave comments along the lines of "+1", "me too" or "any updates", they generate extra noise for PR followers and do not help prioritize for review

Description

azurerm_arc_kubernetes_cluster - support for aad_profile, azure_hybrid_benefit, arc_agent_auto_upgrade_enabled, arc_agent_desired_version and kind properties

azure doc: part of the https://learn.microsoft.com/en-us/azure/aks/hybrid/aks-create-clusters-portal swagger: https://github.com/Azure/azure-rest-api-specs/blob/main/specification/hybridkubernetes/resource-manager/Microsoft.Kubernetes/stable/2024-01-01/connectedClusters.json

PR Checklist

• [x] I have followed the guidelines in our Contributing Documentation.
• [x] I have checked to ensure there aren't other open Pull Requests for the same update/change.
• [x] I have checked if my changes close any open issues. If so please include appropriate closing keywords below.
• [x] I have updated/added Documentation as required written in a helpful and kind way to assist users that may be unfamiliar with the resource / data source.
• [x] I have used a meaningful PR title to help maintainers and other users understand this change and help prevent duplicate work. For example: “resource_name_here - description of change e.g. adding property new_property_name_here”

Changes to existing Resource / Data Source

• [x] I have added an explanation of what my changes do and why I'd like you to include them (This may be covered by linking to an issue above, but may benefit from additional explanation).
• [x] I have written new tests for my resource or datasource changes & updated any relevent documentation.
• [x] I have successfully run tests with my changes locally. If not, please provide details on testing challenges that prevented you running the tests.
• [ ] (For changes that include a state migration only). I have manually tested the migration path between relevant versions of the provider.

Testing

• [x] My submission includes Test coverage as described in the Contribution Guide and the tests pass. (if this is not possible for any reason, please include details of why you did or could not add test coverage)

TF_ACC=1 go test -v ./internal/services/arckubernetes -parallel 5 -run TestAccArcKubernetesCluster_provisionedCluster -timeout 2h -ldflags="-X=github.com/hashicorp/terraform-provider-azurerm/version.ProviderVersion=acc"
=== RUN   TestAccArcKubernetesCluster_provisionedClusterBasic
=== PAUSE TestAccArcKubernetesCluster_provisionedClusterBasic
=== RUN   TestAccArcKubernetesCluster_provisionedClusterAadProfile
=== PAUSE TestAccArcKubernetesCluster_provisionedClusterAadProfile
=== CONT  TestAccArcKubernetesCluster_provisionedClusterBasic
=== CONT  TestAccArcKubernetesCluster_provisionedClusterAadProfile
--- PASS: TestAccArcKubernetesCluster_provisionedClusterBasic (248.16s)
--- PASS: TestAccArcKubernetesCluster_provisionedClusterAadProfile (394.13s)
PASS
ok      github.com/hashicorp/terraform-provider-azurerm/internal/services/arckubernetes 394.151s

Change Log

Below please provide what should go into the changelog (if anything) conforming to the Changelog Format documented here.

• azurerm_arc_kubernetes_cluster - support for aad_profile, azure_hybrid_benefit, arc_agent_auto_upgrade_enabled, arc_agent_desired_version and kind properties [GH-26916]

This is a (please select all that apply):

• [ ] Bug Fix
• [x] New Feature (ie adding a service, resource, or data source)
• [ ] Enhancement
• [ ] Breaking Change

Related Issue(s)

Fixes #0000

[!NOTE] If this PR changes meaningfully during the course of review please update the title and description as required.

[teowa]

--- PASS: TestAccArcKubernetesCluster_provisionedClusterBasic (228.70s)
--- PASS: TestAccArcKubernetesCluster_provisionedClusterComplete (240.45s)
--- PASS: TestAccArcKubernetesCluster_provisionedClusterAadProfile (321.07s)
--- PASS: TestAccArcKubernetesCluster_provisionedClusterUpdate (325.39s)
PASS
ok      github.com/hashicorp/terraform-provider-azurerm/internal/services/arckubernetes 325.412s

[suvarnanith]

@teowa I am trying to deploy a Azure ARC for Kubernetes. My Kubernetes cluster is in on prem and i want to create the Azure Arc for Kubernetes using terraform.

As per documentation i understand that az connectedk8s connect along with --name “asd” --resource-group “rgname” --location “westeurope” --correlation-id “XXX” --tags “Datacenter City StateOrDistrict CountryOrRegion” needs to be run in the Vm which there is current context is the Kubernetes cluster which we want to connect to Arc. This command will do the helm chart deployment to deploy the agent in Kubernetes cluster and connect to Azure.

But in the below link for terraform i don’t see the option of connection type like when we set deny public access, Use proxy server or private access (Private link scopes needs to be created with private endpoint). Its asking to pass public key. What does this mean?

I have seen that you have done some changes in the code

pass public key - Optional ## Does this mean that once agent is installed in the Kubernetes cluster the public key will be picked by Azure once cluster is connected? Hybrid benefit is added arc_agent_auto_upgrade_enabled is added arc_agent_desired_version is added kind is added

So by adding these parameters in the azurerm_arc_kubernetes_cluster block and running the terraform apply will give us the below command in the terraform output

az connectedk8s connect --name --resource-group --azure-hybrid-benefit --location --tags

This command needs to be run in VM which has a Kubernetes context set. Then the Azure ARC Kubernetes will show up as connected in portal. Is this how it will work?

https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/arc_kubernetes_cluster

If yes then i have questions

1. az connectedk8s connect have multiple options like network connection, usage of proxy etc https://learn.microsoft.com/en-us/cli/azure/connectedk8s?view=azure-cli-latest#az-connectedk8s-connect

2. How az connectedk8s update will work with terraform? https://learn.microsoft.com/en-us/cli/azure/connectedk8s?view=azure-cli-latest#az-connectedk8s-update

3. How it will impact the Arc when any of arguments in terraform azurerm_arc_kubernetes_cluster is changed? How the changes will be updated to Arc agent in the Kubernetes cluster? Do we get the update command and then we need to run it in Kubernetes context?

[teowa]

Hi @ms-zhenhua , thanks for reviewing this! I have changed the code, please kindly take another look.

[teowa]

Hi @ms-zhenhua, thanks for reviewing! I have changed my code, please kindly take another look, thanks!

[ms-zhenhua]

Hi @teowa ,

Thank you for your updates. LGTM~

[teowa]

Hi @stephybun ,

The ProvisionedCluster I aim to support corresponds to Kubernetes clusters created on Azure Stack HCI (now renamed to Azure Local).

Based on a verified module release by the internal team, Kubernetes on Azure Stack HCI includes two APIs as referenced in this module:

• Microsoft.Kubernetes/connectedClusters
• Microsoft.HybridContainerService/provisionedClusterInstances

The second API, provisionedClusterInstances, is a child resource of the first one and must have the resource name default. I have submitted a PR for this API here: #27143.

To improve the user experience, we could consider adding these properties into PR#27143. What do you think?

[stephybun]

Apologies if I've misunderstood your suggestion, but I'm not sure we can add these properties to the resource in #27143 since that resource maps to an entirely different API endpoint. Looking through the links you've provided on the documentation and the module, I think that the resource in #27143 actually needs to be renamed.

To break it down my suggestion for the Azure API to Terraform resource mapping should be:

• Microsoft.Kubernetes/connectedClusters with kind provisionedCluster -> azurerm_arc_kubernetes_provisioned_cluster
• Microsoft.HybridContainerService/provisionedClusterInstances -> azurerm_arc_kubernetes_provisioned_cluster_instance

Does this make sense to you?

[teowa]

Thanks @stephybun , this makes sense, I am going to close this one and support these properties in a new resource azurerm_arc_kubernetes_provisioned_cluster

[stephybun]

Appreciate it @teowa! Feel free to ping me when it's ready for review

[teowa]

Hi @stephybun , I have submitted #28216 to supersede this one.

[github-actions[bot]]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.