Hardcoded virtualnetworklink suffix causes issues if virtualnetworklink already exists with different name

[frank-m]

What happened:

https://github.com/kubernetes-sigs/cloud-provider-azure/blob/3e881836b693575d3c541cbe6e63186596b162b3/pkg/provider/azure_storageaccount.go#L365-L375

This piece of code assumes that all virtual networks have a name with the '-vnetlink' suffix.

If this is not the case it will try to create the vnetlink, but if a different link already exists for the same vnet. It will fail.

What you expected to happen:

The code should get all virtualnetworklinks and check if one already exists for that vnet (preferably with the Resource ID of the vNet). If it exists, it should not try to create a new virtualnetworklink as only one can exist per vnet.

How to reproduce it (as minimally and precisely as possible):

We hit this issue while upgrading from AKS 1.27 --> 1.28. This also upgrades the CSI drivers, which use this provider. Initially you notice no issues, but when you try to do any actions on your PVC it will report in the logs with errors about not being able to create the virtualnetworklink.

Anything else we need to know?:

The workaround is to manually delete the vnetlink and recreate it with the '-vnetlink' suffix, however that causes a temporary failure in name resolution (depending on TTL of records)

Environment:

• Kubernetes version (use kubectl version): happened after upgrading from AKS 1.27 --> 1.28.9
• Cloud provider or hardware configuration: Azure AKS
• OS (e.g: cat /etc/os-release): Ubuntu
• Kernel (e.g. uname -a): -
• Install tools: -
• Network plugin and version (if this is a network-related bug):-
• Others:

[Mik4sa]

I think I have the same issue. My upgrade path was 1.27.3 > 1.28.x (don't know which exact patch version > 1.29.5

This is the error I get on a newly created pvc:

Events:
  Type     Reason                Age               From                                                                                               Message
  ----     ------                ----              ----                                                                                               -------
  Normal   ExternalProvisioning  10s               persistentvolume-controller                                                                        Waiting for a volume to be created either by the external provisioner 'file.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal   Provisioning          2s (x4 over 10s)  file.csi.azure.com_csi-azurefile-controller-[...]  External provisioner is provisioning volume for claim "[...]"
  Warning  ProvisioningFailed    1s (x4 over 10s)  file.csi.azure.com_csi-azurefile-controller-[...]  failed to provision volume with StorageClass "my-custom-storage-class": rpc error: code = Internal desc = failed to ensure storage account: get virtual link for vnet(aks-vnet-14154989) and DNS Zone(privatelink.file.core.windows.net) in resourceGroup([...]) returned with GET http://localhost:7788/subscriptions/[...]/resourceGroups/[...]/providers/Microsoft.Network/privateDnsZones/privatelink.file.core.windows.net/virtualNetworkLinks/aks-vnet-14154989-vnetlink
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
  "code": "NotFound",
  "message": "The virtual network link 'aks-vnet-14154989-vnetlink' does not exist for Private DNS zone 'privatelink.file.core.windows.net' in resource group '[...]' of subscription '[...]'."
}
--------------------------------------------------------------------------------

[Mik4sa]

After updating to 1.29.6 I get this error message:

  Warning  ProvisioningFailed  119s                 file.csi.azure.com_csi-azurefile-controller-[...]  failed to provision volume with StorageClass "my-custom-storage-class": rpc error: code = Internal desc = failed to ensure storage account: get private dns zone privatelink.file.core.windows.net returned with GET http://localhost:7788/subscriptions/[...]/resourceGroups/[...]/providers/Microsoft.Network/privateDnsZones/privatelink.file.core.windows.net
--------------------------------------------------------------------------------
RESPONSE 401: 401 Unauthorized
ERROR CODE: ExpiredAuthenticationToken
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ExpiredAuthenticationToken",
    "message": "The access token expiry UTC time '8/2/2024 5:22:59 AM' is earlier than current UTC time '8/2/2024 7:31:24 AM'."
  }
}
--------------------------------------------------------------------------------

[Mik4sa]

After updating to 1.29.7 I was back to the error as in 1.29.5

I recreated the link now with the currently expected name. It worked then as expected.

[nilo19]

@andyzhangx @cvvz could you help check this one?

[pngfei]

I have same issue on 1.30.4.

[andyzhangx]

in this case, is privatelink.file.core.windows.net created by yourself or CSI driver? @frank-m

[gil-tohar-cyera]

I see this issue as well in cases where the private dns zone was created independently. In version 1.29.2 it worked, where it discovered the correct private dns zone virtual network link without requiring a certain naming convention.

It shouldn't require a hardcoded name, it should use the link that already exists, without enforcing a naming convention on the resource.

I tested on multiple versions, including the latest 1.31.2, and still encountered the same issue

[frank-m]

@andyzhangx Apologies for the slow reply. The private dns zone was created by ourselves.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[nilo19]

@andyzhangx do we still want to fix this?

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.