trident
trident copied to clipboard
Published
20 hours ago •
NetApp
NetApp
Trident CSI not provisioning volumes correctly and producing exit code 32
Describe the bug it appears that the CSI is unable to mount the NFS volume to a PVC in azure, I am unsure as to why this is as NFS is installed by default in every node in the AKS vmss cluster
Environment Provide accurate information about the environment to help us reproduce the issue.
- Trident version: latest via
- Trident installation flags used: [e.g. -d -n trident --use-custom-yaml]
- Container runtime: [e.g. Docker 19.03.1-CE]
- Kubernetes version: [e.g. 1.23.1]
- Kubernetes orchestrator: [AKS]
- Kubernetes enabled feature gates: [?]
- OS: [Ubuntu 18.04]
- NetApp backend types: [Net app CSI Trident Operatror Azure CRD's]
- Other:
To Reproduce Deploy WordPress bitnami helm chart as RWX with cluster autoscaling enabled along with Promeuthus and grafana This results in error 32 codes it also looks like Trident expects a minimum of 100GB to be provisioned per volume but I think this value should really be 100G or 100Gi, I'm not sure exactly how to provision 100GB of storage via kubernetes
Expected behavior NFS volume in Netapp mounted to the pod in running state
Additional context
PS C:\Users\danny> kubectl get po --kubeconfig config
NAME READY STATUS RESTARTS AGE
wordpress-7b68f9d9bc-9nz27 0/2 ContainerCreating 0 31m
wordpress-memcached-774f78cc49-48vsx 1/1 Running 0 31m
PS C:\Users\danny> kubectl describe po --kubeconfig config
Name: wordpress-7b68f9d9bc-9nz27
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal TriggeredScaleUp 31m cluster-autoscaler pod triggered scale-up: [{aks-akscluster-26960436-vmss 1->2 (max: 3)}]
Warning FailedScheduling 30m (x4 over 33m) default-scheduler 0/1 nodes are available: 1 Insufficient cpu.
Warning FailedScheduling 29m default-scheduler 0/2 nodes are available: 1 Insufficient cpu, 1 node(s) had taint {node.kubernetes.io/network-unavailable: }, that the pod didn't tolerate.
Normal Scheduled 27m default-scheduler Successfully assigned default/wordpress-7b68f9d9bc-9nz27 to aks-akscluster-26960436-vmss000001
Normal SuccessfulAttachVolume 27m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d"
Warning FailedMount 16m kubelet Unable to attach or mount volumes: unmounted volumes=[wordpress-data], unattached volumes=[kube-api-access-b5scw wordpress-data custom-postinit]: timed out waiting for the condition
Warning FailedMount 3m39s (x10 over 25m) kubelet MountVolume.SetUp failed for volume "pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Warning FailedMount 52s (x11 over 25m) kubelet Unable to attach or mount volumes: unmounted volumes=[wordpress-data], unattached volumes=[wordpress-data custom-postinit kube-api-access-b5scw]: timed out waiting for the condition
The wordpress-memcached-774f78cc49-48vsx is bizarre that it's running as this is using the same storage class in the WordPress bitnami helmchart
PS C:\Users\danny> kubectl logs trident-csi-ddhgs -n netapp --kubeconfig config -c trident-main
time="2022-02-27T11:48:30Z" level=info msg="Running Trident storage orchestrator." binary=/trident_orchestrator build_time="Mon Jan 31 15:52:06 UTC 2022" version=22.01.0
time="2022-02-27T11:48:30Z" level=info msg="Initializing plain CSI helper frontend."
time="2022-02-27T11:48:30Z" level=info msg="Added frontend." name=plain_csi_helper
time="2022-02-27T11:48:30Z" level=info msg="Initializing CSI frontend." name=aks-akscluster-26960436-vmss000001 version=22.01.0
time="2022-02-27T11:48:30Z" level=info msg="Enabling node service capability." capability=STAGE_UNSTAGE_VOLUME
time="2022-02-27T11:48:30Z" level=info msg="Enabling node service capability." capability=EXPAND_VOLUME
time="2022-02-27T11:48:30Z" level=info msg="Enabling node service capability." capability=GET_VOLUME_STATS
time="2022-02-27T11:48:30Z" level=info msg="Enabling volume access mode." mode=SINGLE_NODE_WRITER
time="2022-02-27T11:48:30Z" level=info msg="Enabling volume access mode." mode=SINGLE_NODE_READER_ONLY
time="2022-02-27T11:48:30Z" level=info msg="Enabling volume access mode." mode=MULTI_NODE_READER_ONLY
time="2022-02-27T11:48:30Z" level=info msg="Enabling volume access mode." mode=MULTI_NODE_SINGLE_WRITER
time="2022-02-27T11:48:30Z" level=info msg="Enabling volume access mode." mode=MULTI_NODE_MULTI_WRITER
time="2022-02-27T11:48:30Z" level=info msg="Added frontend." name=csi
time="2022-02-27T11:48:30Z" level=info msg="Initializing HTTPS REST frontend." address=":17546"
time="2022-02-27T11:48:30Z" level=info msg="Added frontend." name="HTTPS REST"
time="2022-02-27T11:48:30Z" level=info msg="Added 0 existing volume(s)" requestID=370caff8-8090-400b-9617-c08b6605d607 requestSource=Internal
time="2022-02-27T11:48:30Z" level=info msg="Activating HTTPS REST frontend." address=":17546"
time="2022-02-27T11:48:30Z" level=info msg="Trident bootstrapped successfully."
time="2022-02-27T11:48:30Z" level=info msg="Activating plain CSI helper frontend."
time="2022-02-27T11:48:30Z" level=info msg="Activating CSI frontend." requestID=c83ef372-7c1b-4c79-acd5-84d41d21a7bd requestSource=Internal
time="2022-02-27T11:48:30Z" level=info msg="Starting periodic node access reconciliation service." requestID=d8bd8486-ec25-43ae-8ae6-cf39cd7178d5 requestSource=Periodic
time="2022-02-27T11:48:30Z" level=warning msg="Could not find iSCSI initiator name." requestID=c83ef372-7c1b-4c79-acd5-84d41d21a7bd requestSource=Internal
time="2022-02-27T11:48:30Z" level=info msg="Discovered IP addresses." IP Addresses="[10.240.0.5 10.244.1.1]" requestID=c83ef372-7c1b-4c79-acd5-84d41d21a7bd requestSource=Internal
time="2022-02-27T11:49:00Z" level=warning msg="Could not update Trident controller with node registration, will retry." error="could not log into the Trident CSI Controller: error communicating with Trident CSI Controller; Put \"https://10.0.86.71:34571/trident/v1/node/aks-akscluster-26960436-vmss000001\": context deadline exceeded (Client.Timeo
ut exceeded while awaiting headers)" increment=9.734991122s requestID=c83ef372-7c1b-4c79-acd5-84d41d21a7bd requestSource=Internal
time="2022-02-27T11:49:10Z" level=info msg="Updated Trident controller with node registration." node=aks-akscluster-26960436-vmss000001 requestID=c83ef372-7c1b-4c79-acd5-84d41d21a7bd requestSource=Internal
time="2022-02-27T11:49:10Z" level=info msg="Listening for GRPC connections." name=/plugin/csi.sock net=unix
time="2022-02-27T11:53:02Z" level=error msg="GRPC error: rpc error: code = Internal desc = error mounting NFS volume 10.0.0.68:/pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d on mountpoint /var/lib/kubelet/pods/439424b3-2be1-468c-8582-e323cb790500/volumes/kubernetes.io~csi/pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d/mount: exit status 32" requestID=92ec9
8f7-78db-4fae-9aaf-0089e7bc5247 requestSource=CSI
time="2022-02-27T11:56:02Z" level=error msg="GRPC error: rpc error: code = Internal desc = error mounting NFS volume 10.0.0.68:/pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d on mountpoint /var/lib/kubelet/pods/439424b3-2be1-468c-8582-e323cb790500/volumes/kubernetes.io~csi/pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d/mount: exit status 32" requestID=b0f9c
53a-b26e-43f9-9d84-84ad95618295 requestSource=CSI
time="2022-02-27T11:59:03Z" level=error msg="GRPC error: rpc error: code = Internal desc = error mounting NFS volume 10.0.0.68:/pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d on mountpoint /var/lib/kubelet/pods/439424b3-2be1-468c-8582-e323cb790500/volumes/kubernetes.io~csi/pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d/mount: exit status 32" requestID=07215
60b-5783-4709-a2ba-c3f13da96770 requestSource=CSI
PS C:\Users\danny> kubectl logs trident-csi-558f9448cd-hxmjt -n netapp -c trident-main --kubeconfig config
time="2022-02-27T11:32:12Z" level=info msg="Running Trident storage orchestrator." binary=/trident_orchestrator build_time="Mon Jan 31 15:52:06 UTC 2022" version=22.01.0
time="2022-02-27T11:32:12Z" level=warning msg="Could not get all server resources." error="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
time="2022-02-27T11:32:12Z" level=info msg="Created Kubernetes clients." namespace=netapp version=v1.23.3
time="2022-02-27T11:32:12Z" level=info msg="Initializing metrics frontend." address=":8001"
time="2022-02-27T11:32:12Z" level=info msg="Added frontend." name=metrics
time="2022-02-27T11:32:12Z" level=info msg="Initializing K8S helper frontend." requestID=f544c0ee-10e2-4c88-bbcf-f672ef60a22f requestSource=Internal
time="2022-02-27T11:32:12Z" level=info msg="K8S helper determined the container orchestrator version." gitVersion=v1.23.3 requestID=f544c0ee-10e2-4c88-bbcf-f672ef60a22f requestSource=Internal version=1.23
time="2022-02-27T11:32:12Z" level=info msg="Added frontend." name=k8s_csi_helper
time="2022-02-27T11:32:12Z" level=info msg="Initializing CSI frontend." name=aks-akscluster-26960436-vmss000000 version=22.01.0
time="2022-02-27T11:32:12Z" level=info msg="Enabling controller service capability." capability=CREATE_DELETE_VOLUME
time="2022-02-27T11:32:12Z" level=info msg="Enabling controller service capability." capability=PUBLISH_UNPUBLISH_VOLUME
time="2022-02-27T11:32:12Z" level=info msg="Enabling controller service capability." capability=LIST_VOLUMES
time="2022-02-27T11:32:12Z" level=info msg="Enabling controller service capability." capability=CREATE_DELETE_SNAPSHOT
time="2022-02-27T11:32:12Z" level=info msg="Enabling controller service capability." capability=LIST_SNAPSHOTS
time="2022-02-27T11:32:12Z" level=info msg="Enabling controller service capability." capability=EXPAND_VOLUME
time="2022-02-27T11:32:12Z" level=info msg="Enabling controller service capability." capability=CLONE_VOLUME
time="2022-02-27T11:32:12Z" level=info msg="Enabling controller service capability." capability=LIST_VOLUMES_PUBLISHED_NODES
time="2022-02-27T11:32:12Z" level=info msg="Enabling volume access mode." mode=SINGLE_NODE_WRITER
time="2022-02-27T11:32:12Z" level=info msg="Enabling volume access mode." mode=SINGLE_NODE_READER_ONLY
time="2022-02-27T11:32:12Z" level=info msg="Enabling volume access mode." mode=MULTI_NODE_READER_ONLY
time="2022-02-27T11:32:12Z" level=info msg="Enabling volume access mode." mode=MULTI_NODE_SINGLE_WRITER
time="2022-02-27T11:32:12Z" level=info msg="Enabling volume access mode." mode=MULTI_NODE_MULTI_WRITER
time="2022-02-27T11:32:12Z" level=info msg="Added frontend." name=csi
time="2022-02-27T11:32:12Z" level=info msg="Initializing Trident CRD controller frontend." namespace=netapp
time="2022-02-27T11:32:12Z" level=info msg="Creating event broadcaster."
time="2022-02-27T11:32:12Z" level=info msg="Setting up CRD controller event handlers."
time="2022-02-27T11:32:12Z" level=info msg="Added frontend." name=crd
time="2022-02-27T11:32:12Z" level=info msg="Initializing HTTP REST frontend." address="127.0.0.1:8000"
time="2022-02-27T11:32:12Z" level=info msg="Added frontend." name="HTTP REST"
time="2022-02-27T11:32:12Z" level=info msg="Initializing HTTPS REST frontend." address=":8443"
time="2022-02-27T11:32:12Z" level=info msg="Added frontend." name="HTTPS REST"
time="2022-02-27T11:32:12Z" level=info msg="Activating metrics frontend." address=":8001"
time="2022-02-27T11:32:12Z" level=info msg="Activating HTTP REST frontend." address="127.0.0.1:8000"
time="2022-02-27T11:32:12Z" level=info msg="Activating HTTPS REST frontend." address=":8443"
time="2022-02-27T11:32:12Z" level=warning msg="Persistent state version not found, creating." OrchestratorAPIVersion=1 PersistentStoreVersion=crdv1 requestID=16f61a84-d01a-43be-b777-96568c4fe9d9 requestSource=Internal
time="2022-02-27T11:32:12Z" level=info msg="Added 0 existing volume(s)" requestID=16f61a84-d01a-43be-b777-96568c4fe9d9 requestSource=Internal
time="2022-02-27T11:32:13Z" level=info msg="Trident bootstrapped successfully."
time="2022-02-27T11:32:13Z" level=info msg="Activating K8S helper frontend." requestID=e80f7bf7-e1c4-4a44-8026-9cc596e3a82e requestSource=Internal
time="2022-02-27T11:32:13Z" level=info msg="Activating CRD frontend."
time="2022-02-27T11:32:13Z" level=info msg="Activating CSI frontend." requestID=e513e51b-fd5a-4572-abc4-96245b0047d2 requestSource=Internal
time="2022-02-27T11:32:13Z" level=info msg="Listening for GRPC connections." name=/plugin/csi.sock net=unix
time="2022-02-27T11:32:13Z" level=info msg="Starting Trident CRD controller."
time="2022-02-27T11:32:13Z" level=info msg="Waiting for informer caches to sync."
time="2022-02-27T11:32:13Z" level=info msg="Starting periodic node access reconciliation service." requestID=5441ec07-dec6-440a-b764-4dceab0645ec requestSource=Periodic
I0227 11:32:14.237758 1 request.go:665] Waited for 1.159699954s due to client-side throttling, not priority and fairness, request: GET:https://10.0.0.1:443/apis/trident.netapp.io/v1/namespaces/netapp/tridentvolumepublications?limit=500&resourceVersion=0
time="2022-02-27T11:32:15Z" level=info msg="Starting workers."
time="2022-02-27T11:32:15Z" level=info msg="Started workers."
time="2022-02-27T11:32:44Z" level=info msg="Determined topology labels for node: map[topology.kubernetes.io/region:uksouth topology.kubernetes.io/zone:uksouth-1]" node=aks-akscluster-26960436-vmss000000 requestID=6df33250-146b-4d48-9951-72f9cf0e77e0 requestSource=REST
time="2022-02-27T11:32:45Z" level=info msg="Added a new node." handler=AddOrUpdateNode node=aks-akscluster-26960436-vmss000000 requestID=6df33250-146b-4d48-9951-72f9cf0e77e0 requestSource=REST
time="2022-02-27T11:35:07Z" level=info msg="Discovered Azure resources." capacityPools=1 crdControllerEvent=add requestID=728c892e-b200-40ed-8bec-b5fac0c5501c requestSource=CRD resourceGroups=1 subnets=1
time="2022-02-27T11:35:07Z" level=info msg="Storage driver initialized." crdControllerEvent=add driver=azure-netapp-files requestID=728c892e-b200-40ed-8bec-b5fac0c5501c requestSource=CRD
time="2022-02-27T11:35:07Z" level=info msg="Created new storage backend." backend="&{0xc00094e3c0 netapp-netapppool true online map[netapp-netapppool_pool:0xc0003b4b40] map[] false}" crdControllerEvent=add requestID=728c892e-b200-40ed-8bec-b5fac0c5501c requestSource=CRD
time="2022-02-27T11:35:07Z" level=info msg="Newly added backend satisfies no storage classes." backend=netapp-netapppool crdControllerEvent=add requestID=728c892e-b200-40ed-8bec-b5fac0c5501c requestSource=CRD
time="2022-02-27T11:35:07Z" level=info msg="Synced 'netapp/nett-app-backend'" crdControllerEvent=add logSource=trident-crd-controller requestID=728c892e-b200-40ed-8bec-b5fac0c5501c requestSource=CRD
time="2022-02-27T11:35:07Z" level=info msg=-------------------------------------------------
time="2022-02-27T11:35:07Z" level=info msg=-------------------------------------------------
time="2022-02-27T11:42:00Z" level=info msg="Storage class satisfied by 1 storage pools." requestID=39e4a9b4-9b8e-49c8-878f-d4fcd8398028 requestSource=Kubernetes storageClass=wordpress-storage-class-netapp
time="2022-02-27T11:42:00Z" level=info msg="K8S helper added a storage class." name=wordpress-storage-class-netapp parameters="map[backendType:azure-netapp-files fsType:nfs]" provisioner=csi.trident.netapp.io requestID=39e4a9b4-9b8e-49c8-878f-d4fcd8398028 requestSource=Kubernetes
time="2022-02-27T11:42:32Z" level=info msg="Found PVC for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." UID=ce36221a-eb54-465a-b271-2a7df09067b5 name=prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=58e5f2a1-6732-4322-acac-92c2c1907732 requestSourc
e=CSI size="{{100 9} {<nil>} 100G DecimalSI}" storageClass=wordpress-storage-class-netapp
time="2022-02-27T11:42:32Z" level=info msg="Found storage class for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." name=wordpress-storage-class-netapp requestID=58e5f2a1-6732-4322-acac-92c2c1907732 requestSource=CSI
time="2022-02-27T11:42:33Z" level=warning msg="Requested size is too small. Setting volume size to the minimum allowable (100 GB)." name=pvc-ce36221a-eb54-465a-b271-2a7df09067b5 requestID=58e5f2a1-6732-4322-acac-92c2c1907732 requestSource=CSI size=100000000000
time="2022-02-27T11:42:34Z" level=info msg="Volume created." API=VolumesClient.CreateOrUpdate correlationID=b95140a3-5622-4e98-b959-7355d5c91a04 creationToken=pvc-ce36221a-eb54-465a-b271-2a7df09067b5 requestID=58e5f2a1-6732-4322-acac-92c2c1907732 requestSource=CSI volume=host_rg/netapp-netappaccount/netapp-netapppool/pvc-ce36221a-eb54-465a-b271-
2a7df09067b5
time="2022-02-27T11:42:34Z" level=info msg="Waiting for volume state." desiredState=Succeeded requestID=58e5f2a1-6732-4322-acac-92c2c1907732 requestSource=CSI
time="2022-02-27T11:42:41Z" level=error msg="Volume state was not Succeeded after 10.00 seconds." requestID=58e5f2a1-6732-4322-acac-92c2c1907732 requestSource=CSI
time="2022-02-27T11:42:41Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=07357a57-d3f9-45ac-a827-c03baf907fd0 error="volume state is Creating, not Succeeded" pool=netapp-netapppool_pool requestID=58e5f2a1-6732-4322-acac-92c2c1907732 requestSource=CSI volume=pvc-ce36221a-eb54-465a-b271-2a7df0906
7b5
time="2022-02-27T11:42:41Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is Creating, not Succeeded" requestID=58e5f2a1-6732-4322-acac-92c2c1907732 requestSource=CSI
time="2022-02-27T11:42:49Z" level=info msg="Found PVC for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." UID=ce36221a-eb54-465a-b271-2a7df09067b5 name=prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=740fbbbf-fee8-47fa-ac6a-df708bcfea96 requestSourc
e=CSI size="{{100 9} {<nil>} 100G DecimalSI}" storageClass=wordpress-storage-class-netapp
time="2022-02-27T11:42:49Z" level=info msg="Found storage class for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." name=wordpress-storage-class-netapp requestID=740fbbbf-fee8-47fa-ac6a-df708bcfea96 requestSource=CSI
time="2022-02-27T11:42:50Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=07357a57-d3f9-45ac-a827-c03baf907fd0 error="volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=740fbbbf-fee8-47fa-ac6a-df708bcfea96 requestSource=CSI volume=pvc-ce36221a-eb54-465a-b271-2a7
df09067b5
time="2022-02-27T11:42:50Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=740fbbbf-fee8-47fa-ac6a-df708bcfea96 requestSource=CSI
time="2022-02-27T11:43:06Z" level=info msg="Found PVC for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." UID=ce36221a-eb54-465a-b271-2a7df09067b5 name=prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=59b7a01f-c8ba-48be-84ea-18ad481646b2 requestSourc
e=CSI size="{{100 9} {<nil>} 100G DecimalSI}" storageClass=wordpress-storage-class-netapp
time="2022-02-27T11:43:06Z" level=info msg="Found storage class for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." name=wordpress-storage-class-netapp requestID=59b7a01f-c8ba-48be-84ea-18ad481646b2 requestSource=CSI
time="2022-02-27T11:43:06Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=07357a57-d3f9-45ac-a827-c03baf907fd0 error="volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=59b7a01f-c8ba-48be-84ea-18ad481646b2 requestSource=CSI volume=pvc-ce36221a-eb54-465a-b271-2a7
df09067b5
time="2022-02-27T11:43:06Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=59b7a01f-c8ba-48be-84ea-18ad481646b2 requestSource=CSI
time="2022-02-27T11:43:36Z" level=info msg="Found PVC for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." UID=ce36221a-eb54-465a-b271-2a7df09067b5 name=prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=7e2033f8-c4e3-4e9f-a13c-43618da910c7 requestSourc
e=CSI size="{{100 9} {<nil>} 100G DecimalSI}" storageClass=wordpress-storage-class-netapp
time="2022-02-27T11:43:36Z" level=info msg="Found storage class for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." name=wordpress-storage-class-netapp requestID=7e2033f8-c4e3-4e9f-a13c-43618da910c7 requestSource=CSI
time="2022-02-27T11:43:36Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=07357a57-d3f9-45ac-a827-c03baf907fd0 error="volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=7e2033f8-c4e3-4e9f-a13c-43618da910c7 requestSource=CSI volume=pvc-ce36221a-eb54-465a-b271-2a7
df09067b5
time="2022-02-27T11:43:36Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=7e2033f8-c4e3-4e9f-a13c-43618da910c7 requestSource=CSI
time="2022-02-27T11:44:06Z" level=info msg="Found PVC for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." UID=ce36221a-eb54-465a-b271-2a7df09067b5 name=prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=3b4febf2-5bcc-4dbb-9e77-80828d0b5e5d requestSourc
e=CSI size="{{100 9} {<nil>} 100G DecimalSI}" storageClass=wordpress-storage-class-netapp
time="2022-02-27T11:44:06Z" level=info msg="Found storage class for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." name=wordpress-storage-class-netapp requestID=3b4febf2-5bcc-4dbb-9e77-80828d0b5e5d requestSource=CSI
time="2022-02-27T11:44:06Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=07357a57-d3f9-45ac-a827-c03baf907fd0 error="volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=3b4febf2-5bcc-4dbb-9e77-80828d0b5e5d requestSource=CSI volume=pvc-ce36221a-eb54-465a-b271-2a7
df09067b5
time="2022-02-27T11:44:06Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=3b4febf2-5bcc-4dbb-9e77-80828d0b5e5d requestSource=CSI
time="2022-02-27T11:44:36Z" level=info msg="Found PVC for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." UID=ce36221a-eb54-465a-b271-2a7df09067b5 name=prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=aed9619f-6b33-4274-bfb7-112b1b10530a requestSourc
e=CSI size="{{100 9} {<nil>} 100G DecimalSI}" storageClass=wordpress-storage-class-netapp
time="2022-02-27T11:44:36Z" level=info msg="Found storage class for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." name=wordpress-storage-class-netapp requestID=aed9619f-6b33-4274-bfb7-112b1b10530a requestSource=CSI
time="2022-02-27T11:44:36Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=07357a57-d3f9-45ac-a827-c03baf907fd0 error="volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=aed9619f-6b33-4274-bfb7-112b1b10530a requestSource=CSI volume=pvc-ce36221a-eb54-465a-b271-2a7
df09067b5
time="2022-02-27T11:44:36Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=aed9619f-6b33-4274-bfb7-112b1b10530a requestSource=CSI
time="2022-02-27T11:45:06Z" level=info msg="Found PVC for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." UID=ce36221a-eb54-465a-b271-2a7df09067b5 name=prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=8433bfd9-c4aa-4c67-98e1-936cd3804d56 requestSourc
e=CSI size="{{100 9} {<nil>} 100G DecimalSI}" storageClass=wordpress-storage-class-netapp
time="2022-02-27T11:45:06Z" level=info msg="Found storage class for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." name=wordpress-storage-class-netapp requestID=8433bfd9-c4aa-4c67-98e1-936cd3804d56 requestSource=CSI
time="2022-02-27T11:45:06Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=07357a57-d3f9-45ac-a827-c03baf907fd0 error="volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=8433bfd9-c4aa-4c67-98e1-936cd3804d56 requestSource=CSI volume=pvc-ce36221a-eb54-465a-b271-2a7
df09067b5
time="2022-02-27T11:45:06Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=8433bfd9-c4aa-4c67-98e1-936cd3804d56 requestSource=CSI
time="2022-02-27T11:45:36Z" level=info msg="Found PVC for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." UID=ce36221a-eb54-465a-b271-2a7df09067b5 name=prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=e1b10a31-3f15-47c9-94fc-ed3519abbd4b requestSourc
e=CSI size="{{100 9} {<nil>} 100G DecimalSI}" storageClass=wordpress-storage-class-netapp
time="2022-02-27T11:45:36Z" level=info msg="Found storage class for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." name=wordpress-storage-class-netapp requestID=e1b10a31-3f15-47c9-94fc-ed3519abbd4b requestSource=CSI
time="2022-02-27T11:45:36Z" level=info msg="Discovered Azure resources." capacityPools=1 requestID=e1b10a31-3f15-47c9-94fc-ed3519abbd4b requestSource=CSI resourceGroups=1 subnets=1
time="2022-02-27T11:45:37Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=07357a57-d3f9-45ac-a827-c03baf907fd0 error="volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=e1b10a31-3f15-47c9-94fc-ed3519abbd4b requestSource=CSI volume=pvc-ce36221a-eb54-465a-b271-2a7
df09067b5
time="2022-02-27T11:45:37Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=e1b10a31-3f15-47c9-94fc-ed3519abbd4b requestSource=CSI
time="2022-02-27T11:46:07Z" level=info msg="Found PVC for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." UID=ce36221a-eb54-465a-b271-2a7df09067b5 name=prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=b24b6a4c-6997-40ae-a679-b825e9ea3490 requestSourc
e=CSI size="{{100 9} {<nil>} 100G DecimalSI}" storageClass=wordpress-storage-class-netapp
time="2022-02-27T11:46:07Z" level=info msg="Found storage class for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." name=wordpress-storage-class-netapp requestID=b24b6a4c-6997-40ae-a679-b825e9ea3490 requestSource=CSI
time="2022-02-27T11:46:07Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=07357a57-d3f9-45ac-a827-c03baf907fd0 error="volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=b24b6a4c-6997-40ae-a679-b825e9ea3490 requestSource=CSI volume=pvc-ce36221a-eb54-465a-b271-2a7
df09067b5
time="2022-02-27T11:46:07Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=b24b6a4c-6997-40ae-a679-b825e9ea3490 requestSource=CSI
time="2022-02-27T11:46:37Z" level=info msg="Found PVC for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." UID=ce36221a-eb54-465a-b271-2a7df09067b5 name=prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=48c52c43-6815-4cd0-a9e4-28bc95e96274 requestSourc
e=CSI size="{{100 9} {<nil>} 100G DecimalSI}" storageClass=wordpress-storage-class-netapp
time="2022-02-27T11:46:37Z" level=info msg="Found storage class for requested volume pvc-ce36221a-eb54-465a-b271-2a7df09067b5." name=wordpress-storage-class-netapp requestID=48c52c43-6815-4cd0-a9e4-28bc95e96274 requestSource=CSI
time="2022-02-27T11:46:37Z" level=warning msg="Volume already exists." name=pvc-ce36221a-eb54-465a-b271-2a7df09067b5 requestID=48c52c43-6815-4cd0-a9e4-28bc95e96274 requestSource=CSI state=Succeeded
time="2022-02-27T11:46:37Z" level=warning msg="Volume already exists." backend=netapp-netapppool requestID=48c52c43-6815-4cd0-a9e4-28bc95e96274 requestSource=CSI volume=pvc-ce36221a-eb54-465a-b271-2a7df09067b5
time="2022-02-27T11:49:10Z" level=info msg="Determined topology labels for node: map[topology.kubernetes.io/region:uksouth topology.kubernetes.io/zone:uksouth-1]" node=aks-akscluster-26960436-vmss000001 requestID=c83ef372-7c1b-4c79-acd5-84d41d21a7bd requestSource=REST
time="2022-02-27T11:49:10Z" level=info msg="Added a new node." handler=AddOrUpdateNode node=aks-akscluster-26960436-vmss000001 requestID=c83ef372-7c1b-4c79-acd5-84d41d21a7bd requestSource=REST
time="2022-02-27T11:49:22Z" level=info msg="Found PVC for requested volume pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d." UID=1d9eeaac-e0fe-498c-8376-9ee39c30fb2d name=wordpress namespace=default requestID=28799fe2-2796-467e-b1a0-548f0d353013 requestSource=CSI size="{{100 9} {<nil>} 100G DecimalSI}" storageClass=wordpress-storage-class-netapp
time="2022-02-27T11:49:22Z" level=info msg="Found storage class for requested volume pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d." name=wordpress-storage-class-netapp requestID=28799fe2-2796-467e-b1a0-548f0d353013 requestSource=CSI
time="2022-02-27T11:49:22Z" level=warning msg="Requested size is too small. Setting volume size to the minimum allowable (100 GB)." name=pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d requestID=28799fe2-2796-467e-b1a0-548f0d353013 requestSource=CSI size=100000000000
time="2022-02-27T11:49:23Z" level=info msg="Volume created." API=VolumesClient.CreateOrUpdate correlationID=1a8c7729-4d30-4219-a475-7d210d7ab4ae creationToken=pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d requestID=28799fe2-2796-467e-b1a0-548f0d353013 requestSource=CSI volume=host_rg/netapp-netappaccount/netapp-netapppool/pvc-1d9eeaac-e0fe-498c-8376-
9ee39c30fb2d
time="2022-02-27T11:49:23Z" level=info msg="Waiting for volume state." desiredState=Succeeded requestID=28799fe2-2796-467e-b1a0-548f0d353013 requestSource=CSI
time="2022-02-27T11:49:30Z" level=error msg="Volume state was not Succeeded after 10.00 seconds." requestID=28799fe2-2796-467e-b1a0-548f0d353013 requestSource=CSI
time="2022-02-27T11:49:30Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=07357a57-d3f9-45ac-a827-c03baf907fd0 error="volume state is Creating, not Succeeded" pool=netapp-netapppool_pool requestID=28799fe2-2796-467e-b1a0-548f0d353013 requestSource=CSI volume=pvc-1d9eeaac-e0fe-498c-8376-9ee39c30f
b2d
time="2022-02-27T11:49:30Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is Creating, not Succeeded" requestID=28799fe2-2796-467e-b1a0-548f0d353013 requestSource=CSI
time="2022-02-27T11:49:38Z" level=info msg="Found PVC for requested volume pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d." UID=1d9eeaac-e0fe-498c-8376-9ee39c30fb2d name=wordpress namespace=default requestID=c28ef4ae-3b46-4b12-9f43-c1fed3eb4748 requestSource=CSI size="{{100 9} {<nil>} 100G DecimalSI}" storageClass=wordpress-storage-class-netapp
time="2022-02-27T11:49:38Z" level=info msg="Found storage class for requested volume pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d." name=wordpress-storage-class-netapp requestID=c28ef4ae-3b46-4b12-9f43-c1fed3eb4748 requestSource=CSI
time="2022-02-27T11:49:38Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=07357a57-d3f9-45ac-a827-c03baf907fd0 error="volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=c28ef4ae-3b46-4b12-9f43-c1fed3eb4748 requestSource=CSI volume=pvc-1d9eeaac-e0fe-498c-8376-9ee
39c30fb2d
time="2022-02-27T11:49:38Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=c28ef4ae-3b46-4b12-9f43-c1fed3eb4748 requestSource=CSI
time="2022-02-27T11:49:54Z" level=info msg="Found PVC for requested volume pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d." UID=1d9eeaac-e0fe-498c-8376-9ee39c30fb2d name=wordpress namespace=default requestID=dcdc9522-09d4-4f3a-9778-fd84b5009db9 requestSource=CSI size="{{100 9} {<nil>} 100G DecimalSI}" storageClass=wordpress-storage-class-netapp
time="2022-02-27T11:49:54Z" level=info msg="Found storage class for requested volume pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d." name=wordpress-storage-class-netapp requestID=dcdc9522-09d4-4f3a-9778-fd84b5009db9 requestSource=CSI
time="2022-02-27T11:49:54Z" level=warning msg="Volume already exists." name=pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d requestID=dcdc9522-09d4-4f3a-9778-fd84b5009db9 requestSource=CSI state=Succeeded
time="2022-02-27T11:49:54Z" level=warning msg="Volume already exists." backend=netapp-netapppool requestID=dcdc9522-09d4-4f3a-9778-fd84b5009db9 requestSource=CSI volume=pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d
PS C:\Users\danny> kubectl get pv,pvc --kubeconfig config
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d 100Gi RWX Delete Bound default/wordpress wordpress-storage-class-netapp 14m
persistentvolume/pvc-7d7688c3-abc4-4628-abc7-1ae206b0b23a 10Gi RWO Delete Bound monitoring/grafana-pvc default 19m
persistentvolume/pvc-ce36221a-eb54-465a-b271-2a7df09067b5 100Gi RWO Delete Bound monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 wordpress-storage-class-netapp 17m
kubectl describe TridentBackendConfig -n netapp --kubeconfig config
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Success 47m trident-crd-controller Backend 'netapp-netapppool' created
S C:\Users\danny> kubectl get all -n netapp --kubeconfig config
NAME READY STATUS RESTARTS AGE
pod/trident-csi-558f9448cd-hxmjt 6/6 Running 0 51m
pod/trident-csi-ddhgs 2/2 Running 0 35m
pod/trident-csi-hlzs8 2/2 Running 0 51m
pod/trident-operator-5b796496d5-qnpps 1/1 Running 0 51m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/trident-csi ClusterIP 10.0.86.71 <none> 34571/TCP,9220/TCP 51m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/trident-csi 2 2 2 2 2 kubernetes.io/arch=amd64,kubernetes.io/os=linux 51m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/trident-csi 1/1 1 1 51m
deployment.apps/trident-operator 1/1 1 1 51m
NAME DESIRED CURRENT READY AGE
replicaset.apps/trident-csi-558f9448cd 1 1 1 51m
replicaset.apps/trident-operator-5b796496d5 1 1 1 51m
additional terraform provisioning
data "template_file" "netapp_backend_config"{
depends_on = [kubernetes_secret_v1.netapp]
template = file("${path.module}/BackendCRD/backend-anf.yaml")
vars = {
# "clientID" = data.azurerm_client_config.current.client_id
# "clientSecret" = var.netappclientsecret
"metadataname" = "nett-app-backend"
"namespace" = kubernetes_namespace.netapp.metadata[0].name
"storagedrivername" = "azure-netapp-files"
"subscriptionid" = data.azurerm_client_config.current.subscription_id
"tenantid" = data.azurerm_client_config.current.tenant_id
"location" = var.location
"servicelevel" = var.servicelevel
"secretcredentails" = kubernetes_secret_v1.netapp.metadata[0].name
"subnet" = var.subnet
"virtualNetwork" = var.virtualNetwork
"nfsMountOptions" = "nfsvers=4"
"backendName" = azurerm_netapp_pool.netapp.name
}
}
resource "kubernetes_storage_class_v1" "example" {
depends_on = [
helm_release.netapphelm,
kubernetes_secret_v1.netapp,
time_sleep.wait_600_seconds
]
metadata {
name = var.storageclass
}
storage_provisioner = "csi.trident.netapp.io"
reclaim_policy = "Delete"
volume_binding_mode = "WaitForFirstConsumer"
allow_volume_expansion = var.allow_volume_exspansion
parameters = {
backendType = "azure-netapp-files"
fsType: "nfs"
}
# mount_options = ["file_mode=0644", "dir_mode=0755", "uid=1001", "gid=1001"]
}
PS C:\Users\danny> kubectl get events --kubeconfig config
LAST SEEN TYPE REASON OBJECT MESSAGE
2m7s Warning KubeletRPCDeadlineExceeded node/aks-akscluster-26960436-vmss000000 (combined from similar events): E0227 13:23:36.051152 3267 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/csi.trident.netapp.io^pvc-ce36221a-eb54-465a-b271-2a7df09067b5 podName: nodeName:}" failed. No retries permitted un
til 2022-02-27 13:25:38.051125775 +0000 UTC m=+6922.364702786 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "pvc-ce36221a-eb54-465a-b271-2a7df09067b5" (UniqueName: "kubernetes.io/csi/csi.trident.netapp.io^pvc-ce36221a-eb54-465a-b271-2a7df09067b5") pod "prometheus-prometheus-kube-prometheus-prometheus-0" (UID: "f670f4db-f
13b-4e8a-8156-65e15f78b6e6") : rpc error: code = DeadlineExceeded desc = context deadline exceeded
2m58s Warning KubeletRPCDeadlineExceeded node/aks-akscluster-26960436-vmss000001 (combined from similar events): E0227 13:22:45.564570 3163 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/csi.trident.netapp.io^pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d podName: nodeName:}" failed. No retries permitted un
til 2022-02-27 13:24:47.564534125 +0000 UTC m=+5801.165314210 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d" (UniqueName: "kubernetes.io/csi/csi.trident.netapp.io^pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d") pod "wordpress-7b68f9d9bc-9nz27" (UID: "439424b3-2be1-468c-8582-e323cb7905
00") : rpc error: code = DeadlineExceeded desc = context deadline exceeded
4m2s Normal Scheduled pod/wordpress-7b68f9d9bc-28lvx Successfully assigned default/wordpress-7b68f9d9bc-28lvx to aks-akscluster-26960436-vmss000001
119s Warning FailedMount pod/wordpress-7b68f9d9bc-28lvx Unable to attach or mount volumes: unmounted volumes=[wordpress-data], unattached volumes=[custom-postinit kube-api-access-5g8pq wordpress-data]: timed out waiting for the condition
12m Warning FailedMount pod/wordpress-7b68f9d9bc-9nz27 Unable to attach or mount volumes: unmounted volumes=[wordpress-data], unattached volumes=[wordpress-data custom-postinit kube-api-access-b5scw]: timed out waiting for the condition
43m Warning FailedMount pod/wordpress-7b68f9d9bc-9nz27 MountVolume.SetUp failed for volume "pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
23m Warning FailedMount pod/wordpress-7b68f9d9bc-9nz27 Unable to attach or mount volumes: unmounted volumes=[wordpress-data], unattached volumes=[kube-api-access-b5scw wordpress-data custom-postinit]: timed out waiting for the condition
3m7s Warning FailedMount pod/wordpress-7b68f9d9bc-9nz27 Unable to attach or mount volumes: unmounted volumes=[wordpress-data], unattached volumes=[custom-postinit kube-api-access-b5scw wordpress-data]: timed out waiting for the condition
4m2s Normal SuccessfulCreate replicaset/wordpress-7b68f9d9bc Created pod: wordpress-7b68f9d9bc-28lvx
55s Warning FailedGetResourceMetric horizontalpodautoscaler/wordpress failed to get memory utilization: unable to get metrics for resource memory: no metrics returned from resource metrics API
PS C:\Users\danny> kubectl describe torc trident --kubeconfig config
Name: trident
Namespace:
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: netapp-trident-operator
meta.helm.sh/release-namespace: netapp
API Version: trident.netapp.io/v1
Kind: TridentOrchestrator
Metadata:
Creation Timestamp: 2022-02-27T11:31:53Z
Generation: 1
Managed Fields:
API Version: trident.netapp.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:meta.helm.sh/release-name:
f:meta.helm.sh/release-namespace:
f:labels:
.:
f:app.kubernetes.io/managed-by:
f:spec:
.:
f:IPv6:
f:autosupportImage:
f:autosupportProxy:
f:debug:
f:enableNodePrep:
f:httpRequestTimeout:
f:k8sTimeout:
f:kubeletDir:
f:logFormat:
f:namespace:
f:probePort:
f:silenceAutosupport:
f:tridentImage:
Manager: terraform-provider-helm_v2.4.1_x5.exe
Operation: Update
Time: 2022-02-27T11:31:53Z
API Version: trident.netapp.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:currentInstallationParams:
.:
f:IPv6:
f:autosupportHostname:
f:autosupportImage:
f:autosupportProxy:
f:autosupportSerialNumber:
f:debug:
f:enableNodePrep:
f:httpRequestTimeout:
f:imagePullSecrets:
f:imageRegistry:
f:k8sTimeout:
f:kubeletDir:
f:logFormat:
f:probePort:
f:silenceAutosupport:
f:tridentImage:
f:message:
f:namespace:
f:status:
f:version:
Manager: trident-operator
Operation: Update
Subresource: status
Time: 2022-02-27T11:31:57Z
Resource Version: 1797
UID: eb6afa5b-c928-4c28-bd45-a73dd814bc7f
Spec:
IPv6: false
Autosupport Image: netapp/trident-autosupport:22.01
Autosupport Proxy: <nil>
Debug: false
Enable Node Prep: false
Http Request Timeout: 90s
k8sTimeout: 0
Kubelet Dir: <nil>
Log Format: text
Namespace: netapp
Probe Port: 17546
Silence Autosupport: false
Trident Image: netapp/trident:22.01.0
Status:
Current Installation Params:
IPv6: false
Autosupport Hostname:
Autosupport Image: netapp/trident-autosupport:22.01
Autosupport Proxy:
Autosupport Serial Number:
Debug: false
Enable Node Prep: false
Http Request Timeout: 90s
Image Pull Secrets:
Image Registry:
k8sTimeout: 30
Kubelet Dir: /var/lib/kubelet
Log Format: text
Probe Port: 17546
Silence Autosupport: false
Trident Image: netapp/trident:22.01.0
Message: Trident installed
Namespace: netapp
Status: Installed
Version: v22.01.0
Events: <none>
PS C:\Users\danny> kubectl logs trident-csi-75s28 -n netapp -c driver-registrar --kubeconfig config
I0227 18:28:10.991464 7550 main.go:166] Version: v2.4.0
I0227 18:28:10.991495 7550 main.go:167] Running node-driver-registrar in mode=registration
I0227 18:28:10.991837 7550 main.go:191] Attempting to open a gRPC connection with: "/plugin/csi.sock"
W0227 18:28:20.992127 7550 connection.go:173] Still connecting to unix:///plugin/csi.sock
W0227 18:28:30.992186 7550 connection.go:173] Still connecting to unix:///plugin/csi.sock
I0227 18:28:38.703115 7550 main.go:198] Calling CSI driver to discover driver name
I0227 18:28:38.704532 7550 main.go:208] CSI driver name: "csi.trident.netapp.io"
I0227 18:28:38.704560 7550 node_register.go:53] Starting Registration Server at: /registration/csi.trident.netapp.io-reg.sock
I0227 18:28:38.704764 7550 node_register.go:62] Registration Server started at: /registration/csi.trident.netapp.io-reg.sock
I0227 18:28:38.704924 7550 node_register.go:92] Skipping HTTP server because endpoint is set to: ""
I0227 18:28:38.723563 7550 main.go:102] Received GetInfo call: &InfoRequest{}
I0227 18:28:38.723833 7550 main.go:109] "Kubelet registration probe created" path="/var/lib/kubelet/plugins/csi.trident.netapp.io/registration"
I0227 18:28:38.772815 7550 main.go:120] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}
PS C:\Users\danny> kubectl describe sc wordpress-storage-class-netapp -A --kubeconfig config
Name: wordpress-storage-class-netapp
IsDefaultClass: No
Annotations: <none>
Provisioner: csi.trident.netapp.io
Parameters: backendType=azure-netapp-files,fsType=nfs
AllowVolumeExpansion: True
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: WaitForFirstConsumer
Events: <none>
kubectl get events --kubeconfig config
36m Normal Installing tridentorchestrator/trident Installing Trident
36m Normal Installed tridentorchestrator/trident Trident installed
21m Warning FailedScheduling pod/wordpress-686b57c696-qzxb6 0/1 nodes are available: 1 Insufficient cpu.
22m Normal TriggeredScaleUp pod/wordpress-686b57c696-qzxb6 pod triggered scale-up: [{aks-akscluster-25291065-vmss 1->2 (max: 3)}]
20m Warning FailedScheduling pod/wordpress-686b57c696-qzxb6 0/2 nodes are available: 1 Insufficient cpu, 1 node(s) had taint {node.kubernetes.io/network-unavailable: }, that the pod didn't tolerate.
19m Normal Scheduled pod/wordpress-686b57c696-qzxb6 Successfully assigned default/wordpress-686b57c696-qzxb6 to aks-akscluster-25291065-vmss000001
19m Normal SuccessfulAttachVolume pod/wordpress-686b57c696-qzxb6 AttachVolume.Attach succeeded for volume "pvc-e7e87fa5-5396-43ed-b53c-e67dfd2a9f57"
3m44s Warning FailedMount pod/wordpress-686b57c696-qzxb6 Unable to attach or mount volumes: unmounted volumes=[wordpress-data], unattached volumes=[wordpress-data custom-postinit kube-api-access-wgh54]: timed out waiting for the condition
2m9s Warning FailedMount pod/wordpress-686b57c696-qzxb6 MountVolume.SetUp failed for volume "pvc-e7e87fa5-5396-43ed-b53c-e67dfd2a9f57" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
89s Warning FailedMount pod/wordpress-686b57c696-qzxb6 Unable to attach or mount volumes: unmounted volumes=[wordpress-data], unattached volumes=[custom-postinit kube-api-access-wgh54 wordpress-data]: timed out waiting for the condition
19m Normal WaitForPodScheduled persistentvolumeclaim/wordpress waiting for pod wordpress-686b57c696-qzxb6 to be scheduled
4m12s Warning FailedGetResourceMetric horizontalpodautoscaler/wordpress failed to get memory utilization: unable to get metrics for resource memory: no metrics returned from resource metrics API
22m Warning FailedGetResourceMetric horizontalpodautoscaler/wordpress failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/wordpress invalid metrics (2 invalid out of 2), first error is: failed to get memory utilization: unable to get metrics for resource memory: no metrics returned from resource metrics API
19m Normal Provisioning persistentvolumeclaim/wordpress External provisioner is provisioning volume for claim "default/wordpress"
19m Normal ExternalProvisioning persistentvolumeclaim/wordpress waiting for a volume to be created, either by external provisioner "csi.trident.netapp.io" or manually created by system administrator
19m Normal ProvisioningFailed persistentvolumeclaim/wordpress volume state is Creating, not Succeeded
19m Warning ProvisioningFailed persistentvolumeclaim/wordpress failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is Creating, not Succeeded
19m Normal ProvisioningSuccess persistentvolumeclaim/wordpress provisioned a volume
19m Normal ProvisioningSucceeded persistentvolumeclaim/wordpress Successfully provisioned volume pvc-e7e87fa5-5396-43ed-b53c-e67dfd2a9f57
25m Warning listen tcp4 :31579: bind: address already in use node/aks-akscluster-25291065-vmss000000 can't open port "nodePort for ingress/nginx-ingress-controller:http" (:31579/tcp4), skipping it
25m Warning listen tcp4 :32165: bind: address already in use node/aks-akscluster-25291065-vmss000000 can't open port "nodePort for ingress/nginx-ingress-controller:https" (:32165/tcp4), skipping it
14m Warning KubeletRPCDeadlineExceeded node/aks-akscluster-25291065-vmss000000 E0227 18:44:34.536956 3224 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/csi.trident.netapp.io^pvc-176a73f0-2eab-4c39-9950-1b0258bcd174 podName: nodeName:}" failed. No retrie
s permitted until 2022-02-27 18:44:35.036929055 +0000 UTC m=+1113.070873727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-176a73f0-2eab-4c39-9950-1b0258bcd174" (UniqueName: "kubernetes.io/csi/csi.trident.netapp.io^pvc-176a73f0-2eab-4c39-9950-1b0258bcd174") pod "prometheus-prometheus-kube-prometheus-prometheus-0" (U
ID: "e9f3b6c1-aa79-4263-982f-977eab160de4") : rpc error: code = DeadlineExceeded desc = context deadline exceeded
14m Warning KubeletRPCDeadlineExceeded node/aks-akscluster-25291065-vmss000000 E0227 18:46:35.058544 3224 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/csi.trident.netapp.io^pvc-176a73f0-2eab-4c39-9950-1b0258bcd174 podName: nodeName:}" failed. No retrie
s permitted until 2022-02-27 18:46:36.058516754 +0000 UTC m=+1234.092461426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "pvc-176a73f0-2eab-4c39-9950-1b0258bcd174" (UniqueName: "kubernetes.io/csi/csi.trident.netapp.io^pvc-176a73f0-2eab-4c39-9950-1b0258bcd174") pod "prometheus-prometheus-kube-prometheus-prometheus-0" (UID:
"e9f3b6c1-aa79-4263-982f-977eab160de4") : rpc error: code = DeadlineExceeded desc = context deadline exceeded
14m Warning KubeletRPCDeadlineExceeded node/aks-akscluster-25291065-vmss000000 E0227 18:48:36.088129 3224 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/csi.trident.netapp.io^pvc-176a73f0-2eab-4c39-9950-1b0258bcd174 podName: nodeName:}" failed. No retrie
s permitted until 2022-02-27 18:48:38.088070727 +0000 UTC m=+1356.122015499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "pvc-176a73f0-2eab-4c39-9950-1b0258bcd174" (UniqueName: "kubernetes.io/csi/csi.trident.netapp.io^pvc-176a73f0-2eab-4c39-9950-1b0258bcd174") pod "prometheus-prometheus-kube-prometheus-prometheus-0" (UID:
"e9f3b6c1-aa79-4263-982f-977eab160de4") : rpc error: code = DeadlineExceeded desc = context deadline exceeded
14m Warning KubeletRPCDeadlineExceeded node/aks-akscluster-25291065-vmss000000 E0227 18:50:38.121172 3224 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/csi.trident.netapp.io^pvc-176a73f0-2eab-4c39-9950-1b0258bcd174 podName: nodeName:}" failed. No retrie
s permitted until 2022-02-27 18:50:42.121143914 +0000 UTC m=+1480.155088586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "pvc-176a73f0-2eab-4c39-9950-1b0258bcd174" (UniqueName: "kubernetes.io/csi/csi.trident.netapp.io^pvc-176a73f0-2eab-4c39-9950-1b0258bcd174") pod "prometheus-prometheus-kube-prometheus-prometheus-0" (UID:
"e9f3b6c1-aa79-4263-982f-977eab160de4") : rpc error: code = DeadlineExceeded desc = context deadline exceeded
13m Normal NoPreemptScheduled node/aks-akscluster-25291065-vmss000000 Node condition PreemptScheduled is now: Unknown, reason: NoPreemptScheduled
13m Normal NoPreemptScheduled node/aks-akscluster-25291065-vmss000000 Node condition PreemptScheduled is now: False, reason: NoPreemptScheduled
12m Warning KubeletRPCDeadlineExceeded node/aks-akscluster-25291065-vmss000000 E0227 18:52:42.220959 3224 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/csi.trident.netapp.io^pvc-176a73f0-2eab-4c39-9950-1b0258bcd174 podName: nodeName:}" failed. No retrie
s permitted until 2022-02-27 18:52:50.220936435 +0000 UTC m=+1608.254881107 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "pvc-176a73f0-2eab-4c39-9950-1b0258bcd174" (UniqueName: "kubernetes.io/csi/csi.trident.netapp.io^pvc-176a73f0-2eab-4c39-9950-1b0258bcd174") pod "prometheus-prometheus-kube-prometheus-prometheus-0" (UID:
"e9f3b6c1-aa79-4263-982f-977eab160de4") : rpc error: code = DeadlineExceeded desc = context deadline exceeded
10m Warning KubeletRPCDeadlineExceeded node/aks-akscluster-25291065-vmss000000 E0227 18:54:50.317495 3224 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/csi.trident.netapp.io^pvc-176a73f0-2eab-4c39-9950-1b0258bcd174 podName: nodeName:}" failed. No retrie
s permitted until 2022-02-27 18:55:06.317469036 +0000 UTC m=+1744.351413708 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "pvc-176a73f0-2eab-4c39-9950-1b0258bcd174" (UniqueName: "kubernetes.io/csi/csi.trident.netapp.io^pvc-176a73f0-2eab-4c39-9950-1b0258bcd174") pod "prometheus-prometheus-kube-prometheus-prometheus-0" (UID
: "e9f3b6c1-aa79-4263-982f-977eab160de4") : rpc error: code = DeadlineExceeded desc = context deadline exceeded
7m46s Warning KubeletRPCDeadlineExceeded node/aks-akscluster-25291065-vmss000000 E0227 18:57:06.376010 3224 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/csi.trident.netapp.io^pvc-176a73f0-2eab-4c39-9950-1b0258bcd174 podName: nodeName:}" failed. No retrie
s permitted until 2022-02-27 18:57:38.375981391 +0000 UTC m=+1896.409926163 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "pvc-176a73f0-2eab-4c39-9950-1b0258bcd174" (UniqueName: "kubernetes.io/csi/csi.trident.netapp.io^pvc-176a73f0-2eab-4c39-9950-1b0258bcd174") pod "prometheus-prometheus-kube-prometheus-prometheus-0" (UID
: "e9f3b6c1-aa79-4263-982f-977eab160de4") : rpc error: code = DeadlineExceeded desc = context deadline exceeded
5m14s Warning KubeletRPCDeadlineExceeded node/aks-akscluster-25291065-vmss000000 E0227 18:59:38.387380 3224 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/csi.trident.netapp.io^pvc-176a73f0-2eab-4c39-9950-1b0258bcd174 podName: nodeName:}" failed. No retrie
s permitted until 2022-02-27 19:00:42.387345942 +0000 UTC m=+2080.421290614 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "pvc-176a73f0-2eab-4c39-9950-1b0258bcd174" (UniqueName: "kubernetes.io/csi/csi.trident.netapp.io^pvc-176a73f0-2eab-4c39-9950-1b0258bcd174") pod "prometheus-prometheus-kube-prometheus-prometheus-0" (UI
D: "e9f3b6c1-aa79-4263-982f-977eab160de4") : rpc error: code = DeadlineExceeded desc = context deadline exceeded
2m10s Warning KubeletRPCDeadlineExceeded node/aks-akscluster-25291065-vmss000000 E0227 19:02:42.401601 3224 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/csi.trident.netapp.io^pvc-176a73f0-2eab-4c39-9950-1b0258bcd174 podName: nodeName:}" failed. No retrie
s permitted until 2022-02-27 19:04:44.40157143 +0000 UTC m=+2322.435516102 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "pvc-176a73f0-2eab-4c39-9950-1b0258bcd174" (UniqueName: "kubernetes.io/csi/csi.trident.netapp.io^pvc-176a73f0-2eab-4c39-9950-1b0258bcd174") pod "prometheus-prometheus-kube-prometheus-prometheus-0" (UID
: "e9f3b6c1-aa79-4263-982f-977eab160de4") : rpc error: code = DeadlineExceeded desc = context deadline exceeded
20m Normal Starting node/aks-akscluster-25291065-vmss000001 Starting kubelet.
20m Warning InvalidDiskCapacity node/aks-akscluster-25291065-vmss000001 invalid capacity 0 on image filesystem
20m Normal NodeHasSufficientMemory node/aks-akscluster-25291065-vmss000001 Node aks-akscluster-25291065-vmss000001 status is now: NodeHasSufficientMemory
20m Normal NodeHasNoDiskPressure node/aks-akscluster-25291065-vmss000001 Node aks-akscluster-25291065-vmss000001 status is now: NodeHasNoDiskPressure
20m Normal NodeHasSufficientPID node/aks-akscluster-25291065-vmss000001 Node aks-akscluster-25291065-vmss000001 status is now: NodeHasSufficientPID
20m Normal NodeAllocatableEnforced node/aks-akscluster-25291065-vmss000001 Updated Node Allocatable limit across pods
20m Normal RegisteredNode node/aks-akscluster-25291065-vmss000001 Node aks-akscluster-25291065-vmss000001 event: Registered Node aks-akscluster-25291065-vmss000001 in Controller
20m Normal NodeReady node/aks-akscluster-25291065-vmss000001 Node aks-akscluster-25291065-vmss000001 status is now: NodeReady
20m Normal Starting node/aks-akscluster-25291065-vmss000001
20m Warning listen tcp4 :31579: bind: address already in use node/aks-akscluster-25291065-vmss000001 can't open port "nodePort for ingress/nginx-ingress-controller:http" (:31579/tcp4), skipping it
20m Warning listen tcp4 :32165: bind: address already in use node/aks-akscluster-25291065-vmss000001 can't open port "nodePort for ingress/nginx-ingress-controller:https" (:32165/tcp4), skipping it
14m Warning KubeletRPCDeadlineExceeded node/aks-akscluster-25291065-vmss000001 E0227 18:47:39.806067 3278 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/csi.trident.netapp.io^pvc-e7e87fa5-5396-43ed-b53c-e67dfd2a9f57 podName: nodeName:}" failed. No retrie
s permitted until 2022-02-27 18:47:40.306043537 +0000 UTC m=+213.349132244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-e7e87fa5-5396-43ed-b53c-e67dfd2a9f57" (UniqueName: "kubernetes.io/csi/csi.trident.netapp.io^pvc-e7e87fa5-5396-43ed-b53c-e67dfd2a9f57") pod "wordpress-686b57c696-qzxb6" (UID: "d98109c3-55d0-4130-b
55c-65cf87621345") : rpc error: code = DeadlineExceeded desc = context deadline exceeded
14m Warning KubeletRPCDeadlineExceeded node/aks-akscluster-25291065-vmss000001 E0227 18:49:40.348043 3278 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/csi.trident.ne
PS C:\Users\danny> kubectl logs trident-operator-5b796496d5-mk62p -n netapp --kubeconfig config
time="2022-02-27T21:24:55Z" level=info msg="Running Trident operator." binary=/trident-operator build_time="Mon Jan 31 15:52:05 UTC 2022" version=22.01.0
time="2022-02-27T21:24:55Z" level=warning msg="Could not get all server resources." error="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
time="2022-02-27T21:24:55Z" level=info msg="Created Kubernetes clients." namespace=netapp version=v1.23.3
time="2022-02-27T21:24:55Z" level=info msg="Initializing controller." Controller="Trident Orchestrator"
time="2022-02-27T21:24:55Z" level=info msg="Activating controller." Controller="Trident Orchestrator"
time="2022-02-27T21:24:55Z" level=warning msg="Could not get all server resources." error="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
time="2022-02-27T21:24:55Z" level=info msg="CRD present." CRD=tridentorchestrators.trident.netapp.io
time="2022-02-27T21:24:55Z" level=info msg="Starting workers"
time="2022-02-27T21:24:55Z" level=info msg="Started workers"
time="2022-02-27T21:24:55Z" level=info msg="CR added." CR=trident CRD=TridentOrchestrator
time="2022-02-27T21:24:55Z" level=info msg="Trident deployment not found."
time="2022-02-27T21:24:55Z" level=info msg="CRD not found." CRD=tridentprovisioners.trident.netapp.io
time="2022-02-27T21:24:55Z" level=info msg="Reconciler found no operator-based Trident installation."
time="2022-02-27T21:24:55Z" level=info msg="CR updated." CR=trident CRD=TridentOrchestrator
time="2022-02-27T21:24:55Z" level=warning msg="Could not get all server resources." error="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
time="2022-02-27T21:24:55Z" level=info msg="Current installed Trident version is not same as the new Trident image version; need to update the Trident installation." currentVersion= newVersion=v22.01.0
time="2022-02-27T21:24:55Z" level=info msg="Starting Trident installation." namespace=netapp
time="2022-02-27T21:24:55Z" level=info msg="Trident service account not found."
time="2022-02-27T21:24:55Z" level=info msg="Created service account."
time="2022-02-27T21:24:55Z" level=info msg="Trident cluster role not found."
time="2022-02-27T21:24:55Z" level=info msg="Created cluster role."
time="2022-02-27T21:24:55Z" level=info msg="Trident cluster role binding not found."
time="2022-02-27T21:24:55Z" level=info msg="Created cluster role binding."
time="2022-02-27T21:24:55Z" level=info msg="Installer will create a fresh CRD." CRD=tridentversions.trident.netapp.io
time="2022-02-27T21:24:55Z" level=info msg="Created CRD." CRD=tridentversions.trident.netapp.io
time="2022-02-27T21:24:56Z" level=info msg="Installer will create a fresh CRD." CRD=tridentbackends.trident.netapp.io
time="2022-02-27T21:24:56Z" level=info msg="Created CRD." CRD=tridentbackends.trident.netapp.io
time="2022-02-27T21:24:56Z" level=info msg="Installer will create a fresh CRD." CRD=tridentmirrorrelationships.trident.netapp.io
time="2022-02-27T21:24:56Z" level=info msg="Created CRD." CRD=tridentmirrorrelationships.trident.netapp.io
time="2022-02-27T21:24:57Z" level=info msg="Installer will create a fresh CRD." CRD=tridentsnapshotinfos.trident.netapp.io
time="2022-02-27T21:24:57Z" level=info msg="Created CRD." CRD=tridentsnapshotinfos.trident.netapp.io
time="2022-02-27T21:24:57Z" level=info msg="Installer will create a fresh CRD." CRD=tridentbackendconfigs.trident.netapp.io
time="2022-02-27T21:24:57Z" level=info msg="Created CRD." CRD=tridentbackendconfigs.trident.netapp.io
time="2022-02-27T21:24:58Z" level=info msg="Installer will create a fresh CRD." CRD=tridentstorageclasses.trident.netapp.io
time="2022-02-27T21:24:58Z" level=info msg="Created CRD." CRD=tridentstorageclasses.trident.netapp.io
time="2022-02-27T21:24:59Z" level=info msg="Installer will create a fresh CRD." CRD=tridentvolumes.trident.netapp.io
time="2022-02-27T21:24:59Z" level=info msg="Created CRD." CRD=tridentvolumes.trident.netapp.io
time="2022-02-27T21:24:59Z" level=info msg="Installer will create a fresh CRD." CRD=tridentvolumepublications.trident.netapp.io
time="2022-02-27T21:24:59Z" level=info msg="Created CRD." CRD=tridentvolumepublications.trident.netapp.io
time="2022-02-27T21:25:00Z" level=info msg="Installer will create a fresh CRD." CRD=tridentnodes.trident.netapp.io
time="2022-02-27T21:25:00Z" level=info msg="Created CRD." CRD=tridentnodes.trident.netapp.io
time="2022-02-27T21:25:00Z" level=info msg="Installer will create a fresh CRD." CRD=tridenttransactions.trident.netapp.io
time="2022-02-27T21:25:00Z" level=info msg="Created CRD." CRD=tridenttransactions.trident.netapp.io
time="2022-02-27T21:25:01Z" level=info msg="Installer will create a fresh CRD." CRD=tridentsnapshots.trident.netapp.io
time="2022-02-27T21:25:01Z" level=info msg="Created CRD." CRD=tridentsnapshots.trident.netapp.io
W0227 21:25:04.379136 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:25:04Z" level=info msg="Trident pod security policy not found."
W0227 21:25:04.391139 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0227 21:25:04.403872 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:25:04Z" level=info msg="Created Trident Pod security policy."
time="2022-02-27T21:25:04Z" level=info msg="CSI driver custom resource not found."
time="2022-02-27T21:25:04Z" level=info msg="Created CSI driver custom resource."
time="2022-02-27T21:25:04Z" level=info msg="Trident service not found."
time="2022-02-27T21:25:04Z" level=info msg="Created Trident service."
time="2022-02-27T21:25:04Z" level=info msg="Trident secret not found."
time="2022-02-27T21:25:04Z" level=info msg="No Trident deployments found by label." label="app=controller.csi.trident.netapp.io" namespace=netapp
time="2022-02-27T21:25:04Z" level=info msg="Trident deployment added or deleted." name=trident-csi namespace=netapp
time="2022-02-27T21:25:04Z" level=info msg="Created Trident deployment."
time="2022-02-27T21:25:04Z" level=info msg="No Trident daemonsets found by label." label="app=node.csi.trident.netapp.io" namespace=netapp
time="2022-02-27T21:25:04Z" level=info msg="Created Trident daemonset."
time="2022-02-27T21:25:04Z" level=info msg="Trident daemonset added or deleted." name=trident-csi namespace=netapp
time="2022-02-27T21:25:04Z" level=info msg="Trident deployment changed." name=trident-csi namespace=netapp
time="2022-02-27T21:25:04Z" level=info msg="Trident daemonset changed." name=trident-csi namespace=netapp
time="2022-02-27T21:25:04Z" level=info msg="Trident deployment changed." name=trident-csi namespace=netapp
time="2022-02-27T21:25:04Z" level=info msg="Trident deployment changed." name=trident-csi namespace=netapp
time="2022-02-27T21:25:11Z" level=info msg="Waiting for Trident pod to start."
time="2022-02-27T21:25:26Z" level=info msg="Trident deployment changed." name=trident-csi namespace=netapp
time="2022-02-27T21:25:31Z" level=info msg="Trident pod started." namespace=netapp pod=trident-csi-558f9448cd-9hb4s
time="2022-02-27T21:25:31Z" level=info msg="Waiting for Trident REST interface."
time="2022-02-27T21:25:31Z" level=info msg="Trident REST interface is up." version=22.01.0
time="2022-02-27T21:25:31Z" level=info msg="Trident is installed." namespace=netapp version=v22.01.0
time="2022-02-27T21:25:31Z" level=info msg="CR updated." CR=trident CRD=TridentOrchestrator
time="2022-02-27T21:25:31Z" level=info msg="Synced Trident Orchestrator CR 'trident'"
time="2022-02-27T21:25:31Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:25:31Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:25:31Z" level=info msg="An operator based Trident CSI deployment was found." deploymentName=trident-csi deploymentNamespace=netapp
time="2022-02-27T21:25:31Z" level=info msg="CRD not found." CRD=tridentprovisioners.trident.netapp.io
time="2022-02-27T21:25:31Z" level=info msg="Reconciler found Trident installation." callingCRName=trident-csi callingResourceType=resourceDeployment controllingCRBasedOnStatus=trident isCSI=true namespace=netapp operatorBasedCSIDeployments="[trident-csi]"
time="2022-02-27T21:25:31Z" level=info msg="Found CR that controls current Trident deployment." name=trident
time="2022-02-27T21:25:31Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:25:31Z" level=info msg="Starting Trident installation." namespace=netapp
time="2022-02-27T21:25:31Z" level=info msg="A Trident Service account found by label." namespace=netapp serviceAccount=trident-csi
time="2022-02-27T21:25:31Z" level=info msg="A Trident cluster role found by label." clusterRole=trident-csi
time="2022-02-27T21:25:31Z" level=info msg="A Trident Cluster role binding was found by label." clusterRoleBinding=trident-csi
time="2022-02-27T21:25:31Z" level=info msg="CRD present." CRD=tridentversions.trident.netapp.io
time="2022-02-27T21:25:31Z" level=info msg="CRD present." CRD=tridentbackends.trident.netapp.io
time="2022-02-27T21:25:31Z" level=info msg="CRD present." CRD=tridentmirrorrelationships.trident.netapp.io
time="2022-02-27T21:25:31Z" level=info msg="CRD present." CRD=tridentsnapshotinfos.trident.netapp.io
time="2022-02-27T21:25:31Z" level=info msg="CRD present." CRD=tridentbackendconfigs.trident.netapp.io
time="2022-02-27T21:25:31Z" level=info msg="CRD present." CRD=tridentstorageclasses.trident.netapp.io
time="2022-02-27T21:25:31Z" level=info msg="CRD present." CRD=tridentvolumes.trident.netapp.io
time="2022-02-27T21:25:31Z" level=info msg="CRD present." CRD=tridentvolumepublications.trident.netapp.io
time="2022-02-27T21:25:31Z" level=info msg="CRD present." CRD=tridentnodes.trident.netapp.io
time="2022-02-27T21:25:31Z" level=info msg="CRD present." CRD=tridenttransactions.trident.netapp.io
time="2022-02-27T21:25:31Z" level=info msg="CRD present." CRD=tridentsnapshots.trident.netapp.io
W0227 21:25:31.999224 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:25:31Z" level=info msg="A Trident Pod security policy was found by label." podSecurityPolicy=tridentpods
W0227 21:25:32.006875 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0227 21:25:32.014119 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:25:32Z" level=info msg="A Trident CSI driver found by label." TridentCSIDriver=csi.trident.netapp.io
time="2022-02-27T21:25:32Z" level=info msg="A Trident service was found by label." namespace=netapp service=trident-csi
time="2022-02-27T21:25:32Z" level=info msg="A Trident secret was found by label." namespace=netapp secret=trident-csi
time="2022-02-27T21:25:32Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:25:32Z" level=info msg="A Trident daemonSet was found by label." daemonSet=trident-csi namespace=netapp
time="2022-02-27T21:25:39Z" level=info msg="Waiting for Trident pod to start."
time="2022-02-27T21:25:39Z" level=info msg="Trident pod started." namespace=netapp pod=trident-csi-558f9448cd-9hb4s
time="2022-02-27T21:25:39Z" level=info msg="Waiting for Trident REST interface."
time="2022-02-27T21:25:39Z" level=info msg="Trident REST interface is up." version=22.01.0
time="2022-02-27T21:25:39Z" level=info msg="Trident is installed." namespace=netapp version=v22.01.0
time="2022-02-27T21:25:39Z" level=info msg="New status is same as the old status, no update needed." tridentOrchestratorCR=trident
time="2022-02-27T21:25:39Z" level=info msg="Synced deployment 'netapp/trident-csi'"
time="2022-02-27T21:25:39Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:25:39Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:25:39Z" level=info msg="An operator based Trident CSI deployment was found." deploymentName=trident-csi deploymentNamespace=netapp
time="2022-02-27T21:25:39Z" level=info msg="CRD not found." CRD=tridentprovisioners.trident.netapp.io
time="2022-02-27T21:25:39Z" level=info msg="Reconciler found Trident installation." callingCRName=trident-csi callingResourceType=resourceDaemonset controllingCRBasedOnStatus=trident isCSI=true namespace=netapp operatorBasedCSIDeployments="[trident-csi]"
time="2022-02-27T21:25:39Z" level=info msg="Found CR that controls current Trident deployment." name=trident
time="2022-02-27T21:25:39Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:25:39Z" level=info msg="Starting Trident installation." namespace=netapp
time="2022-02-27T21:25:39Z" level=info msg="A Trident Service account found by label." namespace=netapp serviceAccount=trident-csi
time="2022-02-27T21:25:39Z" level=info msg="A Trident cluster role found by label." clusterRole=trident-csi
time="2022-02-27T21:25:39Z" level=info msg="A Trident Cluster role binding was found by label." clusterRoleBinding=trident-csi
time="2022-02-27T21:25:39Z" level=info msg="CRD present." CRD=tridentversions.trident.netapp.io
time="2022-02-27T21:25:39Z" level=info msg="CRD present." CRD=tridentbackends.trident.netapp.io
time="2022-02-27T21:25:39Z" level=info msg="CRD present." CRD=tridentmirrorrelationships.trident.netapp.io
time="2022-02-27T21:25:39Z" level=info msg="CRD present." CRD=tridentsnapshotinfos.trident.netapp.io
time="2022-02-27T21:25:39Z" level=info msg="CRD present." CRD=tridentbackendconfigs.trident.netapp.io
time="2022-02-27T21:25:39Z" level=info msg="CRD present." CRD=tridentstorageclasses.trident.netapp.io
time="2022-02-27T21:25:39Z" level=info msg="CRD present." CRD=tridentvolumes.trident.netapp.io
time="2022-02-27T21:25:39Z" level=info msg="CRD present." CRD=tridentvolumepublications.trident.netapp.io
time="2022-02-27T21:25:39Z" level=info msg="CRD present." CRD=tridentnodes.trident.netapp.io
time="2022-02-27T21:25:39Z" level=info msg="CRD present." CRD=tridenttransactions.trident.netapp.io
time="2022-02-27T21:25:39Z" level=info msg="CRD present." CRD=tridentsnapshots.trident.netapp.io
W0227 21:25:39.998696 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:25:39Z" level=info msg="A Trident Pod security policy was found by label." podSecurityPolicy=tridentpods
W0227 21:25:40.004833 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0227 21:25:40.013701 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:25:40Z" level=info msg="A Trident CSI driver found by label." TridentCSIDriver=csi.trident.netapp.io
time="2022-02-27T21:25:40Z" level=info msg="A Trident service was found by label." namespace=netapp service=trident-csi
time="2022-02-27T21:25:40Z" level=info msg="A Trident secret was found by label." namespace=netapp secret=trident-csi
time="2022-02-27T21:25:40Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:25:40Z" level=info msg="A Trident daemonSet was found by label." daemonSet=trident-csi namespace=netapp
time="2022-02-27T21:25:44Z" level=info msg="Trident daemonset changed." name=trident-csi namespace=netapp
time="2022-02-27T21:25:47Z" level=info msg="Waiting for Trident pod to start."
time="2022-02-27T21:25:47Z" level=info msg="Trident pod started." namespace=netapp pod=trident-csi-558f9448cd-9hb4s
time="2022-02-27T21:25:47Z" level=info msg="Waiting for Trident REST interface."
time="2022-02-27T21:25:47Z" level=info msg="Trident REST interface is up." version=22.01.0
time="2022-02-27T21:25:47Z" level=info msg="Trident is installed." namespace=netapp version=v22.01.0
time="2022-02-27T21:25:47Z" level=info msg="New status is same as the old status, no update needed." tridentOrchestratorCR=trident
time="2022-02-27T21:25:47Z" level=info msg="Synced daemonset 'netapp/trident-csi'"
time="2022-02-27T21:25:47Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:25:47Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:25:47Z" level=info msg="An operator based Trident CSI deployment was found." deploymentName=trident-csi deploymentNamespace=netapp
time="2022-02-27T21:25:47Z" level=info msg="CRD not found." CRD=tridentprovisioners.trident.netapp.io
time="2022-02-27T21:25:47Z" level=info msg="Reconciler found Trident installation." callingCRName=trident callingResourceType=resourceTridentOrchestratorCR controllingCRBasedOnStatus=trident isCSI=true namespace=netapp operatorBasedCSIDeployments="[trident-csi]"
time="2022-02-27T21:25:47Z" level=info msg="Found CR that controls current Trident deployment." name=trident
time="2022-02-27T21:25:47Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:25:47Z" level=info msg="Starting Trident installation." namespace=netapp
time="2022-02-27T21:25:47Z" level=info msg="A Trident Service account found by label." namespace=netapp serviceAccount=trident-csi
time="2022-02-27T21:25:47Z" level=info msg="A Trident cluster role found by label." clusterRole=trident-csi
time="2022-02-27T21:25:47Z" level=info msg="A Trident Cluster role binding was found by label." clusterRoleBinding=trident-csi
time="2022-02-27T21:25:47Z" level=info msg="CRD present." CRD=tridentversions.trident.netapp.io
time="2022-02-27T21:25:47Z" level=info msg="CRD present." CRD=tridentbackends.trident.netapp.io
time="2022-02-27T21:25:47Z" level=info msg="CRD present." CRD=tridentmirrorrelationships.trident.netapp.io
time="2022-02-27T21:25:47Z" level=info msg="CRD present." CRD=tridentsnapshotinfos.trident.netapp.io
time="2022-02-27T21:25:47Z" level=info msg="CRD present." CRD=tridentbackendconfigs.trident.netapp.io
time="2022-02-27T21:25:47Z" level=info msg="CRD present." CRD=tridentstorageclasses.trident.netapp.io
time="2022-02-27T21:25:47Z" level=info msg="CRD present." CRD=tridentvolumes.trident.netapp.io
time="2022-02-27T21:25:47Z" level=info msg="CRD present." CRD=tridentvolumepublications.trident.netapp.io
time="2022-02-27T21:25:47Z" level=info msg="CRD present." CRD=tridentnodes.trident.netapp.io
time="2022-02-27T21:25:47Z" level=info msg="CRD present." CRD=tridenttransactions.trident.netapp.io
time="2022-02-27T21:25:47Z" level=info msg="CRD present." CRD=tridentsnapshots.trident.netapp.io
W0227 21:25:48.006867 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:25:48Z" level=info msg="A Trident Pod security policy was found by label." podSecurityPolicy=tridentpods
W0227 21:25:48.018701 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0227 21:25:48.025810 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:25:48Z" level=info msg="A Trident CSI driver found by label." TridentCSIDriver=csi.trident.netapp.io
time="2022-02-27T21:25:48Z" level=info msg="A Trident service was found by label." namespace=netapp service=trident-csi
time="2022-02-27T21:25:48Z" level=info msg="A Trident secret was found by label." namespace=netapp secret=trident-csi
time="2022-02-27T21:25:48Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:25:48Z" level=info msg="A Trident daemonSet was found by label." daemonSet=trident-csi namespace=netapp
time="2022-02-27T21:25:55Z" level=info msg="Waiting for Trident pod to start."
time="2022-02-27T21:25:55Z" level=info msg="Trident pod started." namespace=netapp pod=trident-csi-558f9448cd-9hb4s
time="2022-02-27T21:25:55Z" level=info msg="Waiting for Trident REST interface."
time="2022-02-27T21:25:55Z" level=info msg="Trident REST interface is up." version=22.01.0
time="2022-02-27T21:25:55Z" level=info msg="Trident is installed." namespace=netapp version=v22.01.0
time="2022-02-27T21:25:55Z" level=info msg="New status is same as the old status, no update needed." tridentOrchestratorCR=trident
time="2022-02-27T21:25:55Z" level=info msg="Synced Trident Orchestrator CR 'trident'"
time="2022-02-27T21:25:55Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:25:55Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:25:55Z" level=info msg="An operator based Trident CSI deployment was found." deploymentName=trident-csi deploymentNamespace=netapp
time="2022-02-27T21:25:55Z" level=info msg="CRD not found." CRD=tridentprovisioners.trident.netapp.io
time="2022-02-27T21:25:55Z" level=info msg="Reconciler found Trident installation." callingCRName=trident-csi callingResourceType=resourceDaemonset controllingCRBasedOnStatus=trident isCSI=true namespace=netapp operatorBasedCSIDeployments="[trident-csi]"
time="2022-02-27T21:25:55Z" level=info msg="Found CR that controls current Trident deployment." name=trident
time="2022-02-27T21:25:55Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:25:55Z" level=info msg="Starting Trident installation." namespace=netapp
time="2022-02-27T21:25:55Z" level=info msg="A Trident Service account found by label." namespace=netapp serviceAccount=trident-csi
time="2022-02-27T21:25:55Z" level=info msg="A Trident cluster role found by label." clusterRole=trident-csi
time="2022-02-27T21:25:55Z" level=info msg="A Trident Cluster role binding was found by label." clusterRoleBinding=trident-csi
time="2022-02-27T21:25:55Z" level=info msg="CRD present." CRD=tridentversions.trident.netapp.io
time="2022-02-27T21:25:55Z" level=info msg="CRD present." CRD=tridentbackends.trident.netapp.io
time="2022-02-27T21:25:55Z" level=info msg="CRD present." CRD=tridentmirrorrelationships.trident.netapp.io
time="2022-02-27T21:25:55Z" level=info msg="CRD present." CRD=tridentsnapshotinfos.trident.netapp.io
time="2022-02-27T21:25:55Z" level=info msg="CRD present." CRD=tridentbackendconfigs.trident.netapp.io
time="2022-02-27T21:25:55Z" level=info msg="CRD present." CRD=tridentstorageclasses.trident.netapp.io
time="2022-02-27T21:25:55Z" level=info msg="CRD present." CRD=tridentvolumes.trident.netapp.io
time="2022-02-27T21:25:55Z" level=info msg="CRD present." CRD=tridentvolumepublications.trident.netapp.io
time="2022-02-27T21:25:55Z" level=info msg="CRD present." CRD=tridentnodes.trident.netapp.io
time="2022-02-27T21:25:55Z" level=info msg="CRD present." CRD=tridenttransactions.trident.netapp.io
time="2022-02-27T21:25:55Z" level=info msg="CRD present." CRD=tridentsnapshots.trident.netapp.io
W0227 21:25:55.967104 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:25:55Z" level=info msg="A Trident Pod security policy was found by label." podSecurityPolicy=tridentpods
W0227 21:25:55.973411 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0227 21:25:55.981887 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:25:55Z" level=info msg="A Trident CSI driver found by label." TridentCSIDriver=csi.trident.netapp.io
time="2022-02-27T21:25:56Z" level=info msg="A Trident service was found by label." namespace=netapp service=trident-csi
time="2022-02-27T21:25:56Z" level=info msg="A Trident secret was found by label." namespace=netapp secret=trident-csi
time="2022-02-27T21:25:56Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:25:56Z" level=info msg="A Trident daemonSet was found by label." daemonSet=trident-csi namespace=netapp
time="2022-02-27T21:26:03Z" level=info msg="Waiting for Trident pod to start."
time="2022-02-27T21:26:03Z" level=info msg="Trident pod started." namespace=netapp pod=trident-csi-558f9448cd-9hb4s
time="2022-02-27T21:26:03Z" level=info msg="Waiting for Trident REST interface."
time="2022-02-27T21:26:03Z" level=info msg="Trident REST interface is up." version=22.01.0
time="2022-02-27T21:26:03Z" level=info msg="Trident is installed." namespace=netapp version=v22.01.0
time="2022-02-27T21:26:03Z" level=info msg="New status is same as the old status, no update needed." tridentOrchestratorCR=trident
time="2022-02-27T21:26:03Z" level=info msg="Synced daemonset 'netapp/trident-csi'"
time="2022-02-27T21:26:03Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:26:03Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:29:55Z" level=info msg="CR updated." CR=trident CRD=TridentOrchestrator
time="2022-02-27T21:29:55Z" level=info msg="An operator based Trident CSI deployment was found." deploymentName=trident-csi deploymentNamespace=netapp
time="2022-02-27T21:29:55Z" level=info msg="CRD not found." CRD=tridentprovisioners.trident.netapp.io
time="2022-02-27T21:29:55Z" level=info msg="Reconciler found Trident installation." callingCRName=trident callingResourceType=resourceTridentOrchestratorCR controllingCRBasedOnStatus=trident isCSI=true namespace=netapp operatorBasedCSIDeployments="[trident-csi]"
time="2022-02-27T21:29:55Z" level=info msg="Found CR that controls current Trident deployment." name=trident
time="2022-02-27T21:29:55Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:29:55Z" level=info msg="Starting Trident installation." namespace=netapp
time="2022-02-27T21:29:55Z" level=info msg="A Trident Service account found by label." namespace=netapp serviceAccount=trident-csi
time="2022-02-27T21:29:55Z" level=info msg="A Trident cluster role found by label." clusterRole=trident-csi
time="2022-02-27T21:29:55Z" level=info msg="A Trident Cluster role binding was found by label." clusterRoleBinding=trident-csi
time="2022-02-27T21:29:55Z" level=info msg="CRD present." CRD=tridentversions.trident.netapp.io
time="2022-02-27T21:29:55Z" level=info msg="CRD present." CRD=tridentbackends.trident.netapp.io
time="2022-02-27T21:29:55Z" level=info msg="CRD present." CRD=tridentmirrorrelationships.trident.netapp.io
time="2022-02-27T21:29:55Z" level=info msg="CRD present." CRD=tridentsnapshotinfos.trident.netapp.io
time="2022-02-27T21:29:55Z" level=info msg="CRD present." CRD=tridentbackendconfigs.trident.netapp.io
time="2022-02-27T21:29:55Z" level=info msg="CRD present." CRD=tridentstorageclasses.trident.netapp.io
time="2022-02-27T21:29:55Z" level=info msg="CRD present." CRD=tridentvolumes.trident.netapp.io
time="2022-02-27T21:29:55Z" level=info msg="CRD present." CRD=tridentvolumepublications.trident.netapp.io
time="2022-02-27T21:29:55Z" level=info msg="CRD present." CRD=tridentnodes.trident.netapp.io
time="2022-02-27T21:29:55Z" level=info msg="CRD present." CRD=tridenttransactions.trident.netapp.io
time="2022-02-27T21:29:55Z" level=info msg="CRD present." CRD=tridentsnapshots.trident.netapp.io
W0227 21:29:56.030552 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:29:56Z" level=info msg="A Trident Pod security policy was found by label." podSecurityPolicy=tridentpods
W0227 21:29:56.037413 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0227 21:29:56.044346 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:29:56Z" level=info msg="A Trident CSI driver found by label." TridentCSIDriver=csi.trident.netapp.io
time="2022-02-27T21:29:56Z" level=info msg="A Trident service was found by label." namespace=netapp service=trident-csi
time="2022-02-27T21:29:56Z" level=info msg="A Trident secret was found by label." namespace=netapp secret=trident-csi
time="2022-02-27T21:29:56Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:29:56Z" level=info msg="A Trident daemonSet was found by label." daemonSet=trident-csi namespace=netapp
time="2022-02-27T21:30:03Z" level=info msg="Waiting for Trident pod to start."
time="2022-02-27T21:30:03Z" level=info msg="Trident pod started." namespace=netapp pod=trident-csi-558f9448cd-9hb4s
time="2022-02-27T21:30:03Z" level=info msg="Waiting for Trident REST interface."
time="2022-02-27T21:30:03Z" level=info msg="Trident REST interface is up." version=22.01.0
time="2022-02-27T21:30:03Z" level=info msg="Trident is installed." namespace=netapp version=v22.01.0
time="2022-02-27T21:30:03Z" level=info msg="New status is same as the old status, no update needed." tridentOrchestratorCR=trident
time="2022-02-27T21:30:03Z" level=info msg="Synced Trident Orchestrator CR 'trident'"
time="2022-02-27T21:30:03Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:30:03Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:31:11Z" level=info msg="CR updated." CR=trident CRD=TridentOrchestrator
time="2022-02-27T21:31:11Z" level=info msg="An operator based Trident CSI deployment was found." deploymentName=trident-csi deploymentNamespace=netapp
time="2022-02-27T21:31:11Z" level=info msg="CRD not found." CRD=tridentprovisioners.trident.netapp.io
time="2022-02-27T21:31:11Z" level=info msg="Reconciler found Trident installation." callingCRName=trident callingResourceType=resourceTridentOrchestratorCR controllingCRBasedOnStatus=trident isCSI=true namespace=netapp operatorBasedCSIDeployments="[trident-csi]"
time="2022-02-27T21:31:11Z" level=info msg="Found CR that controls current Trident deployment." name=trident
time="2022-02-27T21:31:11Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:31:12Z" level=info msg="Starting Trident installation." namespace=netapp
time="2022-02-27T21:31:12Z" level=info msg="A Trident Service account found by label." namespace=netapp serviceAccount=trident-csi
time="2022-02-27T21:31:12Z" level=info msg="A Trident cluster role found by label." clusterRole=trident-csi
time="2022-02-27T21:31:12Z" level=info msg="A Trident Cluster role binding was found by label." clusterRoleBinding=trident-csi
time="2022-02-27T21:31:12Z" level=info msg="CRD present." CRD=tridentversions.trident.netapp.io
time="2022-02-27T21:31:12Z" level=info msg="CRD present." CRD=tridentbackends.trident.netapp.io
time="2022-02-27T21:31:12Z" level=info msg="CRD present." CRD=tridentmirrorrelationships.trident.netapp.io
time="2022-02-27T21:31:12Z" level=info msg="CRD present." CRD=tridentsnapshotinfos.trident.netapp.io
time="2022-02-27T21:31:12Z" level=info msg="CRD present." CRD=tridentbackendconfigs.trident.netapp.io
time="2022-02-27T21:31:12Z" level=info msg="CRD present." CRD=tridentstorageclasses.trident.netapp.io
time="2022-02-27T21:31:12Z" level=info msg="CRD present." CRD=tridentvolumes.trident.netapp.io
time="2022-02-27T21:31:12Z" level=info msg="CRD present." CRD=tridentvolumepublications.trident.netapp.io
time="2022-02-27T21:31:12Z" level=info msg="CRD present." CRD=tridentnodes.trident.netapp.io
time="2022-02-27T21:31:12Z" level=info msg="CRD present." CRD=tridenttransactions.trident.netapp.io
time="2022-02-27T21:31:12Z" level=info msg="CRD present." CRD=tridentsnapshots.trident.netapp.io
W0227 21:31:12.561125 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:31:12Z" level=info msg="A Trident Pod security policy was found by label." podSecurityPolicy=tridentpods
W0227 21:31:12.568759 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0227 21:31:12.576692 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:31:12Z" level=info msg="A Trident CSI driver found by label." TridentCSIDriver=csi.trident.netapp.io
time="2022-02-27T21:31:12Z" level=info msg="A Trident service was found by label." namespace=netapp service=trident-csi
time="2022-02-27T21:31:12Z" level=info msg="A Trident secret was found by label." namespace=netapp secret=trident-csi
time="2022-02-27T21:31:12Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:31:12Z" level=info msg="A Trident daemonSet was found by label." daemonSet=trident-csi namespace=netapp
time="2022-02-27T21:31:19Z" level=info msg="Waiting for Trident pod to start."
time="2022-02-27T21:31:19Z" level=info msg="Trident pod started." namespace=netapp pod=trident-csi-558f9448cd-9hb4s
time="2022-02-27T21:31:19Z" level=info msg="Waiting for Trident REST interface."
time="2022-02-27T21:31:19Z" level=info msg="Trident REST interface is up." version=22.01.0
time="2022-02-27T21:31:19Z" level=info msg="Trident is installed." namespace=netapp version=v22.01.0
time="2022-02-27T21:31:19Z" level=info msg="New status is same as the old status, no update needed." tridentOrchestratorCR=trident
time="2022-02-27T21:31:19Z" level=info msg="Synced Trident Orchestrator CR 'trident'"
time="2022-02-27T21:31:19Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:31:19Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:31:28Z" level=info msg="Trident daemonset changed." name=trident-csi namespace=netapp
time="2022-02-27T21:31:28Z" level=info msg="An operator based Trident CSI deployment was found." deploymentName=trident-csi deploymentNamespace=netapp
time="2022-02-27T21:31:28Z" level=info msg="CRD not found." CRD=tridentprovisioners.trident.netapp.io
time="2022-02-27T21:31:28Z" level=info msg="Reconciler found Trident installation." callingCRName=trident-csi callingResourceType=resourceDaemonset controllingCRBasedOnStatus=trident isCSI=true namespace=netapp operatorBasedCSIDeployments="[trident-csi]"
time="2022-02-27T21:31:28Z" level=info msg="Found CR that controls current Trident deployment." name=trident
time="2022-02-27T21:31:28Z" level=warning msg="Could not get all server resources." error="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
time="2022-02-27T21:31:28Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:31:29Z" level=warning msg="Could not get all server resources." error="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
time="2022-02-27T21:31:29Z" level=info msg="Starting Trident installation." namespace=netapp
time="2022-02-27T21:31:29Z" level=info msg="A Trident Service account found by label." namespace=netapp serviceAccount=trident-csi
time="2022-02-27T21:31:29Z" level=info msg="A Trident cluster role found by label." clusterRole=trident-csi
time="2022-02-27T21:31:29Z" level=info msg="A Trident Cluster role binding was found by label." clusterRoleBinding=trident-csi
time="2022-02-27T21:31:29Z" level=info msg="CRD present." CRD=tridentversions.trident.netapp.io
time="2022-02-27T21:31:29Z" level=info msg="CRD present." CRD=tridentbackends.trident.netapp.io
time="2022-02-27T21:31:29Z" level=info msg="CRD present." CRD=tridentmirrorrelationships.trident.netapp.io
time="2022-02-27T21:31:29Z" level=info msg="CRD present." CRD=tridentsnapshotinfos.trident.netapp.io
time="2022-02-27T21:31:29Z" level=info msg="CRD present." CRD=tridentbackendconfigs.trident.netapp.io
time="2022-02-27T21:31:29Z" level=info msg="CRD present." CRD=tridentstorageclasses.trident.netapp.io
time="2022-02-27T21:31:29Z" level=info msg="CRD present." CRD=tridentvolumes.trident.netapp.io
time="2022-02-27T21:31:29Z" level=info msg="CRD present." CRD=tridentvolumepublications.trident.netapp.io
time="2022-02-27T21:31:29Z" level=info msg="CRD present." CRD=tridentnodes.trident.netapp.io
time="2022-02-27T21:31:29Z" level=info msg="CRD present." CRD=tridenttransactions.trident.netapp.io
time="2022-02-27T21:31:29Z" level=info msg="CRD present." CRD=tridentsnapshots.trident.netapp.io
W0227 21:31:29.542908 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:31:29Z" level=info msg="A Trident Pod security policy was found by label." podSecurityPolicy=tridentpods
W0227 21:31:29.550602 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0227 21:31:29.560878 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:31:29Z" level=info msg="A Trident CSI driver found by label." TridentCSIDriver=csi.trident.netapp.io
time="2022-02-27T21:31:29Z" level=info msg="A Trident service was found by label." namespace=netapp service=trident-csi
time="2022-02-27T21:31:29Z" level=info msg="A Trident secret was found by label." namespace=netapp secret=trident-csi
time="2022-02-27T21:31:29Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:31:29Z" level=info msg="A Trident daemonSet was found by label." daemonSet=trident-csi namespace=netapp
time="2022-02-27T21:31:31Z" level=info msg="CR updated." CR=trident CRD=TridentOrchestrator
time="2022-02-27T21:31:36Z" level=info msg="Waiting for Trident pod to start."
time="2022-02-27T21:31:36Z" level=info msg="Trident pod started." namespace=netapp pod=trident-csi-558f9448cd-9hb4s
time="2022-02-27T21:31:36Z" level=info msg="Waiting for Trident REST interface."
time="2022-02-27T21:31:36Z" level=info msg="Trident REST interface is up." version=22.01.0
time="2022-02-27T21:31:36Z" level=info msg="Trident is installed." namespace=netapp version=v22.01.0
time="2022-02-27T21:31:36Z" level=info msg="New status is same as the old status, no update needed." tridentOrchestratorCR=trident
time="2022-02-27T21:31:36Z" level=info msg="Synced daemonset 'netapp/trident-csi'"
time="2022-02-27T21:31:36Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:31:36Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:31:36Z" level=info msg="An operator based Trident CSI deployment was found." deploymentName=trident-csi deploymentNamespace=netapp
time="2022-02-27T21:31:36Z" level=info msg="CRD not found." CRD=tridentprovisioners.trident.netapp.io
time="2022-02-27T21:31:36Z" level=info msg="Reconciler found Trident installation." callingCRName=trident callingResourceType=resourceTridentOrchestratorCR controllingCRBasedOnStatus=trident isCSI=true namespace=netapp operatorBasedCSIDeployments="[trident-csi]"
time="2022-02-27T21:31:36Z" level=info msg="Found CR that controls current Trident deployment." name=trident
time="2022-02-27T21:31:37Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:31:37Z" level=info msg="Starting Trident installation." namespace=netapp
time="2022-02-27T21:31:37Z" level=info msg="A Trident Service account found by label." namespace=netapp serviceAccount=trident-csi
time="2022-02-27T21:31:37Z" level=info msg="A Trident cluster role found by label." clusterRole=trident-csi
time="2022-02-27T21:31:37Z" level=info msg="A Trident Cluster role binding was found by label." clusterRoleBinding=trident-csi
time="2022-02-27T21:31:37Z" level=info msg="CRD present." CRD=tridentversions.trident.netapp.io
time="2022-02-27T21:31:37Z" level=info msg="CRD present." CRD=tridentbackends.trident.netapp.io
time="2022-02-27T21:31:37Z" level=info msg="CRD present." CRD=tridentmirrorrelationships.trident.netapp.io
time="2022-02-27T21:31:37Z" level=info msg="CRD present." CRD=tridentsnapshotinfos.trident.netapp.io
time="2022-02-27T21:31:37Z" level=info msg="CRD present." CRD=tridentbackendconfigs.trident.netapp.io
time="2022-02-27T21:31:37Z" level=info msg="CRD present." CRD=tridentstorageclasses.trident.netapp.io
time="2022-02-27T21:31:37Z" level=info msg="CRD present." CRD=tridentvolumes.trident.netapp.io
time="2022-02-27T21:31:37Z" level=info msg="CRD present." CRD=tridentvolumepublications.trident.netapp.io
time="2022-02-27T21:31:37Z" level=info msg="CRD present." CRD=tridentnodes.trident.netapp.io
time="2022-02-27T21:31:37Z" level=info msg="CRD present." CRD=tridenttransactions.trident.netapp.io
time="2022-02-27T21:31:37Z" level=info msg="CRD present." CRD=tridentsnapshots.trident.netapp.io
W0227 21:31:37.589143 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:31:37Z" level=info msg="A Trident Pod security policy was found by label." podSecurityPolicy=tridentpods
W0227 21:31:37.596234 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0227 21:31:37.604762 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:31:37Z" level=info msg="A Trident CSI driver found by label." TridentCSIDriver=csi.trident.netapp.io
time="2022-02-27T21:31:37Z" level=info msg="A Trident service was found by label." namespace=netapp service=trident-csi
time="2022-02-27T21:31:37Z" level=info msg="A Trident secret was found by label." namespace=netapp secret=trident-csi
time="2022-02-27T21:31:37Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:31:37Z" level=info msg="A Trident daemonSet was found by label." daemonSet=trident-csi namespace=netapp
time="2022-02-27T21:31:44Z" level=info msg="Waiting for Trident pod to start."
time="2022-02-27T21:31:44Z" level=info msg="Trident pod started." namespace=netapp pod=trident-csi-558f9448cd-9hb4s
time="2022-02-27T21:31:44Z" level=info msg="Waiting for Trident REST interface."
time="2022-02-27T21:31:44Z" level=info msg="Trident REST interface is up." version=22.01.0
time="2022-02-27T21:31:44Z" level=info msg="Trident is installed." namespace=netapp version=v22.01.0
time="2022-02-27T21:31:44Z" level=info msg="New status is same as the old status, no update needed." tridentOrchestratorCR=trident
time="2022-02-27T21:31:44Z" level=info msg="Synced Trident Orchestrator CR 'trident'"
time="2022-02-27T21:31:44Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:31:44Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:36:31Z" level=info msg="CR updated." CR=trident CRD=TridentOrchestrator
time="2022-02-27T21:36:31Z" level=info msg="An operator based Trident CSI deployment was found." deploymentName=trident-csi deploymentNamespace=netapp
time="2022-02-27T21:36:31Z" level=info msg="CRD not found." CRD=tridentprovisioners.trident.netapp.io
time="2022-02-27T21:36:31Z" level=info msg="Reconciler found Trident installation." callingCRName=trident callingResourceType=resourceTridentOrchestratorCR controllingCRBasedOnStatus=trident isCSI=true namespace=netapp operatorBasedCSIDeployments="[trident-csi]"
time="2022-02-27T21:36:31Z" level=info msg="Found CR that controls current Trident deployment." name=trident
time="2022-02-27T21:36:31Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:36:31Z" level=info msg="Starting Trident installation." namespace=netapp
time="2022-02-27T21:36:31Z" level=info msg="A Trident Service account found by label." namespace=netapp serviceAccount=trident-csi
time="2022-02-27T21:36:31Z" level=info msg="A Trident cluster role found by label." clusterRole=trident-csi
time="2022-02-27T21:36:31Z" level=info msg="A Trident Cluster role binding was found by label." clusterRoleBinding=trident-csi
time="2022-02-27T21:36:31Z" level=info msg="CRD present." CRD=tridentversions.trident.netapp.io
time="2022-02-27T21:36:31Z" level=info msg="CRD present." CRD=tridentbackends.trident.netapp.io
time="2022-02-27T21:36:31Z" level=info msg="CRD present." CRD=tridentmirrorrelationships.trident.netapp.io
time="2022-02-27T21:36:31Z" level=info msg="CRD present." CRD=tridentsnapshotinfos.trident.netapp.io
time="2022-02-27T21:36:31Z" level=info msg="CRD present." CRD=tridentbackendconfigs.trident.netapp.io
time="2022-02-27T21:36:31Z" level=info msg="CRD present." CRD=tridentstorageclasses.trident.netapp.io
time="2022-02-27T21:36:31Z" level=info msg="CRD present." CRD=tridentvolumes.trident.netapp.io
time="2022-02-27T21:36:31Z" level=info msg="CRD present." CRD=tridentvolumepublications.trident.netapp.io
time="2022-02-27T21:36:31Z" level=info msg="CRD present." CRD=tridentnodes.trident.netapp.io
time="2022-02-27T21:36:31Z" level=info msg="CRD present." CRD=tridenttransactions.trident.netapp.io
time="2022-02-27T21:36:31Z" level=info msg="CRD present." CRD=tridentsnapshots.trident.netapp.io
W0227 21:36:31.766153 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:36:31Z" level=info msg="A Trident Pod security policy was found by label." podSecurityPolicy=tridentpods
W0227 21:36:31.781734 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0227 21:36:31.791523 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:36:31Z" level=info msg="A Trident CSI driver found by label." TridentCSIDriver=csi.trident.netapp.io
time="2022-02-27T21:36:31Z" level=info msg="A Trident service was found by label." namespace=netapp service=trident-csi
time="2022-02-27T21:36:31Z" level=info msg="A Trident secret was found by label." namespace=netapp secret=trident-csi
time="2022-02-27T21:36:31Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:36:31Z" level=info msg="A Trident daemonSet was found by label." daemonSet=trident-csi namespace=netapp
time="2022-02-27T21:36:38Z" level=info msg="Waiting for Trident pod to start."
time="2022-02-27T21:36:38Z" level=info msg="Trident pod started." namespace=netapp pod=trident-csi-558f9448cd-9hb4s
time="2022-02-27T21:36:38Z" level=info msg="Waiting for Trident REST interface."
time="2022-02-27T21:36:39Z" level=info msg="Trident REST interface is up." version=22.01.0
time="2022-02-27T21:36:39Z" level=info msg="Trident is installed." namespace=netapp version=v22.01.0
time="2022-02-27T21:36:39Z" level=info msg="New status is same as the old status, no update needed." tridentOrchestratorCR=trident
time="2022-02-27T21:36:39Z" level=info msg="Synced Trident Orchestrator CR 'trident'"
time="2022-02-27T21:36:39Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:36:39Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:41:31Z" level=info msg="CR updated." CR=trident CRD=TridentOrchestrator
time="2022-02-27T21:41:31Z" level=info msg="An operator based Trident CSI deployment was found." deploymentName=trident-csi deploymentNamespace=netapp
time="2022-02-27T21:41:31Z" level=info msg="CRD not found." CRD=tridentprovisioners.trident.netapp.io
time="2022-02-27T21:41:31Z" level=info msg="Reconciler found Trident installation." callingCRName=trident callingResourceType=resourceTridentOrchestratorCR controllingCRBasedOnStatus=trident isCSI=true namespace=netapp operatorBasedCSIDeployments="[trident-csi]"
time="2022-02-27T21:41:31Z" level=info msg="Found CR that controls current Trident deployment." name=trident
time="2022-02-27T21:41:31Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:41:31Z" level=info msg="Starting Trident installation." namespace=netapp
time="2022-02-27T21:41:31Z" level=info msg="A Trident Service account found by label." namespace=netapp serviceAccount=trident-csi
time="2022-02-27T21:41:31Z" level=info msg="A Trident cluster role found by label." clusterRole=trident-csi
time="2022-02-27T21:41:31Z" level=info msg="A Trident Cluster role binding was found by label." clusterRoleBinding=trident-csi
time="2022-02-27T21:41:31Z" level=info msg="CRD present." CRD=tridentversions.trident.netapp.io
time="2022-02-27T21:41:31Z" level=info msg="CRD present." CRD=tridentbackends.trident.netapp.io
time="2022-02-27T21:41:31Z" level=info msg="CRD present." CRD=tridentmirrorrelationships.trident.netapp.io
time="2022-02-27T21:41:31Z" level=info msg="CRD present." CRD=tridentsnapshotinfos.trident.netapp.io
time="2022-02-27T21:41:31Z" level=info msg="CRD present." CRD=tridentbackendconfigs.trident.netapp.io
time="2022-02-27T21:41:31Z" level=info msg="CRD present." CRD=tridentstorageclasses.trident.netapp.io
time="2022-02-27T21:41:31Z" level=info msg="CRD present." CRD=tridentvolumes.trident.netapp.io
time="2022-02-27T21:41:31Z" level=info msg="CRD present." CRD=tridentvolumepublications.trident.netapp.io
time="2022-02-27T21:41:31Z" level=info msg="CRD present." CRD=tridentnodes.trident.netapp.io
time="2022-02-27T21:41:31Z" level=info msg="CRD present." CRD=tridenttransactions.trident.netapp.io
time="2022-02-27T21:41:31Z" level=info msg="CRD present." CRD=tridentsnapshots.trident.netapp.io
W0227 21:41:31.786720 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:41:31Z" level=info msg="A Trident Pod security policy was found by label." podSecurityPolicy=tridentpods
W0227 21:41:31.794659 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0227 21:41:31.803532 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:41:31Z" level=info msg="A Trident CSI driver found by label." TridentCSIDriver=csi.trident.netapp.io
time="2022-02-27T21:41:31Z" level=info msg="A Trident service was found by label." namespace=netapp service=trident-csi
time="2022-02-27T21:41:31Z" level=info msg="A Trident secret was found by label." namespace=netapp secret=trident-csi
time="2022-02-27T21:41:31Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:41:31Z" level=info msg="A Trident daemonSet was found by label." daemonSet=trident-csi namespace=netapp
time="2022-02-27T21:41:38Z" level=info msg="Waiting for Trident pod to start."
time="2022-02-27T21:41:38Z" level=info msg="Trident pod started." namespace=netapp pod=trident-csi-558f9448cd-9hb4s
time="2022-02-27T21:41:38Z" level=info msg="Waiting for Trident REST interface."
time="2022-02-27T21:41:39Z" level=info msg="Trident REST interface is up." version=22.01.0
time="2022-02-27T21:41:39Z" level=info msg="Trident is installed." namespace=netapp version=v22.01.0
time="2022-02-27T21:41:39Z" level=info msg="New status is same as the old status, no update needed." tridentOrchestratorCR=trident
time="2022-02-27T21:41:39Z" level=info msg="Synced Trident Orchestrator CR 'trident'"
time="2022-02-27T21:41:39Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:41:39Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:46:31Z" level=info msg="CR updated." CR=trident CRD=TridentOrchestrator
time="2022-02-27T21:46:31Z" level=info msg="An operator based Trident CSI deployment was found." deploymentName=trident-csi deploymentNamespace=netapp
time="2022-02-27T21:46:31Z" level=info msg="CRD not found." CRD=tridentprovisioners.trident.netapp.io
time="2022-02-27T21:46:31Z" level=info msg="Reconciler found Trident installation." callingCRName=trident callingResourceType=resourceTridentOrchestratorCR controllingCRBasedOnStatus=trident isCSI=true namespace=netapp operatorBasedCSIDeployments="[trident-csi]"
time="2022-02-27T21:46:31Z" level=info msg="Found CR that controls current Trident deployment." name=trident
time="2022-02-27T21:46:31Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:46:31Z" level=info msg="Starting Trident installation." namespace=netapp
time="2022-02-27T21:46:31Z" level=info msg="A Trident Service account found by label." namespace=netapp serviceAccount=trident-csi
time="2022-02-27T21:46:31Z" level=info msg="A Trident cluster role found by label." clusterRole=trident-csi
time="2022-02-27T21:46:31Z" level=info msg="A Trident Cluster role binding was found by label." clusterRoleBinding=trident-csi
time="2022-02-27T21:46:31Z" level=info msg="CRD present." CRD=tridentversions.trident.netapp.io
time="2022-02-27T21:46:31Z" level=info msg="CRD present." CRD=tridentbackends.trident.netapp.io
time="2022-02-27T21:46:31Z" level=info msg="CRD present." CRD=tridentmirrorrelationships.trident.netapp.io
time="2022-02-27T21:46:31Z" level=info msg="CRD present." CRD=tridentsnapshotinfos.trident.netapp.io
time="2022-02-27T21:46:31Z" level=info msg="CRD present." CRD=tridentbackendconfigs.trident.netapp.io
time="2022-02-27T21:46:31Z" level=info msg="CRD present." CRD=tridentstorageclasses.trident.netapp.io
time="2022-02-27T21:46:31Z" level=info msg="CRD present." CRD=tridentvolumes.trident.netapp.io
time="2022-02-27T21:46:31Z" level=info msg="CRD present." CRD=tridentvolumepublications.trident.netapp.io
time="2022-02-27T21:46:31Z" level=info msg="CRD present." CRD=tridentnodes.trident.netapp.io
time="2022-02-27T21:46:31Z" level=info msg="CRD present." CRD=tridenttransactions.trident.netapp.io
time="2022-02-27T21:46:31Z" level=info msg="CRD present." CRD=tridentsnapshots.trident.netapp.io
W0227 21:46:31.785108 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:46:31Z" level=info msg="A Trident Pod security policy was found by label." podSecurityPolicy=tridentpods
W0227 21:46:31.794644 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0227 21:46:31.803616 1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
time="2022-02-27T21:46:31Z" level=info msg="A Trident CSI driver found by label." TridentCSIDriver=csi.trident.netapp.io
time="2022-02-27T21:46:31Z" level=info msg="A Trident service was found by label." namespace=netapp service=trident-csi
time="2022-02-27T21:46:31Z" level=info msg="A Trident secret was found by label." namespace=netapp secret=trident-csi
time="2022-02-27T21:46:31Z" level=info msg="A Trident deployment was found by label" deployment=trident-csi namespace=netapp
time="2022-02-27T21:46:31Z" level=info msg="A Trident daemonSet was found by label." daemonSet=trident-csi namespace=netapp
time="2022-02-27T21:46:38Z" level=info msg="Waiting for Trident pod to start."
time="2022-02-27T21:46:38Z" level=info msg="Trident pod started." namespace=netapp pod=trident-csi-558f9448cd-9hb4s
time="2022-02-27T21:46:38Z" level=info msg="Waiting for Trident REST interface."
time="2022-02-27T21:46:39Z" level=info msg="Trident REST interface is up." version=22.01.0
time="2022-02-27T21:46:39Z" level=info msg="Trident is installed." namespace=netapp version=v22.01.0
time="2022-02-27T21:46:39Z" level=info msg="New status is same as the old status, no update needed." tridentOrchestratorCR=trident
time="2022-02-27T21:46:39Z" level=info msg="Synced Trident Orchestrator CR 'trident'"
time="2022-02-27T21:46:39Z" level=info msg=-------------------------------------------------
time="2022-02-27T21:46:39Z" level=info msg=-------------------------------------------------
Was wondering does the CSI helm chat use the ports 31579 and 32165 in any way and can these ports be changed to something else via the helm chart?
seems like the most consistent message is
time="2022-02-27T21:24:55Z" level=info msg="CRD not found." CRD=tridentprovisioners.trident.netapp.io
can only assume based on logs that this is bug
Hi @dc232
Can you confirm if 10.0.0.68:/pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d is mountable by your Kubernetes cluster?
Since this is with azure-netapp-files, you should check the export policy tab of the associated volume. You will also need to confirm that 10.0.0.68 is accessible by the node in question. This looks to be a n/w issue
Hi @balaramesh sorry for the late reply, i can now confirm that the network issue has been resolved via VNET peering however the exit 32 issue still remains and leaves the PVC in a pending state, via the nodes I am able to do the following
via azue bastion
ubuntu@aks-akscluster-38019429-vmss000000:~$ sudo ufw status
Status: inactive
ubuntu@aks-akscluster-38019429-vmss000000:~$ sudo apt-get install nfs-common
Reading package lists... Done
Building dependency tree
Reading state information... Done
nfs-common is already the newest version (1:1.3.4-2.1ubuntu5.5).
The following packages were automatically installed and are no longer required:
libapr1 libaprutil1
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
mount target IP: 192.1.0.4 --> this is the interface of the netapp appliance which is vnet peered
ubuntu@aks-akscluster-38019429-vmss000000:~$ ping 192.1.0.4
PING 192.1.0.4 (192.1.0.4) 56(84) bytes of data.
64 bytes from 192.1.0.4: icmp_seq=1 ttl=63 time=2.54 ms
64 bytes from 192.1.0.4: icmp_seq=2 ttl=63 time=1.45 ms
64 bytes from 192.1.0.4: icmp_seq=3 ttl=63 time=1.39 ms
64 bytes from 192.1.0.4: icmp_seq=4 ttl=63 time=1.47 ms
64 bytes from 192.1.0.4: icmp_seq=5 ttl=63 time=1.45 ms
64 bytes from 192.1.0.4: icmp_seq=6 ttl=63 time=1.33 ms
64 bytes from 192.1.0.4: icmp_seq=7 ttl=63 time=1.37 ms
64 bytes from 192.1.0.4: icmp_seq=8 ttl=63 time=1.53 ms
we can see the mount ip from the vnet peer, this proves a bug in the CSI or WordPress bitnmi helm char
ubuntu@aks-akscluster-38019429-vmss000000:~$ sudo mount -t nfs -o rw,hard,rsize=65536,wsize=65536,sec=sys,vers=4.1,tcp 192.1.0.4:/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789 pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789
ubuntu@aks-akscluster-38019429-vmss000000:~$ cd pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789/
-bash: cd: pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789/: Permission denied
ubuntu@aks-akscluster-38019429-vmss000000:~$ sudo cd pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789/
sudo: cd: command not found
above is interesting
ubuntu@aks-akscluster-38019429-vmss000000:~$ ls -l
total 4
drwxrwx--- 2 nobody 4294967294 4096 Mar 13 14:05 pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789
hmm, wonder why the above is creating folders owned by nobody?
ubuntu@aks-akscluster-38019429-vmss000000:~$ less /etc/passwd
ubuntu:x:1000:1000:Ubuntu:/home/ubuntu:/bin/bash
ubuntu@aks-akscluster-38019429-vmss000000:~$ sudo chown ubuntu:ubuntu pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789
ubuntu@aks-akscluster-38019429-vmss000000:~$ cd pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789/
ubuntu@aks-akscluster-38019429-vmss000000:~/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789$
Remounted as lost connection with azure bastion
ubuntu@aks-akscluster-38019429-vmss000000:~/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789$ sudo mount -t nfs -o rw,hard,rsize=65536,wsize=65536,sec=sys,vers=4.1,tcp 192.1.0.4:/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789 pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789
mount.nfs: mount point pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789 does not exist
ubuntu@aks-akscluster-38019429-vmss000000:~/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789$ cd ..
ubuntu@aks-akscluster-38019429-vmss000000:~$ sudo mount -t nfs -o rw,hard,rsize=65536,wsize=65536,sec=sys,vers=4.1,tcp 192.1.0.4:/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789 pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789
ubuntu@aks-akscluster-38019429-vmss000000:~$
ubuntu@aks-akscluster-38019429-vmss000000:~$
ubuntu@aks-akscluster-38019429-vmss000000:~$ cd pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789/
ubuntu@aks-akscluster-38019429-vmss000000:~/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789$ ls -l
total 1074300
-rw-r----- 1 ubuntu ubuntu 1095761920 Dec 21 2012 linuxmint-14-kde-dvd-64bit.iso
ubuntu@aks-akscluster-38019429-vmss000000:~/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789$ wget https://mirror.math.princeton.edu/pub/ubuntu-iso/20.04.4/ubuntu-20.04.4-desktop-amd64.iso
--2022-03-13 14:58:30-- https://mirror.math.princeton.edu/pub/ubuntu-iso/20.04.4/ubuntu-20.04.4-desktop-amd64.iso
Resolving mirror.math.princeton.edu (mirror.math.princeton.edu)... 128.112.18.21
Connecting to mirror.math.princeton.edu (mirror.math.princeton.edu)|128.112.18.21|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3379068928 (3.1G) [application/octet-stream]
Saving to: ‘ubuntu-20.04.4-desktop-amd64.iso’
ubuntu-20.04.4-desktop-amd64.iso 100%[===================================================================================================================================================================>] 3.15G 5.38MB/s in 6m 9s
2022-03-13 15:07:00 (8.73 MB/s) - ‘ubuntu-20.04.4-desktop-amd64.iso’ saved [3379068928/3379068928]
The above leads me to belive thee is either a bug with he CSI or the bitnami Wordpress helm chart, however i am not sure which one is the culprit
Curent Kubernetes output
kubectl get po -A --kubeconfig config
NAME READY STATUS RESTARTS AGE
phpmyadmin-69bc76d6c4-d94wb 2/2 Running 0 77m
wordpress-6bcfb47ff8-2q74h 0/2 ContainerCreating 0 77m
wordpress-memcached-79545df5d7-w9lj4 1/1 Running 0 77m
netapp trident-csi-558f9448cd-gt7j6 6/6 Running 0 93m
netapp trident-csi-fv8st 2/2 Running 0 77m
netapp trident-csi-hpcfq 2/2 Running 0 93m
netapp trident-operator-5b796496d5-gj97c 1/1 Running 0 93m
kubectl get pv,pvc --kubeconfig config
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM
STORAGECLASS REASON AGE
persistentvolume/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789 100Gi RWX Delete Bound default/wordpress
wordpress-storage-class-netapp 73m
persistentvolume/pvc-25f174fe-47ef-4c53-b023-b3771ccad845 10Gi RWO Delete Bound monitoring/grafana-pvc
default 79m
persistentvolume/pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd 100Gi RWO Delete Bound monitoring/prometheus-prometheus-kube-prometheus-promethe
us-db-prometheus-prometheus-kube-prometheus-prometheus-0 wordpress-storage-class-netapp 76m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/wordpress Bound pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789 100Gi RWX wordpress-storage-class-netapp 78m
kubectl describe po wordpress-6bcfb47ff8-2q74h --kubeconfig config
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 34m (x9 over 65m) kubelet Unable to attach or mount volumes: unmounted volumes=[wordpress-data], unattached volumes=[wordpress-data custom-po
stinit kube-api-access-tf8kh]: timed out waiting for the condition
Warning FailedMount 13m (x8 over 72m) kubelet Unable to attach or mount volumes: unmounted volumes=[wordpress-data], unattached volumes=[custom-postinit kube-api
-access-tf8kh wordpress-data]: timed out waiting for the condition
Warning FailedMount 3m8s (x43 over 74m) kubelet MountVolume.SetUp failed for volume "pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789" : rpc error: code = Internal desc =
error mounting NFS volume 192.1.0.4:/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789 on mountpoint /var/lib/kubelet/pods/d02f9bbf-8a69-49c1-ba61-9532f0a239c1/volumes/kubernetes
.io~csi/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789/mount: exit status 32
kubectl logs trident-csi-558f9448cd-gt7j6 -n netapp --kubeconfig config -c trident-main
time="2022-03-13T13:48:08Z" level=info msg="Running Trident storage orchestrator." binary=/trident_orchestrator build_time="Mon Jan 31 15:52:06 UTC 2022" version=22.01.0
time="2022-03-13T13:48:08Z" level=info msg="Created Kubernetes clients." namespace=netapp version=v1.23.3
time="2022-03-13T13:48:08Z" level=info msg="Initializing metrics frontend." address=":8001"
time="2022-03-13T13:48:08Z" level=info msg="Added frontend." name=metrics
time="2022-03-13T13:48:08Z" level=info msg="Initializing K8S helper frontend." requestID=5eec05f3-176b-45c2-b4ee-c166eb32ff11 requestSource=Internal
time="2022-03-13T13:48:08Z" level=info msg="K8S helper determined the container orchestrator version." gitVersion=v1.23.3 requestID=5eec05f3-176b-45c2-b4ee-c166eb32ff11
requestSource=Internal version=1.23
time="2022-03-13T13:48:08Z" level=info msg="Added frontend." name=k8s_csi_helper
time="2022-03-13T13:48:08Z" level=info msg="Initializing CSI frontend." name=aks-akscluster-38019429-vmss000000 version=22.01.0
time="2022-03-13T13:48:08Z" level=info msg="Enabling controller service capability." capability=CREATE_DELETE_VOLUME
time="2022-03-13T13:48:08Z" level=info msg="Enabling controller service capability." capability=PUBLISH_UNPUBLISH_VOLUME
time="2022-03-13T13:48:08Z" level=info msg="Enabling controller service capability." capability=LIST_VOLUMES
time="2022-03-13T13:48:08Z" level=info msg="Enabling controller service capability." capability=CREATE_DELETE_SNAPSHOT
time="2022-03-13T13:48:08Z" level=info msg="Enabling controller service capability." capability=LIST_SNAPSHOTS
time="2022-03-13T13:48:08Z" level=info msg="Enabling controller service capability." capability=EXPAND_VOLUME
time="2022-03-13T13:48:08Z" level=info msg="Enabling controller service capability." capability=CLONE_VOLUME
time="2022-03-13T13:48:08Z" level=info msg="Enabling controller service capability." capability=LIST_VOLUMES_PUBLISHED_NODES
time="2022-03-13T13:48:08Z" level=info msg="Enabling volume access mode." mode=SINGLE_NODE_WRITER
time="2022-03-13T13:48:08Z" level=info msg="Enabling volume access mode." mode=SINGLE_NODE_READER_ONLY
time="2022-03-13T13:48:08Z" level=info msg="Enabling volume access mode." mode=MULTI_NODE_READER_ONLY
time="2022-03-13T13:48:08Z" level=info msg="Enabling volume access mode." mode=MULTI_NODE_SINGLE_WRITER
time="2022-03-13T13:48:08Z" level=info msg="Enabling volume access mode." mode=MULTI_NODE_MULTI_WRITER
time="2022-03-13T13:48:08Z" level=info msg="Added frontend." name=csi
time="2022-03-13T13:48:08Z" level=info msg="Initializing Trident CRD controller frontend." namespace=netapp
time="2022-03-13T13:48:08Z" level=info msg="Creating event broadcaster."
time="2022-03-13T13:48:08Z" level=info msg="Setting up CRD controller event handlers."
time="2022-03-13T13:48:08Z" level=info msg="Added frontend." name=crd
time="2022-03-13T13:48:08Z" level=info msg="Initializing HTTP REST frontend." address="127.0.0.1:8000"
time="2022-03-13T13:48:08Z" level=info msg="Added frontend." name="HTTP REST"
time="2022-03-13T13:48:08Z" level=info msg="Initializing HTTPS REST frontend." address=":8443"
time="2022-03-13T13:48:08Z" level=info msg="Added frontend." name="HTTPS REST"
time="2022-03-13T13:48:08Z" level=info msg="Activating metrics frontend." address=":8001"
time="2022-03-13T13:48:08Z" level=info msg="Activating HTTP REST frontend." address="127.0.0.1:8000"
time="2022-03-13T13:48:08Z" level=info msg="Activating HTTPS REST frontend." address=":8443"
time="2022-03-13T13:48:08Z" level=warning msg="Persistent state version not found, creating." OrchestratorAPIVersion=1 PersistentStoreVersion=crdv1 requestID=1f8764fd-9c
f6-4605-961a-beefbe861c51 requestSource=Internal
time="2022-03-13T13:48:09Z" level=info msg="Added 0 existing volume(s)" requestID=1f8764fd-9cf6-4605-961a-beefbe861c51 requestSource=Internal
time="2022-03-13T13:48:09Z" level=info msg="Trident bootstrapped successfully."
time="2022-03-13T13:48:09Z" level=info msg="Activating K8S helper frontend." requestID=1bfa2756-f1bb-4226-b7a8-3451405dcae3 requestSource=Internal
time="2022-03-13T13:48:09Z" level=info msg="Activating CRD frontend."
time="2022-03-13T13:48:09Z" level=info msg="Activating CSI frontend." requestID=f0ff9850-39ce-41d1-971d-7bc6038e6f3a requestSource=Internal
time="2022-03-13T13:48:09Z" level=info msg="Listening for GRPC connections." name=/plugin/csi.sock net=unix
time="2022-03-13T13:48:09Z" level=info msg="Starting Trident CRD controller."
time="2022-03-13T13:48:09Z" level=info msg="Waiting for informer caches to sync."
time="2022-03-13T13:48:09Z" level=info msg="Starting periodic node access reconciliation service." requestID=e4523045-6479-49a9-8e06-2d82b726d607 requestSource=Periodic
I0313 13:48:10.643763 1 request.go:665] Waited for 1.153890579s due to client-side throttling, not priority and fairness, request: GET:https://10.0.0.1:443/apis/tr
ident.netapp.io/v1/namespaces/netapp/tridentversions?limit=500&resourceVersion=0
time="2022-03-13T13:48:11Z" level=info msg="Starting workers."
time="2022-03-13T13:48:11Z" level=info msg="Started workers."
time="2022-03-13T13:48:39Z" level=info msg="Determined topology labels for node: map[topology.kubernetes.io/region:uksouth topology.kubernetes.io/zone:uksouth-1]" node=a
ks-akscluster-38019429-vmss000000 requestID=619cafed-5bd0-46fb-8fd3-988e5ff94ca2 requestSource=REST
time="2022-03-13T13:48:39Z" level=info msg="Added a new node." handler=AddOrUpdateNode node=aks-akscluster-38019429-vmss000000 requestID=619cafed-5bd0-46fb-8fd3-988e5ff9
4ca2 requestSource=REST
time="2022-03-13T13:51:01Z" level=warning msg="The debug setting in the configuration file is now ignored; use the command line --debug switch instead." crdControllerEve
nt=add requestID=e18d2b61-333f-4698-9e76-700660a86bd5 requestSource=CRD
time="2022-03-13T13:51:02Z" level=info msg="Discovered Azure resources." capacityPools=1 crdControllerEvent=add requestID=e18d2b61-333f-4698-9e76-700660a86bd5 requestSou
rce=CRD resourceGroups=1 subnets=1
time="2022-03-13T13:51:02Z" level=info msg="Storage driver initialized." crdControllerEvent=add driver=azure-netapp-files requestID=e18d2b61-333f-4698-9e76-700660a86bd5
requestSource=CRD
time="2022-03-13T13:51:02Z" level=info msg="Created new storage backend." backend="&{0xc0004e23c0 netapp-netapppool true online map[netapp-netapppool_pool:0xc0003b2420]
map[] false}" crdControllerEvent=add requestID=e18d2b61-333f-4698-9e76-700660a86bd5 requestSource=CRD
time="2022-03-13T13:51:02Z" level=info msg="Newly added backend satisfies no storage classes." backend=netapp-netapppool crdControllerEvent=add requestID=e18d2b61-333f-4
698-9e76-700660a86bd5 requestSource=CRD
time="2022-03-13T13:51:02Z" level=info msg="Synced 'netapp/nett-app-backend'" crdControllerEvent=add logSource=trident-crd-controller requestID=e18d2b61-333f-4698-9e76-7
00660a86bd5 requestSource=CRD
time="2022-03-13T13:51:02Z" level=info msg=-------------------------------------------------
time="2022-03-13T13:51:02Z" level=info msg=-------------------------------------------------
time="2022-03-13T13:57:56Z" level=info msg="Storage class satisfied by 1 storage pools." requestID=b5764e05-f67b-4fed-abf0-b00d09acb7cd requestSource=Kubernetes storageC
lass=wordpress-storage-class-netapp
time="2022-03-13T13:57:56Z" level=info msg="K8S helper added a storage class." name=wordpress-storage-class-netapp parameters="map[backendType:azure-netapp-files fsType:
nfs]" provisioner=csi.trident.netapp.io requestID=b5764e05-f67b-4fed-abf0-b00d09acb7cd requestSource=Kubernetes
time="2022-03-13T13:58:22Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=f9cf3f03-383d-45d7-9a1c-8c63b14c08f4 req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T13:58:22Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=f9cf3f03-383d-45d7-9a1c-8c63b14c08f4 requestSource=CSI
time="2022-03-13T13:58:24Z" level=info msg="Volume created." API=VolumesClient.CreateOrUpdate correlationID=0b666528-c616-4b2c-b3b4-e1250c647475 creationToken=pvc-b399ea
e4-6210-45b9-95f0-a2fbbb2223dd requestID=f9cf3f03-383d-45d7-9a1c-8c63b14c08f4 requestSource=CSI volume=netapp_rg/netapp-netappaccount/netapp-netapppool/pvc-b399eae4-6210
-45b9-95f0-a2fbbb2223dd
time="2022-03-13T13:58:24Z" level=info msg="Waiting for volume state." desiredState=Succeeded requestID=f9cf3f03-383d-45d7-9a1c-8c63b14c08f4 requestSource=CSI
time="2022-03-13T13:58:32Z" level=error msg="Volume state was not Succeeded after 10.00 seconds." requestID=f9cf3f03-383d-45d7-9a1c-8c63b14c08f4 requestSource=CSI
time="2022-03-13T13:58:32Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is Creating, not Succeeded" pool=netapp-netapppool_pool requestID=f9cf3f03-383d-45d7-9a1c-8c63b14c08f4 requestSource=CSI volume=pvc-b399eae4-6210-45b9-95f0-
a2fbbb2223dd
time="2022-03-13T13:58:32Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is Creating, not Succeeded" requestID=f9cf3f03-383d-45d7-9a1c-8c63
b14c08f4 requestSource=CSI
time="2022-03-13T13:58:40Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=33aca4d8-5585-47b0-aa6a-b20ef073c061 req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T13:58:40Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=33aca4d8-5585-47b0-aa6a-b20ef073c061 requestSource=CSI
time="2022-03-13T13:58:40Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=33aca4d8-5585-47b0-aa6a-b20ef073c061 requestSource=CSI volume=pvc-b399eae4-6210-45b9
-95f0-a2fbbb2223dd
time="2022-03-13T13:58:40Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=33aca4d8-5585-47b0-aa6
a-b20ef073c061 requestSource=CSI
time="2022-03-13T13:58:56Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=07cd36e7-f7a9-4893-a060-43a7663604fc req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T13:58:56Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=07cd36e7-f7a9-4893-a060-43a7663604fc requestSource=CSI
time="2022-03-13T13:58:56Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=07cd36e7-f7a9-4893-a060-43a7663604fc requestSource=CSI volume=pvc-b399eae4-6210-45b9
-95f0-a2fbbb2223dd
time="2022-03-13T13:58:56Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=07cd36e7-f7a9-4893-a06
0-43a7663604fc requestSource=CSI
time="2022-03-13T13:59:26Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=af88008d-570e-4fe1-bc06-5647849fbb16 req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T13:59:26Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=af88008d-570e-4fe1-bc06-5647849fbb16 requestSource=CSI
time="2022-03-13T13:59:26Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=af88008d-570e-4fe1-bc06-5647849fbb16 requestSource=CSI volume=pvc-b399eae4-6210-45b9
-95f0-a2fbbb2223dd
time="2022-03-13T13:59:26Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=af88008d-570e-4fe1-bc0
6-5647849fbb16 requestSource=CSI
time="2022-03-13T13:59:56Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=e00c1768-17f0-46b0-8ea7-0ee3eea40bfe req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T13:59:56Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=e00c1768-17f0-46b0-8ea7-0ee3eea40bfe requestSource=CSI
time="2022-03-13T13:59:57Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=e00c1768-17f0-46b0-8ea7-0ee3eea40bfe requestSource=CSI volume=pvc-b399eae4-6210-45b9
-95f0-a2fbbb2223dd
time="2022-03-13T13:59:57Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=e00c1768-17f0-46b0-8ea
7-0ee3eea40bfe requestSource=CSI
time="2022-03-13T14:00:27Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=b989e40c-fd15-4c86-9d9e-e5f321cccf7b req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T14:00:27Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=b989e40c-fd15-4c86-9d9e-e5f321cccf7b requestSource=CSI
time="2022-03-13T14:00:27Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=b989e40c-fd15-4c86-9d9e-e5f321cccf7b requestSource=CSI volume=pvc-b399eae4-6210-45b9
-95f0-a2fbbb2223dd
time="2022-03-13T14:00:27Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=b989e40c-fd15-4c86-9d9
e-e5f321cccf7b requestSource=CSI
time="2022-03-13T14:00:57Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=73b66a4d-5cf7-4a86-82d2-35d8b18fa6d5 req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T14:00:57Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=73b66a4d-5cf7-4a86-82d2-35d8b18fa6d5 requestSource=CSI
time="2022-03-13T14:00:57Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=73b66a4d-5cf7-4a86-82d2-35d8b18fa6d5 requestSource=CSI volume=pvc-b399eae4-6210-45b9
-95f0-a2fbbb2223dd
time="2022-03-13T14:00:57Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=73b66a4d-5cf7-4a86-82d
2-35d8b18fa6d5 requestSource=CSI
time="2022-03-13T14:01:27Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=38f60596-12b3-4b93-a066-2ec9351267d5 req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T14:01:27Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=38f60596-12b3-4b93-a066-2ec9351267d5 requestSource=CSI
time="2022-03-13T14:01:27Z" level=info msg="Discovered Azure resources." capacityPools=1 requestID=38f60596-12b3-4b93-a066-2ec9351267d5 requestSource=CSI resourceGroups=
1 subnets=1
time="2022-03-13T14:01:27Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=38f60596-12b3-4b93-a066-2ec9351267d5 requestSource=CSI volume=pvc-b399eae4-6210-45b9
-95f0-a2fbbb2223dd
time="2022-03-13T14:01:27Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=38f60596-12b3-4b93-a06
6-2ec9351267d5 requestSource=CSI
time="2022-03-13T14:01:57Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=276b1543-332c-4be9-8eef-ca70649bd02b req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T14:01:57Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=276b1543-332c-4be9-8eef-ca70649bd02b requestSource=CSI
time="2022-03-13T14:01:57Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=276b1543-332c-4be9-8eef-ca70649bd02b requestSource=CSI volume=pvc-b399eae4-6210-45b9
-95f0-a2fbbb2223dd
time="2022-03-13T14:01:57Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=276b1543-332c-4be9-8ee
f-ca70649bd02b requestSource=CSI
time="2022-03-13T14:02:27Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=7f8234b3-0c46-4804-89ab-edd7d31e110e req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T14:02:27Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=7f8234b3-0c46-4804-89ab-edd7d31e110e requestSource=CSI
time="2022-03-13T14:02:28Z" level=warning msg="Volume already exists." name=pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd requestID=7f8234b3-0c46-4804-89ab-edd7d31e110e reque
stSource=CSI state=Succeeded
time="2022-03-13T14:02:28Z" level=warning msg="Volume already exists." backend=netapp-netapppool requestID=7f8234b3-0c46-4804-89ab-edd7d31e110e requestSource=CSI volume=
pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd
time="2022-03-13T14:05:13Z" level=info msg="Determined topology labels for node: map[topology.kubernetes.io/region:uksouth topology.kubernetes.io/zone:uksouth-1]" node=a
ks-akscluster-38019429-vmss000001 requestID=fc4ee0c3-e632-4074-8b2f-d9f469446e8e requestSource=REST
time="2022-03-13T14:05:13Z" level=info msg="Added a new node." handler=AddOrUpdateNode node=aks-akscluster-38019429-vmss000001 requestID=fc4ee0c3-e632-4074-8b2f-d9f46944
6e8e requestSource=REST
time="2022-03-13T14:05:18Z" level=info msg="Found PVC for requested volume pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789." UID=0099b9fa-704b-4110-b1c7-cf8dfe2c4789 name=wordp
ress namespace=default requestID=ef959676-4251-4808-b5e1-b84bb1a7d2b8 requestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-c
lass-netapp
time="2022-03-13T14:05:18Z" level=info msg="Found storage class for requested volume pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789." name=wordpress-storage-class-netapp reque
stID=ef959676-4251-4808-b5e1-b84bb1a7d2b8 requestSource=CSI
time="2022-03-13T14:05:19Z" level=info msg="Volume created." API=VolumesClient.CreateOrUpdate correlationID=91f801e7-9736-460b-aaf0-ac38d84528cf creationToken=pvc-0099b9
fa-704b-4110-b1c7-cf8dfe2c4789 requestID=ef959676-4251-4808-b5e1-b84bb1a7d2b8 requestSource=CSI volume=netapp_rg/netapp-netappaccount/netapp-netapppool/pvc-0099b9fa-704b
-4110-b1c7-cf8dfe2c4789
time="2022-03-13T14:05:19Z" level=info msg="Waiting for volume state." desiredState=Succeeded requestID=ef959676-4251-4808-b5e1-b84bb1a7d2b8 requestSource=CSI
time="2022-03-13T14:05:27Z" level=error msg="Volume state was not Succeeded after 10.00 seconds." requestID=ef959676-4251-4808-b5e1-b84bb1a7d2b8 requestSource=CSI
time="2022-03-13T14:05:27Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is Creating, not Succeeded" pool=netapp-netapppool_pool requestID=ef959676-4251-4808-b5e1-b84bb1a7d2b8 requestSource=CSI volume=pvc-0099b9fa-704b-4110-b1c7-
cf8dfe2c4789
time="2022-03-13T14:05:27Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is Creating, not Succeeded" requestID=ef959676-4251-4808-b5e1-b84b
b1a7d2b8 requestSource=CSI
time="2022-03-13T14:05:35Z" level=info msg="Found PVC for requested volume pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789." UID=0099b9fa-704b-4110-b1c7-cf8dfe2c4789 name=wordp
ress namespace=default requestID=b27e72c7-0cc5-4222-ac2d-8654d0dc5bf7 requestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-c
lass-netapp
time="2022-03-13T14:05:35Z" level=info msg="Found storage class for requested volume pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789." name=wordpress-storage-class-netapp reque
stID=b27e72c7-0cc5-4222-ac2d-8654d0dc5bf7 requestSource=CSI
time="2022-03-13T14:05:35Z" level=warning msg="Volume already exists." name=pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789 requestID=b27e72c7-0cc5-4222-ac2d-8654d0dc5bf7 reque
stSource=CSI state=Succeeded
time="2022-03-13T14:05:35Z" level=warning msg="Volume already exists." backend=netapp-netapppool requestID=b27e72c7-0cc5-4222-ac2d-8654d0dc5bf7 requestSource=CSI volume=
pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789
kubectl logs trident-csi-558f9448cd-gt7j6 -n netapp --kubeconfig config -c csi-provisioner
W0313 13:48:16.466379 1 feature_gate.go:237] Setting GA feature gate Topology=true. It will be removed in a future release.
I0313 13:48:16.466415 1 feature_gate.go:245] feature gates: &{map[Topology:true]}
I0313 13:48:16.466425 1 csi-provisioner.go:139] Version: v3.1.0
I0313 13:48:16.466430 1 csi-provisioner.go:162] Building kube configs for running in cluster...
I0313 13:48:16.467747 1 common.go:111] Probing CSI driver for readiness
I0313 13:48:16.470794 1 csi-provisioner.go:206] Detected CSI driver csi.trident.netapp.io
I0313 13:48:16.471942 1 csi-provisioner.go:275] CSI driver supports PUBLISH_UNPUBLISH_VOLUME, watching VolumeAttachments
I0313 13:48:16.472273 1 controller.go:732] Using saving PVs to API server in background
I0313 13:48:16.572825 1 controller.go:811] Starting provisioner controller csi.trident.netapp.io_trident-csi-558f9448cd-gt7j6_b2fb0794-3d1f-4675-8e01-899c5817de0b!
I0313 13:48:16.572930 1 clone_controller.go:66] Starting CloningProtection controller
I0313 13:48:16.573016 1 clone_controller.go:82] Started CloningProtection controller
I0313 13:48:16.573155 1 volume_store.go:97] Starting save volume queue
I0313 13:48:16.674025 1 controller.go:860] Started provisioner controller csi.trident.netapp.io_trident-csi-558f9448cd-gt7j6_b2fb0794-3d1f-4675-8e01-899c5817de0b!
I0313 13:58:22.656495 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 13:58:22.656577 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 13:58:22.657145 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 13:58:32.453817 1 controller.go:1075] Final error received, removing PVC b399eae4-6210-45b9-95f0-a2fbbb2223dd from claims in progress
W0313 13:58:32.453840 1 controller.go:934] Retrying syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd", failure 0
E0313 13:58:32.453868 1 controller.go:957] error syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is Creating, not Succeeded
I0313 13:58:32.453886 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is
Creating, not Succeeded
I0313 13:58:40.454126 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 13:58:40.454203 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 13:58:40.454725 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 13:58:40.577047 1 controller.go:1075] Final error received, removing PVC b399eae4-6210-45b9-95f0-a2fbbb2223dd from claims in progress
W0313 13:58:40.577289 1 controller.go:934] Retrying syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd", failure 1
E0313 13:58:40.577316 1 controller.go:957] error syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is still Creating, not Succeeded
I0313 13:58:40.577582 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is
still Creating, not Succeeded
I0313 13:58:56.577576 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 13:58:56.577644 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 13:58:56.577767 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 13:58:56.669190 1 controller.go:1075] Final error received, removing PVC b399eae4-6210-45b9-95f0-a2fbbb2223dd from claims in progress
W0313 13:58:56.669221 1 controller.go:934] Retrying syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd", failure 2
E0313 13:58:56.669315 1 controller.go:957] error syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is still Creating, not Succeeded
I0313 13:58:56.669387 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is
still Creating, not Succeeded
I0313 13:59:26.669806 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 13:59:26.669896 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 13:59:26.670406 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 13:59:26.751062 1 controller.go:1075] Final error received, removing PVC b399eae4-6210-45b9-95f0-a2fbbb2223dd from claims in progress
W0313 13:59:26.751083 1 controller.go:934] Retrying syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd", failure 3
E0313 13:59:26.751107 1 controller.go:957] error syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is still Creating, not Succeeded
I0313 13:59:26.751389 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is
still Creating, not Succeeded
I0313 13:59:56.752135 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 13:59:56.752205 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 13:59:56.752607 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 13:59:57.260992 1 controller.go:1075] Final error received, removing PVC b399eae4-6210-45b9-95f0-a2fbbb2223dd from claims in progress
W0313 13:59:57.261012 1 controller.go:934] Retrying syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd", failure 4
E0313 13:59:57.261031 1 controller.go:957] error syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is still Creating, not Succeeded
I0313 13:59:57.261368 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is
still Creating, not Succeeded
I0313 14:00:27.261485 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 14:00:27.261560 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 14:00:27.262179 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 14:00:27.390891 1 controller.go:1075] Final error received, removing PVC b399eae4-6210-45b9-95f0-a2fbbb2223dd from claims in progress
W0313 14:00:27.390926 1 controller.go:934] Retrying syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd", failure 5
E0313 14:00:27.390947 1 controller.go:957] error syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is still Creating, not Succeeded
I0313 14:00:27.390971 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is
still Creating, not Succeeded
I0313 14:00:57.391081 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 14:00:57.391169 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 14:00:57.391372 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 14:00:57.480296 1 controller.go:1075] Final error received, removing PVC b399eae4-6210-45b9-95f0-a2fbbb2223dd from claims in progress
W0313 14:00:57.480311 1 controller.go:934] Retrying syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd", failure 6
E0313 14:00:57.480326 1 controller.go:957] error syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is still Creating, not Succeeded
I0313 14:00:57.480600 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is
still Creating, not Succeeded
I0313 14:01:27.480879 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 14:01:27.480953 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 14:01:27.481002 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 14:01:27.796824 1 controller.go:1075] Final error received, removing PVC b399eae4-6210-45b9-95f0-a2fbbb2223dd from claims in progress
W0313 14:01:27.796847 1 controller.go:934] Retrying syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd", failure 7
E0313 14:01:27.796867 1 controller.go:957] error syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is still Creating, not Succeeded
I0313 14:01:27.797064 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is
still Creating, not Succeeded
I0313 14:01:57.797001 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 14:01:57.797071 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 14:01:57.797570 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 14:01:57.937150 1 controller.go:1075] Final error received, removing PVC b399eae4-6210-45b9-95f0-a2fbbb2223dd from claims in progress
W0313 14:01:57.937586 1 controller.go:934] Retrying syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd", failure 8
E0313 14:01:57.937613 1 controller.go:957] error syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is still Creating, not Succeeded
I0313 14:01:57.937936 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is
still Creating, not Succeeded
I0313 14:02:27.938371 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 14:02:27.938447 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 14:02:27.938664 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 14:02:28.185989 1 controller.go:858] successfully created PV pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd for PVC prometheus-prometheus-kube-prometheus-prometheu
s-db-prometheus-prometheus-kube-prometheus-prometheus-0 and csi volume name pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd
I0313 14:02:28.186028 1 controller.go:1442] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": volume "pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd" provisioned
I0313 14:02:28.186041 1 controller.go:1455] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": succeeded
I0313 14:02:28.193407 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd
I0313 14:05:18.404717 1 controller.go:1337] provision "default/wordpress" class "wordpress-storage-class-netapp": started
W0313 14:05:18.405162 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 14:05:18.405523 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"wordpress", UID:"0099b9fa-704b-4110-b1c7-cf
8dfe2c4789", APIVersion:"v1", ResourceVersion:"10033", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "defau
lt/wordpress"
I0313 14:05:27.442119 1 controller.go:1075] Final error received, removing PVC 0099b9fa-704b-4110-b1c7-cf8dfe2c4789 from claims in progress
W0313 14:05:27.442147 1 controller.go:934] Retrying syncing claim "0099b9fa-704b-4110-b1c7-cf8dfe2c4789", failure 0
E0313 14:05:27.442166 1 controller.go:957] error syncing claim "0099b9fa-704b-4110-b1c7-cf8dfe2c4789": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is Creating, not Succeeded
I0313 14:05:27.442423 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"wordpress", UID:"0099b9fa-704b-4110-b1c7-cf
8dfe2c4789", APIVersion:"v1", ResourceVersion:"10033", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpre
ss-storage-class-netapp": rpc error: code = Unknown desc = volume state is Creating, not Succeeded
I0313 14:05:35.442355 1 controller.go:1337] provision "default/wordpress" class "wordpress-storage-class-netapp": started
W0313 14:05:35.442457 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 14:05:35.442712 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"wordpress", UID:"0099b9fa-704b-4110-b1c7-cf
8dfe2c4789", APIVersion:"v1", ResourceVersion:"10033", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "defau
lt/wordpress"
I0313 14:05:35.638854 1 controller.go:858] successfully created PV pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789 for PVC wordpress and csi volume name pvc-0099b9fa-704b
-4110-b1c7-cf8dfe2c4789
I0313 14:05:35.639292 1 controller.go:1442] provision "default/wordpress" class "wordpress-storage-class-netapp": volume "pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789"
provisioned
I0313 14:05:35.639472 1 controller.go:1455] provision "default/wordpress" class "wordpress-storage-class-netapp": succeeded
I0313 14:05:35.645968 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"wordpress", UID:"0099b9fa-704b-4110-b1c7-cf
8dfe2c4789", APIVersion:"v1", ResourceVersion:"10033", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-0099b9fa-704b-4
110-b1c7-cf8dfe2c4789
PS C:\Users\danny\Desktop> kubectl logs trident-csi-558f9448cd-gt7j6 -n netapp --kubeconfig config -c csi-attacher
I0313 13:48:19.167837 1 main.go:92] Version: v3.4.0
I0313 13:48:19.169447 1 common.go:111] Probing CSI driver for readiness
I0313 13:48:19.172137 1 main.go:148] CSI driver name: "csi.trident.netapp.io"
I0313 13:48:19.172899 1 main.go:223] CSI driver supports ControllerPublishUnpublish, using real CSI handler
I0313 13:48:19.172920 1 main.go:231] CSI driver supports list volumes published nodes. Using capability to reconcile volume attachment objects with actual backend
state
I0313 13:48:19.173005 1 controller.go:128] Starting CSI attacher
I0313 14:02:28.690526 1 csi_handler.go:248] Attaching "csi-e885b2bab91f5ca1dd74ec1802fa49e42d68d428fe971bf19784dae3dd9fff48"
I0313 14:02:28.989566 1 csi_handler.go:261] Attached "csi-e885b2bab91f5ca1dd74ec1802fa49e42d68d428fe971bf19784dae3dd9fff48"
I0313 14:05:36.502663 1 csi_handler.go:248] Attaching "csi-5c376092415886c47032b0ba380374146edfe61d3e5beefd99cfc6218531c07d"
I0313 14:05:36.703642 1 csi_handler.go:261] Attached "csi-5c376092415886c47032b0ba380374146edfe61d3e5beefd99cfc6218531c07d"
@balaramesh where can I find information about the export policy for azure? this could aid in troubleshooting the error, from the logs it looks like the volume is trying to be created in Netapp but failing for some reason to be created within a 10-second time frame, is there a way this time frame can be increased?
I'm using a contributor role or the secret
also is there any documentation that aid in muting the warning
fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
however stillspeificy NFS4.1 to be used as the protocol?
@dc232 if that's the output of the export policy then it looks ok to me. I would recommend opening a ticket with NetApp Support if this issue still persists. You could look at using mountOptions in your storageClass and set vers=4.1 to set the protocol type.
@balaramesh can confirm that is definitely the export policy
I did notice something very odd about about the created volumes but not sure why this is, which seems to be no valid mount target, i'm not sure what's causing this, as NSG rule for peered networks seem to allow for NFS on 111 and 2049 via AllowVnetOutBound and AllowVnetInBound
mount options seem to be set via on the backend config yaml
nfsMountOptions: nfsvers=4
would this be the correct for the mountoptions for the storageclass
resource "kubernetes_storage_class_v1" "example" {
depends_on = [
helm_release.netapphelm,
kubernetes_secret_v1.netapp,
time_sleep.wait_600_seconds
]
metadata {
name = var.storageclass
}
storage_provisioner = "csi.trident.netapp.io"
reclaim_policy = "Delete"
volume_binding_mode = "WaitForFirstConsumer"
allow_volume_expansion = var.allow_volume_exspansion
parameters = {
backendType = "azure-netapp-files"
fsType: "nfs"
}
mount_options = ["vers=4.1"]
also how can I open a netapp support ticket, it doesn't seem to let me open one on the site?
Hey @balaramesh ignore the above message I have managed to fix that one
I found in the logs for wordpress bitnami helm chart
Warning FailedMount 75s (x9 over 3m24s) kubelet MountVolume.SetUp failed for volume "pvc-f5b15aae-b758-4126-9012-c7f2642d6d36" : rpc error: code = Internal desc = error mounting NFS volume 192.1.0.4:/pvc-f5b15aae-b758-4126-9012-c7f2642d6d36 on mountpoint /var/lib/kubelet/pods/9627f587-b082-4b4b-87aa-ea81151d73f8/
volumes/kubernetes.io~csi/pvc-f5b15aae-b758-4126-9012-c7f2642d6d36/mount: exit status 32
was wondering what things I could to try and debug I have VNET peering enabled have 111 and 2049 NFS enabled in the NSG I don't see any traffic flowing through and i don't have a firewall in the resource group is there anything else that I can check that springs to mind?
via prometheus bitnami helm chart
vents:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 14m default-scheduler Successfully assigned monitoring/prometheus-prometheus-kube-prometheus-prometheus-0 to aks-akscluster-24936088-vmss000000
Normal SuccessfulAttachVolume 14m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-bd2ecd02-c760-4285-9ee3-b2c5b8c17219"
Warning FailedMount 10m kubelet Unable to attach or mount volumes: unmounted volumes=[prometheus-prometheus-kube-prometheus-prometheus-db], unattached volumes=[prometheus-prometheus-kube-prometheus-prometheus-db web-config config config-out prometheus-prometheus-kube-prometheus-prometheus-rulefiles
-0 kube-api-access-np5kb tls-assets]: timed out waiting for the condition
Warning FailedMount 7m59s kubelet Unable to attach or mount volumes: unmounted volumes=[prometheus-prometheus-kube-prometheus-prometheus-db], unattached volumes=[config-out prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 kube-api-access-np5kb tls-assets prometheus-prometheus-kube-prometh
eus-prometheus-db web-config config]: timed out waiting for the condition
Warning FailedMount 72s (x4 over 12m) kubelet Unable to attach or mount volumes: unmounted volumes=[prometheus-prometheus-kube-prometheus-prometheus-db], unattached volumes=[config config-out prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 kube-api-access-np5kb tls-assets prometheus-prometheus-kube-
prometheus-prometheus-db web-config]: timed out waiting for the condition
Warning FailedMount 12s (x15 over 14m) kubelet MountVolume.SetUp failed for volume "pvc-bd2ecd02-c760-4285-9ee3-b2c5b8c17219" : rpc error: code = Internal desc = error mounting NFS volume 192.1.0.4:/pvc-bd2ecd02-c760-4285-9ee3-b2c5b8c17219 on mountpoint /var/lib/kubelet/pods/d297951f-6f16-4e1b-8855-83293870846a/v
olumes/kubernetes.io~csi/pvc-bd2ecd02-c760-4285-9ee3-b2c5b8c17219/mount: exit status 32
when logging into the node I found that the CSI does not seem to be creating the needed folders to mount the pod via the kubelet events
Apr 02 10:38:50 aks-akscluster-16723567-vmss000000 kubelet[3301]: E0402 10:38:50.003698 3301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/csi.trident.netapp.io^pvc-4d2b85eb-3de7-4524-825a-8b216e045418 podName: nodeName:}" failed. No
retries permitted until 2022-04-02 10:40:52.003671542 +0000 UTC m=+2038.016591882 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "pvc-4d2b85eb-3de7-4524-825a-8b216e045418" (UniqueName: "kubernetes.io/csi/csi.trident.netapp.io^pvc-4d2b85eb-3de7-
4524-825a-8b216e045418") pod "prometheus-prometheus-kube-prometheus-prometheus-0" (UID: "929e1d86-7415-43ab-a814-36f7d3577d25") : rpc error: code = Internal desc = error mounting NFS volume 192.1.0.4:/pvc-4d2b85eb-3de7-4524-825a-8b216e045418 on mountpoint /var/lib/kub
elet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~csi/pvc-4d2b85eb-3de7-4524-825a-8b216e045418/mount: exit status 32
root@aks-akscluster-16723567-vmss000000:~# ls -l /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~csi/
total 0
tried
root@aks-akscluster-16723567-vmss000000:~# mkdir -p /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~csi/pvc-4d2b85eb-3de7-4524-825a-8b216e045418/mount
root@aks-akscluster-16723567-vmss000000:~# ls -l /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~csi/pvc-4d2b85eb-3de7-4524-825a-8b216e045418
total 4
drwxr-x--- 2 root root 4096 Apr 2 10:55 mount
root@aks-akscluster-16723567-vmss000000:~# chmod 0755 /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~csi/pvc-4d2b85eb-3de7-4524-825a-8b216e045418/mount
after a while it is just cleared?
root@aks-akscluster-16723567-vmss000000:/var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~csi# ls -l pvc-4d2b85eb-3de7-4524-825a-8b216e045418/mount
total 0
root@aks-akscluster-16723567-vmss000000:/var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~csi# ls -l pvc-4d2b85eb-3de7-4524-825a-8b216e045418/mount
ls: cannot access 'pvc-4d2b85eb-3de7-4524-825a-8b216e045418/mount': No such file or directory
whilst I understand this may be a network issue of some sort i still find it baffling that the required folders are not being created by the CSI
@balaramesh looking into the mount on the node the mount seems to read tempfs
root@aks-akscluster-16723567-vmss000000:~# mount | grep -i 929e1d86-7415-43ab-a814-36f7d3577d25
tmpfs on /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~secret/config type tmpfs (rw,relatime,size=5493280k)
tmpfs on /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~secret/config type tmpfs (rw,relatime,size=5493280k)
tmpfs on /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~projected/tls-assets type tmpfs (rw,relatime,size=5493280k)
tmpfs on /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~projected/tls-assets type tmpfs (rw,relatime,size=5493280k)
tmpfs on /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~secret/web-config type tmpfs (rw,relatime,size=5493280k)
tmpfs on /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~secret/web-config type tmpfs (rw,relatime,size=5493280k)
tmpfs on /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~projected/kube-api-access-xshcg type tmpfs (rw,relatime,size=5493280k)
tmpfs on /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~projected/kube-api-access-xshcg type tmpfs (rw,relatime,size=5493280k)
is there meant to be a corresponding nfs mount to each pod or is it just a single mount? I don't see the netapp interface ip in the mount for the volume, i.e if I run mount against the IP which in this case is 192.1.0.4 i would expect to see something like this via the mount command on the node
192.1.0.4:/pvc-4d2b85eb-3de7-4524-825a-8b216e045418 on /root/pvc-4d2b85eb-3de7-4524-825a-8b216e045418 type nfs4 (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.1.0.4,local_lock=none,addr=192.1.0.4)
if the above is typed in manually and the directory created i am able to download a file to the folder as illustrated below
root@aks-akscluster-16723567-vmss000000:~/pvc-4d2b85eb-3de7-4524-825a-8b216e045418# wget https://speed.hetzner.de/10GB.bin
--2022-04-02 11:16:55-- https://speed.hetzner.de/10GB.bin
Resolving speed.hetzner.de (speed.hetzner.de)... 88.198.248.254, 2a01:4f8:0:59ed::2
Connecting to speed.hetzner.de (speed.hetzner.de)|88.198.248.254|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10485760000 (9.8G) [application/octet-stream]
Saving to: ‘10GB.bin’
10GB.bin 14%[=====================> ] 1.38G 1.20MB/s eta 2m 24s
and I can then see the azure netapp file volume being utilised This leads me to believe here is a bug in the way the CSI is provisioning the volumes for consumption or the CRD which handles this is simply missing,
can I confirm that the helm chart at this repo location bundles the CRD's in with the install, I'm assuming they are installed automatically
]
name = "netapp-trident-operator"
namespace = kubernetes_namespace.netapp.metadata[0].name
repository = "https://netapp.github.io/trident-helm-chart"
chart = "trident-operator"
}
also how des the CSI recognise the netapp interface IP and what does it need in order to successfully bind with it? and does it matter which namespace the netapp backend config is deployed into?
interestingly I have tried via this kubectl manifest and can form that this made the file called test
apiVersion: v1
kind: Pod
metadata:
name: test
labels:
app.kubernetes.io/name: alpine
app.kubernetes.io/part-of: kubernetes-complete-reference
app.kubernetes.io/created-by: ssbostan
spec:
containers:
- name: alpine
image: alpine:latest
command:
- touch
- /data/test
volumeMounts:
- name: nfs-volume
mountPath: /data
volumes:
- name: nfs-volume
nfs:
server: 192.1.0.4
path: /pvc-4d2b85eb-3de7-4524-825a-8b216e045418
readOnly: no
root@aks-akscluster-16723567-vmss000000:~/pvc-4d2b85eb-3de7-4524-825a-8b216e045418# ls -l
total 0
root@aks-akscluster-16723567-vmss000000:~/pvc-4d2b85eb-3de7-4524-825a-8b216e045418# ls -l
total 0
-rw-r--r-- 1 nobody 4294967294 0 Apr 2 13:33 test
root@aks-akscluster-16723567-vmss000000:~/pvc-4d2b85eb-3de7-4524-825a-8b216e045418#
I then revised the configuration to
apiVersion: v1
kind: Pod
metadata:
name: test
labels:
app.kubernetes.io/name: alpine
app.kubernetes.io/part-of: kubernetes-complete-reference
app.kubernetes.io/created-by: ssbostan
spec:
containers:
- name: alpine
image: alpine:latest
command:
- cat
- /data/test
volumeMounts:
- name: nfs-volume
mountPath: /data
volumes:
- name: nfs-volume
nfs:
server: 192.1.0.4
path: /pvc-4d2b85eb-3de7-4524-825a-8b216e045418
readOnly: no
whilst modifying the test file to contain
root@aks-akscluster-16723567-vmss000000:~/pvc-4d2b85eb-3de7-4524-825a-8b216e045418# echo "this is a test to see if the pod is able to read from the netapp mountpoint" > test
root@aks-akscluster-16723567-vmss000000:~/pvc-4d2b85eb-3de7-4524-825a-8b216e045418# cat test
this is a test to see if the pod is able to read from the netapp mountpoint
when reading back the file from the pod it displayed the correct message when bypassing storageclass as the volume had been provisioned by the netapp backend previously (anj)
PS C:\Users\danny> kubectl logs test --kubeconfig config
this is a test to see if the pod is able to read from the netapp mountpoint
This suggests a a problem with the storage class when communicating with the Netapp backend @balaramesh is there any additional documentation on how to configure the storage class to work alongside the Netapp backend?
It appears ANF created volumes with no mount targets, which will cause the exit status 32 errors. Is this still an issue? If this happens again, please open an Azure support ticket.
I don't the issue exists anymore, it did work when I have tested it in the past