trident
trident copied to clipboard
NetApp
Unable to import volume using ontap-san-economy drivers
Hello, we are currently working on DR flow for our aws fsx ontap filesystem solution, our use-case is - we use ontap-san-economy drivers for 100 luns (PVC's) per flexVol - tagged and backed up using aws Backup service, then restored into new FSX filesystem (new cluster, new svm, volumes containing multiple LUNs imported from recovery points) next logical step is recreate pvc+pv pairs in the new cluster: we have those backed up using velero, current strategy is to take the LUN data from pv internalID: /svm/svm-8b492990/flexvol/trident_lun_pool_trident_HUUIAUUPCE/lun/trident_pvc_551d8895_a01f_479c_9437_feba9edca656
take the new fsx filesystem backendUUID, and use tridentctl import volume:
https://docs.netapp.com/us-en/trident/trident-use/vol-import.html#ontap-san
docs specifically state support for ontap-san-economy storage drivers, but do not give example for single LUN import.
recreate the pvc template for the new cluster, e.g.:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: workspaces-ws-lpaav-data
namespace: ws-ns-workspaces-ws-lpaav
labels:
workspace-resource: workspaces-ws-lpaav
workspace.devx.sap.com/backup: "true"
workspace.devx.sap.com/persistence-fsx: "true"
wsid: workspaces-ws-lpaav
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: netapp-economy-encrypted
and import (restored from the restore point) volume(s) into trident using:
tridentctl -n trident import volume fs-04c800c498b492990-economy trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_63250213_5073_4e9c_aa30_4d1a3dd19151 -f ws-ckv8p-pvc.yaml
but the command does not work:
seastranger@W-5CG3503TSZ:~/repos/esm/trident/ws$ tridentctl -n trident import volume fs-04c800c498b492990-economy trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_63250213_5073_4e9c_aa30_4d1a3dd19151 -f ws-ckv8p-pvc.yaml
Error: error communicating with Trident REST API; Post "http://127.0.0.1:8000/trident/v1/volume/import": EOF
command terminated with exit code 1
Error: exit status 1
i see the following logs in the controller:
time="2025-12-10T15:41:41Z" level=debug msg="ImportVolume: created pending PVC." PVC="&PersistentVolumeClaim{ObjectMeta:{workspaces-ws-lpaav-data ws-ns-workspaces-ws-lpaav a9f94a12-8757-424e-922b-4fb48871dd7a 55716001 0 2025-12-10 15:41:41 +0000 UTC <nil> <nil> map[workspace-resource:workspaces-ws-lpaav workspace.devx.sap.com/backup:true workspace.devx.sap.com/persistence-fsx:true wsid:workspaces-ws-lpaav] map[trident.netapp.io/importBackendUUID:0591493b-1bfc-43f6-b010-4dae8c28b844 trident.netapp.io/importOriginalName:trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_63250213_5073_4e9c_aa30_4d1a3dd19151 trident.netapp.io/notManaged:false volume.beta.kubernetes.io/storage-provisioner:csi.trident.netapp.io] [] [kubernetes.io/pvc-protection] [{trident_orchestrator Update v1 2025-12-10 15:41:41 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:trident.netapp.io/importBackendUUID\":{},\"f:trident.netapp.io/importOriginalName\":{},\"f:trident.netapp.io/notManaged\":{},\"f:volume.beta.kubernetes.io/storage-provisioner\":{}},\"f:labels\":{\".\":{},\"f:workspace-resource\":{},\"f:workspace.devx.sap.com/backup\":{},\"f:workspace.devx.sap.com/persistence-fsx\":{},\"f:wsid\":{}}},\"f:spec\":{\"f:accessModes\":{},\"f:resources\":{\"f:requests\":{\".\":{},\"f:storage\":{}}},\"f:storageClassName\":{},\"f:volumeMode\":{}}} }]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:VolumeResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{10756292608 0} {<nil>} 10756292608 DecimalSI},},},VolumeName:,Selector:nil,StorageClassName:*netapp-economy-encrypted,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,VolumeAttributesClassName:nil,},Status:PersistentVolumeClaimStatus{Phase:Pending,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},AllocatedResourceStatuses:map[ResourceName]ClaimResourceStatus{},CurrentVolumeAttributesClassName:nil,ModifyVolumeStatus:nil,},}" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:41Z" level=info msg=">>>> waitForCachedPVByName" MaxElapsedTime=3m0s Name=pvc-a9f94a12-8757-424e-922b-4fb48871dd7a logLayer=rest_frontend namespace=ws-ns-workspaces-ws-lpaav pvc=workspaces-ws-lpaav-data requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:41Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-a9f94a12-8757-424e-922b-4fb48871dd7a requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:41Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:41Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:42Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-a9f94a12-8757-424e-922b-4fb48871dd7a requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:42Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:42Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:43Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-a9f94a12-8757-424e-922b-4fb48871dd7a requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:43Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:43Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:45Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-a9f94a12-8757-424e-922b-4fb48871dd7a requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:45Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:46Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:47Z" level=debug msg="REST API call received." Duration="9.635µs" Method=GET RequestURL=/trident/v1/version Route=GetVersion logLayer=rest_frontend requestID=c75ac54b-bbff-477b-aa05-25be7ab477a3 requestSource=REST workflow="trident_rest=logger"
time="2025-12-10T15:41:47Z" level=debug msg="REST API call complete." Duration="142.291µs" Method=GET RequestURL=/trident/v1/version Route=GetVersion StatusCode=200 logLayer=rest_frontend requestID=c75ac54b-bbff-477b-aa05-25be7ab477a3 requestSource=REST workflow="trident_rest=logger"
time="2025-12-10T15:41:48Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-a9f94a12-8757-424e-922b-4fb48871dd7a requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:48Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:48Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:53Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-a9f94a12-8757-424e-922b-4fb48871dd7a requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:53Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:53Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
Could you please update the docs with example of volume import for single lun in multi-lun flexVol, using ontap-san-economy drivers ? Am i doing something wrong with my approach for volume/pvc restoration in separate cluster/fsx filesystem from aws vol backup ?
Thanks in advance
Hi to import a SAN_ECO volume, you can use the following example:
cat << EOF | kubectl apply -f -
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: lun-import2
annotations:
trident.netapp.io/importOriginalName: "scenario7_4_2/lun0"
trident.netapp.io/importBackendUUID: "8a802f70-8782-4acf-bd24-8d287f9e3e6c"
trident.netapp.io/notManaged: "false"
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: storage-class-iscsi-economy
EOF
notice the OriginalName parameter contains both ONTAP volume name & LUN name
more details here: https://github.com/YvosOnTheHub/LabNetApp/tree/master/Kubernetes_v6/Trident_Scenarios/Scenario07/4_iSCSI_Import#b-import-multiple-luns-with-ontap-san-economy--kubectl
This is exactly what i do:
seastranger@W-5CG3503TSZ:~/repos/esm/trident$ tridentctl -n trident import volume fs-04c800c498b492990-economy trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_551d8895_a01f_479c_9437_feba9edca656 -f ws/ws-w2yn2-pvc.yaml
Error: error communicating with Trident REST API; Post "http://127.0.0.1:8000/trident/v1/volume/import": EOF
command terminated with exit code 1
Error: exit status 1
seastranger@W-5CG3503TSZ:~/repos/esm/trident$ k get pvc -n ws-ns-workspaces-ws-w2yn2 -o yaml
apiVersion: v1
items:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
trident.netapp.io/importBackendUUID: 0591493b-1bfc-43f6-b010-4dae8c28b844
trident.netapp.io/importOriginalName: trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_551d8895_a01f_479c_9437_feba9edca656
trident.netapp.io/notManaged: "false"
volume.beta.kubernetes.io/storage-provisioner: csi.trident.netapp.io
creationTimestamp: "2025-12-11T09:37:40Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
workspace-resource: workspaces-ws-w2yn2
workspace.devx.sap.com/backup: "true"
workspace.devx.sap.com/persistence-fsx: "true"
wsid: workspaces-ws-w2yn2
name: workspaces-ws-w2yn2-data
namespace: ws-ns-workspaces-ws-w2yn2
resourceVersion: "55901928"
uid: d4c8d0ad-4b38-48e3-87c5-b449e180587a
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "10756292608"
storageClassName: netapp-economy-encrypted
volumeMode: Filesystem
status:
phase: Pending
kind: List
metadata:
resourceVersion: ""
inside svm the lun does exist:
seastranger@W-5CG3503TSZ:~/repos/esm/trident$ k node-shell ip-10-250-20-191.eu-north-1.compute.internal
spawning "nsenter-5wm2cz" on "ip-10-250-20-191.eu-north-1.compute.internal"
If you don't see a command prompt, try pressing enter.
root@ip-10-250-20-191:/# ssh fsxadmin@management.fs-04c800c498b492990.fsx.eu-north-1.amazonaws.com
(fsxadmin@management.fs-04c800c498b492990.fsx.eu-north-1.amazonaws.com) Password:
Last login time: 12/11/2025 07:57:58
FsxId04c800c498b492990::> set -privilege advanced
Warning: These advanced commands are potentially dangerous; use them only when directed to do so by NetApp personnel.
Do you want to continue? {y|n}: y
FsxId04c800c498b492990::*> lun show -vserver svm-8b492990 -fields path,serial,uuid,size
vserver path size serial uuid
------------ ----------------------------------------------------------------------------------------- ------- ------------ ------------------------------------
svm-8b492990 /vol/trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_551d8895_a01f_479c_9437_feba9edca656 10.02GB lWB4V]ZLM96t e415b9e4-934a-4d8b-a977-6f983cc30377
svm-8b492990 /vol/trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_551d8895_a01f_479c_9437_feba9edca656_snapshot_snapshot_cf8f0548_bd43_4aa3_a9ee_234348a63954
10.02GB lWB4V]ZLM96u 9cb0035a-208f-4284-ba00-1c1e870f1f79
svm-8b492990 /vol/trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_63250213_5073_4e9c_aa30_4d1a3dd19151 10.02GB lWB4V]ZLM96v fd6d3f7c-5dc3-4688-ab88-c3bfc7df0d98
3 entries were displayed.
FsxId04c800c498b492990::*>
the backend uuid is correct:
seastranger@W-5CG3503TSZ:~/repos/esm/trident$ k get tridentbackend -n trident | grep fs-04c800c498b492990-economy
tbe-h8r2w fs-04c800c498b492990-economy 0591493b-1bfc-43f6-b010-4dae8c28b844
the import fails without root cause in the controller log:
cat trident.log | grep d2510c24-e271-41b5-8874-5d7fbe9f4503
time="2025-12-11T09:37:40Z" level=debug msg="REST API call received." Duration="7.458µs" Method=POST RequestURL=/trident/v1/volume/import Route=ImportVolume logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="trident_rest=logger"
time="2025-12-11T09:37:40Z" level=debug msg="Found requested frontend." logLayer=core name=k8s_csi_helper requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:40Z" level=debug msg="Orchestrator#GetVolumeForImport" backendName=fs-04c800c498b492990-economy logLayer=core requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST volumeID=trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_551d8895_a01f_479c_9437_feba9edca656 workflow="volume=import"
time="2025-12-11T09:37:40Z" level=debug msg="Calculated data size after subtracting snapshot reserve from total size." dataSizeBytes=10756292608 logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST sizeWithoutSnapReserve=1.0756292608e+10 snapReserveMultiplier=1 snapshotReserve=0 totalSize=10756292608 workflow="volume=import"
time="2025-12-11T09:37:40Z" level=debug msg="Volume import determined volume size." claimSize="{{10756292608 0} {<nil>} 10756292608 DecimalSI}" data size=10756292608 logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST snapshot reserve=0 total size=10756292608 workflow="volume=import"
time="2025-12-11T09:37:40Z" level=debug msg="ImportVolume: created pending PVC." PVC="&PersistentVolumeClaim{ObjectMeta:{workspaces-ws-w2yn2-data ws-ns-workspaces-ws-w2yn2 d4c8d0ad-4b38-48e3-87c5-b449e180587a 55901928 0 2025-12-11 09:37:40 +0000 UTC <nil> <nil> map[workspace-resource:workspaces-ws-w2yn2 workspace.devx.sap.com/backup:true workspace.devx.sap.com/persistence-fsx:true wsid:workspaces-ws-w2yn2] map[trident.netapp.io/importBackendUUID:0591493b-1bfc-43f6-b010-4dae8c28b844 trident.netapp.io/importOriginalName:trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_551d8895_a01f_479c_9437_feba9edca656 trident.netapp.io/notManaged:false volume.beta.kubernetes.io/storage-provisioner:csi.trident.netapp.io] [] [kubernetes.io/pvc-protection] [{trident_orchestrator Update v1 2025-12-11 09:37:40 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:trident.netapp.io/importBackendUUID\":{},\"f:trident.netapp.io/importOriginalName\":{},\"f:trident.netapp.io/notManaged\":{},\"f:volume.beta.kubernetes.io/storage-provisioner\":{}},\"f:finalizers\":{\".\":{},\"v:\\\"kubernetes.io/pvc-protection\\\"\":{}},\"f:labels\":{\".\":{},\"f:workspace-resource\":{},\"f:workspace.devx.sap.com/backup\":{},\"f:workspace.devx.sap.com/persistence-fsx\":{},\"f:wsid\":{}}},\"f:spec\":{\"f:accessModes\":{},\"f:resources\":{\"f:requests\":{\".\":{},\"f:storage\":{}}},\"f:storageClassName\":{},\"f:volumeMode\":{}}} }]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:VolumeResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{10756292608 0} {<nil>} 10756292608 DecimalSI},},},VolumeName:,Selector:nil,StorageClassName:*netapp-economy-encrypted,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,VolumeAttributesClassName:nil,},Status:PersistentVolumeClaimStatus{Phase:Pending,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},AllocatedResourceStatuses:map[ResourceName]ClaimResourceStatus{},CurrentVolumeAttributesClassName:nil,ModifyVolumeStatus:nil,},}" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:40Z" level=info msg=">>>> waitForCachedPVByName" MaxElapsedTime=3m0s Name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a logLayer=rest_frontend namespace=ws-ns-workspaces-ws-w2yn2 pvc=workspaces-ws-w2yn2-data requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:40Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:40Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:40Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:41Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:41Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:41Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:43Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:43Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:43Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:45Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:45Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:45Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:47Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:47Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:47Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:51Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:51Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:51Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:56Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:56Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:56Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:01Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:01Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:01Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:07Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:07Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:07Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:12Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:12Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:12Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:16Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:16Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:16Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:22Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:22Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:22Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:27Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:27Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:27Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:32Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:32Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:32Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:37Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:37Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:37Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:42Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:42Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:42Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:47Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:47Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:47Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:53Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:53Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:53Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:57Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:57Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:57Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:03Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:03Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:03Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:07Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:07Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:07Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:13Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:13Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:13Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:17Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:17Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:17Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:23Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:23Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:23Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:28Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:28Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:28Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:32Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:32Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:32Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:37Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:37Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:37Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:42Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:42Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:42Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:47Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:47Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:47Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:52Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:52Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:52Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:57Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:57Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:57Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:02Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:02Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:02Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:07Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:07Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:07Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:11Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:11Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:11Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:16Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:16Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:16Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:21Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:21Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:21Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:25Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:25Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:25Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:30Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:30Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:30Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:35Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:35Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:35Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:39Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:39Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:39Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:39Z" level=info msg="<<<< waitForCachedPVByName" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:39Z" level=error msg="error waiting for PV pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a; PV pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a was not in cache after 180.00 seconds" handler=ImportVolume logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="trident_rest=logger"
time="2025-12-11T09:40:39Z" level=debug msg="REST API call complete." Duration=2m59.457350493s Method=POST RequestURL=/trident/v1/volume/import Route=ImportVolume StatusCode=400 logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="trident_rest=logger"
Please advise on how i can troubleshoot this
I'am a bit confused
- the import process failed
- a PVC is in 'pending' state
can you confirm that the PVC did not exist before trying to import the LUN ?
the pvc did not exist, it was created as a result of the tridentctl volume import command, it receives the pvc yaml file (without annot) as param:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
finalizers:
- kubernetes.io/pvc-protection
labels:
workspace-resource: workspaces-ws-w2yn2
workspace.devx.sap.com/backup: "true"
workspace.devx.sap.com/persistence-fsx: "true"
wsid: workspaces-ws-w2yn2
name: workspaces-ws-w2yn2-data
namespace: ws-ns-workspaces-ws-w2yn2
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: netapp-economy-encrypted
then the command:
tridentctl -n trident import volume fs-04c800c498b492990-economy trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_551d8895_a01f_479c_9437_feba9edca656 -f ws/ws-w2yn2-pvc.yaml
i am guessing the command adds the annotations and applies it in the cluster
The annotations:
trident.netapp.io/importOriginalName: "scenario7_4_2/lun0"
trident.netapp.io/importBackendUUID: "8a802f70-8782-4acf-bd24-8d287f9e3e6c"
trident.netapp.io/notManaged: "false"
replace the need for running the tridentctl import command
Using the annotation as suggested with the importOriginalName in the format of "volume/lun" you can import the lun you desire. With the import command you are using you are only specifying a volume and not telling Trident the name of the lun to use.
but i see that the annotation that is created on the PVC does contain both volume and lun:
annotations:
trident.netapp.io/importBackendUUID: 0591493b-1bfc-43f6-b010-4dae8c28b844
trident.netapp.io/importOriginalName: trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_551d8895_a01f_479c_9437_feba9edca656
trident.netapp.io/notManaged: "false"
vol - trident_lun_pool_trident_HUUIAUUPCE lun - trident_pvc_551d8895_a01f_479c_9437_feba9edca656 match what i see in svm (see output above)
so i do follow the documentation, and it does not work (tridentvolume and pv are not created in the cluster) any ideas why this could happen ?
@NA-Scott i have tried creating PVC with the annotation as you suggest, this does not trigger the import flow output in controller log at all, only the command tridentctl volume import triggers the logs, and the failure. applying PVC with the annotation does nothing as far as i can see
Hi, @sergeykuperman. The PVC that you entered manually or via tridentctl should trigger a CSI call to Trident just like any other PVC. The annotations @NA-Scott mentioned are applied for you by tridentctl and let Trident know it is an import instead of a new PVC. I don't see anything obviously wrong with your PVC or tridentctl invocation. But your log indicates that the PVC request never made it back to Trident via K8s, the provisioner sidecar, and CSI. I would ensure the netapp-economy-encrypted storage class exists and points to Trident as its provisioner, and then check the provisioner sidecar logs and PVC events for more clues. You could also try creating a new PVC in the new FSxN filesystem to ensure it is created and bound correctly. If that doesn't yield anything, please open a support case so @NA-Scott can look into this further.
i saw the following event in the namespace where import is attempted:
0s Normal WaitForFirstConsumer persistentvolumeclaim/workspaces-ws-w2yn2-data waiting for first consumer to be created before binding
which makes sense since the storageclass:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
labels:
name: netapp-economy-encrypted
parameters:
backendType: ontap-san-economy
csi.storage.k8s.io/node-expand-secret-name: luks-passphrases-secret
csi.storage.k8s.io/node-expand-secret-namespace: ${pvc.namespace}
csi.storage.k8s.io/node-publish-secret-name: luks-passphrases-secret
csi.storage.k8s.io/node-publish-secret-namespace: ${pvc.namespace}
csi.storage.k8s.io/node-stage-secret-name: luks-passphrases-secret
csi.storage.k8s.io/node-stage-secret-namespace: ${pvc.namespace}
fsType: ext4
selector: storagetype=devspace-economy-encrypted
provisioner: csi.trident.netapp.io
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
is wait for first consumer, so i created the luks secret with the matching passphrase from the original lun (before it was backed up) and a pod mounting this pvc, and then i got the following events:
seastranger@W-5CG3503TSZ:~/repos/esm/trident/ws$ k get events -n test -w
LAST SEEN TYPE REASON OBJECT MESSAGE
3m55s Normal Scheduled pod/test-pod Successfully assigned test/test-pod to ip-10-250-19-210.eu-north-1.compute.internal
3m54s Normal SuccessfulAttachVolume pod/test-pod AttachVolume.Attach succeeded for volume "pvc-c11773aa-b2a0-4b17-be88-ccc5836b93a0"
24s Warning FailedMount pod/test-pod MountVolume.MountDevice failed for volume "pvc-c11773aa-b2a0-4b17-be88-ccc5836b93a0" : rpc error: code = Internal desc = rpc error: code = Internal desc = failed to stage volume: LUN trident_pvc_c11773aa_b2a0_4b17_be88_ccc5836b93a0, device dm-0 already formatted with other filesystem: crypto_LUKS
14h Normal WaitForFirstConsumer persistentvolumeclaim/workspaces-ws-w2yn2-data waiting for first consumer to be created before binding
15m Normal WaitForFirstConsumer persistentvolumeclaim/workspaces-ws-w2yn2-data waiting for first consumer to be created before binding
7m51s Normal WaitForFirstConsumer persistentvolumeclaim/workspaces-ws-w2yn2-data waiting for first consumer to be created before binding
3m59s Normal Provisioning persistentvolumeclaim/workspaces-ws-w2yn2-data External provisioner is provisioning volume for claim "test/workspaces-ws-w2yn2-data"
3m59s Normal ExternalProvisioning persistentvolumeclaim/workspaces-ws-w2yn2-data Waiting for a volume to be created either by the external provisioner 'csi.trident.netapp.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
3m57s Normal ProvisioningSuccess persistentvolumeclaim/workspaces-ws-w2yn2-data provisioned a volume
3m57s Normal ProvisioningSucceeded persistentvolumeclaim/workspaces-ws-w2yn2-data Successfully provisioned volume pvc-c11773aa-b2a0-4b17-be88-ccc5836b93a0
0s Warning FailedMount pod/test-pod MountVolume.MountDevice failed for volume "pvc-c11773aa-b2a0-4b17-be88-ccc5836b93a0" : rpc error: code = Internal desc = rpc error: code = Internal desc = failed to stage volume: LUN trident_pvc_c11773aa_b2a0_4b17_be88_ccc5836b93a0, device dm-0 already formatted with other filesystem: crypto_LUKS
0s Warning FailedMount pod/test-pod MountVolume.MountDevice failed for volume "pvc-c11773aa-b2a0-4b17-be88-ccc5836b93a0" : rpc error: code = Internal desc = rpc error: code = Internal desc = failed to stage volume: LUN trident_pvc_c11773aa_b2a0_4b17_be88_ccc5836b93a0, device dm-0 already formatted with other filesystem: crypto_LUKS
0s Warning FailedMount pod/test-pod MountVolume.MountDevice failed for volume "pvc-c11773aa-b2a0-4b17-be88-ccc5836b93a0" : rpc error: code = Internal desc = rpc error: code = Internal desc = failed to stage volume: LUN trident_pvc_c11773aa_b2a0_4b17_be88_ccc5836b93a0, device dm-0 already formatted with other filesystem: crypto_LUKS
0s Warning FailedMount pod/test-pod MountVolume.MountDevice failed for volume "pvc-c11773aa-b2a0-4b17-be88-ccc5836b93a0" : rpc error: code = Internal desc = rpc error: code = Internal desc = failed to stage volume: LUN trident_pvc_c11773aa_b2a0_4b17_be88_ccc5836b93a0, device dm-0 already formatted with other filesystem: crypto_LUKS
and the logs in the provisioner, and a snippet from main controller i am attaching:
It seems that the volume is imported, and then attacher attempts to do luks format on it even tho it is already formatted in the imported volume. when i create straight new pvc in the new filesystem everything works.
@clintonk you mean aws support ticket correct ?
Thanks
K8s & CSI don't know anything about volume import, so PVCs annotated for import come in to Trident like any other PVC; your use of WaitForFirstConsumer explains why the import didn't get handled immediately. If you import a volume with Immediate binding mode, does everything work as expected? Or is there still an issue with LUKS format? I'm unsure how a LUKS issue would be limited to an import workflow. If you want to open a support ticket, I'd start with NetApp support.
no, it does not work, the binding mode only affects if the issue with LUKS format happens immediately, or when trying to attach PVC to the pod. I have tried to create account here https://mysupport.netapp.com/site/user/registration To create support ticket, but this site seems bugged, it does not let me past the email verification step, keeps asking for more codes, and sending them to email, after entering the codes i see a short message that the code has been verified, then i am redirected to the same page where i need to verify again in a loop