Unable to attach iscsi volumes

[santimar]

I am using a single virtualized TrueNAS SCALE Dragonfish-24.04.2.2 to provide storage to various k3s clusters. Each cluster has it's own dataset and use truenas-csp version 2.5.1 to create volumes.

I followed install instruction and everything seems to work, since pvc can be provisioned and mounted.

However, after 7/8 nodes of different clusters start to connect to truenas, volumes struggle when they are mounted. I start to see the following event:

AttachVolume.Attach failed for volume "pvc-6cbede1c-c5e1-46dd-b6eb-8203c1e9badc" : rpc error: code = Aborted desc = There is already an operation pending for the specified id ControllerPublishVolume:my-data_first.dataset_pvc-6cbede1c-c5e1-46dd-b6eb-8203c1e9badc:1d1d92c3-5fff-4b13-4f87-dba44e3067c6
AttachVolume.Attach failed for volume "pvc-6cbede1c-c5e1-46dd-b6eb-8203c1e9badc" : rpc error: code = DeadlineExceeded desc = context deadline exceeded

Sometimes, they are still successfully mounted after some time (that varies from 10 seconds to 1 hour) but other times they stay in this error state. Also, sometimes volumes are mounted without problems.

Volumes are always correctly provisioned, since i can see them listed in truenas web ui, thay just fail to be mounted. I don't seen any obvious error or timeout log on hpe-csi-controller nor hpe-csi-node nor truenas-csp, even after enabling debug log. It just seems that the mount request is pending indefinitely (and sometimes ends successfully).

TrueNAS doesn't seems to show any resource problem like high cpu usage, and I wasn't able to find any error log even there.

Any idea?