mayastor
mayastor copied to clipboard
openebs
requesting a new Mayastor PVC, error related to a comparison function.
Hello,
Describe the bug
When requesting a new Mayastor PVC, the PVC remains in Pending state. We encounter an error related to a comparison function in the Rust Code .
Logs :
failed to provision volume with StorageClass "mayastor-repl2": rpc error: code = Unavailable desc = error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: "Failed to wait for thread", message: "task 36179 panicked with message "user-provided comparison function does not correctly implement a total order"", kind: Aborted }'
To Reproduce Create (or apply) a new PVC.
Expected behavior The PVC should be successfully bound and reach the Bound state without any errors.
Mayastor setup
Events:
Type Reason Age From Message
Warning ProvisioningFailed 6s io.openebs.csi-mayastor_prd-app-kub-wsl12_160dff92-82a9-4ad0-bea0-b3f2cf2a2f91 failed to provision volume with StorageClass "mayastor-repl1": rpc error: code = Unavailable desc = error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: "Failed to wait for thread", message: "task 40900 panicked with message \"user-provided comparison function does not correctly implement a total order\"", kind: Aborted }'
Normal Provisioning 3s (x3 over 6s) io.openebs.csi-mayastor_prd-app-kub-wsl12_160dff92-82a9-4ad0-bea0-b3f2cf2a2f91 External provisioner is provisioning volume for claim "mayastor/pvc-test-4"
Normal ExternalProvisioning 3s (x3 over 6s) persistentvolume-controller Waiting for a volume to be created either by the external provisioner 'io.openebs.csi-mayastor' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
Warning ProvisioningFailed 3s (x2 over 5s) io.openebs.csi-mayastor_prd-app-kub-wsl12_160dff92-82a9-4ad0-bea0-b3f2cf2a2f91 failed to provision volume with StorageClass "mayastor-repl1": rpc error: code = Aborted desc = Existing volume 9f6e0192-3d61-4694-be59-d7535666189c is in insufficient state: Creating
pool mayastor :
NAMESPACE NAME NODE STATE POOL_STATUS ENCRYPTED CAPACITY USED AVAILABLE
mayastor pool-prd-app-kub-wsf11 prd-app-kub-wsf11 Created Online false 1.7 TiB 59.1 GiB 1.7 TiB
mayastor pool-prd-app-kub-wsf12 prd-app-kub-wsf12 Created Online false 1.7 TiB 120 GiB 1.6 TiB
mayastor pool-prd-app-kub-wsf13 prd-app-kub-wsf13 Created Online false 1.7 TiB 68 GiB 1.7 TiB
mayastor pool-prd-app-kub-wsf14 prd-app-kub-wsf14 Created Online false 1.7 TiB 75.9 GiB 1.7 TiB
mayastor pool-prd-app-kub-wsf15 prd-app-kub-wsf15 Created Online false 1.7 TiB 83 GiB 1.7 TiB
mayastor pool-prd-app-kub-wsf16 prd-app-kub-wsf16 Created Online false 1.7 TiB 53.3 GiB 1.7 TiB
mayastor pool-prd-app-kub-wsf17 prd-app-kub-wsf17 Created Online false 1.7 TiB 229 GiB 1.5 TiB
mayastor pool-prd-app-kub-wsf18 prd-app-kub-wsf18 Created Online false 1.7 TiB 120.7 GiB 1.6 TiB
mayastor pool-prd-app-kub-wsf19 prd-app-kub-wsf19 Created Online false 1.7 TiB 61.8 GiB 1.7 TiB
mayastor pool-prd-app-kub-wsf20 prd-app-kub-wsf20 Created Online false 1.7 TiB 116.1 GiB 1.6 TiB
mayastor pool-prd-app-kub-wsf21 prd-app-kub-wsf21 Created Online false 1.7 TiB 54.8 GiB 1.7 TiB
mayastor pool-prd-app-kub-wsf22 prd-app-kub-wsf22 Created Online false 1.7 TiB 125.8 GiB 1.6 TiB
mayastor pool-prd-app-kub-wsf23 prd-app-kub-wsf23 Created Online false 1.7 TiB 133.8 GiB 1.6 TiB
mayastor pool-prd-app-kub-wsf24 prd-app-kub-wsf24 Created Online false 1.7 TiB 120.9 GiB 1.6 TiB
mayastor pool-prd-app-kub-wsf25 prd-app-kub-wsf25 Created Online false 1.7 TiB 103.2 GiB 1.6 TiB
mayastor pool-prd-app-kub-wsf26 prd-app-kub-wsf26 Created Online false 1.7 TiB 57.4 GiB 1.7 TiB
mayastor pool-prd-app-kub-wsf27 prd-app-kub-wsf27 Created Online false 1.7 TiB 168.8 GiB 1.6 TiB
mayastor pool-prd-app-kub-wsf28 prd-app-kub-wsf28 Created Online false 1.7 TiB 55.6 GiB 1.7 TiB
mayastor pool-prd-app-kub-wsf29 prd-app-kub-wsf29 Created Online false 1.7 TiB 103.6 GiB 1.6 TiB
mayastor pool-prd-app-kub-wsf30 prd-app-kub-wsf30 Created Online false 1.7 TiB 110.7 GiB 1.6 TiB
mayastor pool-prd-app-kub-wsf31 prd-app-kub-wsf31 Created Online false 1.7 TiB 50 GiB 1.7 TiB
mayastor pool-prd-app-kub-wsf32 prd-app-kub-wsf32 Created Online false 1.7 TiB 132.2 GiB 1.6 TiB
mayastor pool-prd-app-kub-wsf33 prd-app-kub-wsf33 Created Online false 1.7 TiB 126.5 GiB 1.6 TiB
mayastor pool-prd-app-kub-wsf34 prd-app-kub-wsf34 Created Online false 1.7 TiB 61.9 GiB 1.7 TiB
stern 'ac-chart-mayastor-agent-(core|ha-node)' -n mayastor --max-log-requests 100 | grep -Ei 'panic|fail|error|warn|backtrace'
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core stack backtrace:
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core 0: 0x561aefc3b0ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core 3: 0x561aefc3cbd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core 4: 0x561aefc3c83e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core 5: 0x561aefc3d4cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core 6: 0x561aefc3d183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core 7: 0x561aefc3b5a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core 9: 0x561aefc67173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core 10: 0x561aefc6bbdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core 31: 0x561aefbda94e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
kmay get pools --show-labels
ID DISKS MANAGED NODE STATUS CAPACITY ALLOCATED AVAILABLE COMMITTED ENCRYPTED LABELS
pool-prd-app-kub-wsf27 aio:///dev/disk/by-id/virtio-mayastor_wsf27?uuid=9893fb1d-aaa2-49c0-8a24-d6f94f1c1210 true prd-app-kub-wsf27 Online 1.7 TiB 168.8 GiB 1.6 TiB 168.8 GiB false node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf20 aio:///dev/disk/by-id/virtio-mayastor_wsf20?uuid=5f319944-feb3-4761-b80c-693149e6f59a true prd-app-kub-wsf20 Online 1.7 TiB 116.1 GiB 1.6 TiB 116.1 GiB false node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf26 aio:///dev/disk/by-id/virtio-mayastor_wsf26?uuid=d600f886-340a-45a1-9076-6c61f3057af5 true prd-app-kub-wsf26 Online 1.7 TiB 57.4 GiB 1.7 TiB 57.4 GiB false node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf34 aio:///dev/disk/by-id/virtio-mayastor_wsf34?uuid=8b9ff5ae-8a5c-4f23-99c8-99de71d85a11 true prd-app-kub-wsf34 Online 1.7 TiB 61.9 GiB 1.7 TiB 61.9 GiB false node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf25 aio:///dev/disk/by-id/virtio-mayastor_wsf25?uuid=943b4ee3-6992-4d69-921d-f39711b33c6c true prd-app-kub-wsf25 Online 1.7 TiB 103.2 GiB 1.6 TiB 103.2 GiB false node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf30 aio:///dev/disk/by-id/virtio-mayastor_wsf30?uuid=73ee0168-73a4-4414-9dee-d66328e4446a true prd-app-kub-wsf30 Online 1.7 TiB 110.7 GiB 1.6 TiB 110.7 GiB false node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf11 aio:///dev/disk/by-id/virtio-mayastor_wsf11?uuid=e86ee600-6a39-48bc-9fd6-68da14ac17a0 true prd-app-kub-wsf11 Online 1.7 TiB 59.1 GiB 1.7 TiB 59.1 GiB false node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf24 aio:///dev/disk/by-id/virtio-mayastor_wsf24?uuid=6100f580-a7fe-4dc2-ad78-50dab9cb9d91 true prd-app-kub-wsf24 Online 1.7 TiB 120.9 GiB 1.6 TiB 120.9 GiB false node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf28 aio:///dev/disk/by-id/virtio-mayastor_wsf28?uuid=d2ad2f3d-5b6a-49ad-9ab8-5d7f3526d547 true prd-app-kub-wsf28 Online 1.7 TiB 55.6 GiB 1.7 TiB 55.6 GiB false node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf17 aio:///dev/disk/by-id/virtio-mayastor_wsf17?uuid=1aee9514-ac83-4b2a-bde4-5bc599dba07e true prd-app-kub-wsf17 Online 1.7 TiB 229 GiB 1.5 TiB 229 GiB false node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf12 aio:///dev/disk/by-id/virtio-mayastor_wsf12?uuid=85c2d5f5-9fde-4ca2-8e2f-05bf743d0537 true prd-app-kub-wsf12 Online 1.7 TiB 120 GiB 1.6 TiB 120 GiB false node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf31 aio:///dev/disk/by-id/virtio-mayastor_wsf31?uuid=e4d9e618-2347-40be-96d8-b891118cb12e true prd-app-kub-wsf31 Online 1.7 TiB 50 GiB 1.7 TiB 50 GiB false node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf14 aio:///dev/disk/by-id/virtio-mayastor_wsf14?uuid=c2ccb31e-b538-4dc2-89ef-dd990d4fe153 true prd-app-kub-wsf14 Online 1.7 TiB 75.9 GiB 1.7 TiB 75.9 GiB false node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf16 aio:///dev/disk/by-id/virtio-mayastor_wsf16?uuid=3ace06e5-3c90-4bf2-a0d0-92fce9594932 true prd-app-kub-wsf16 Online 1.7 TiB 53.3 GiB 1.7 TiB 53.3 GiB false node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf32 aio:///dev/disk/by-id/virtio-mayastor_wsf32?uuid=16053a58-57be-40d1-8f71-c93bd3e364a5 true prd-app-kub-wsf32 Online 1.7 TiB 132.2 GiB 1.6 TiB 132.2 GiB false node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf33 aio:///dev/disk/by-id/virtio-mayastor_wsf33?uuid=8e43a15b-8c4c-4335-be96-eba1dc25980f true prd-app-kub-wsf33 Online 1.7 TiB 126.5 GiB 1.6 TiB 126.5 GiB false node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf13 aio:///dev/disk/by-id/virtio-mayastor_wsf13?uuid=38a11fd4-f589-4018-889e-36731bccd568 true prd-app-kub-wsf13 Online 1.7 TiB 68 GiB 1.7 TiB 68 GiB false node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf18 aio:///dev/disk/by-id/virtio-mayastor_wsf18?uuid=37d34532-48e8-47ae-997f-dc93d5ab2741 true prd-app-kub-wsf18 Online 1.7 TiB 120.7 GiB 1.6 TiB 120.7 GiB false node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf21 aio:///dev/disk/by-id/virtio-mayastor_wsf21?uuid=8f44cb99-b74e-4be6-9361-f7bccc9855c4 true prd-app-kub-wsf21 Online 1.7 TiB 54.8 GiB 1.7 TiB 54.8 GiB false node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf29 aio:///dev/disk/by-id/virtio-mayastor_wsf29?uuid=6261ea27-0bd2-44e8-93ad-df0655ae4b20 true prd-app-kub-wsf29 Online 1.7 TiB 103.6 GiB 1.6 TiB 103.6 GiB false node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf22 aio:///dev/disk/by-id/virtio-mayastor_wsf22?uuid=9ff65f3d-96e5-43fb-9fb9-5127c9dafa43 true prd-app-kub-wsf22 Online 1.7 TiB 125.8 GiB 1.6 TiB 125.8 GiB false node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf23 aio:///dev/disk/by-id/virtio-mayastor_wsf23?uuid=3c7d7bec-4ed1-4590-8ede-39ea31bbc343 true prd-app-kub-wsf23 Online 1.7 TiB 133.8 GiB 1.6 TiB 133.8 GiB false node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf15 aio:///dev/disk/by-id/virtio-mayastor_wsf15?uuid=d941064d-efeb-49a7-86bd-b12798265805 true prd-app-kub-wsf15 Online 1.7 TiB 83 GiB 1.7 TiB 83 GiB false node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf19 aio:///dev/disk/by-id/virtio-mayastor_wsf19?uuid=c3508212-b080-48c8-a0ad-2d4ba26cd12e true prd-app-kub-wsf19 Online 1.7 TiB 61.8 GiB 1.7 TiB 61.8 GiB false node.k8s.toto/datacenter=datacenter2
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
labels:
argocd.argoproj.io/instance: ac-chart-mayastor
name: mayastor-repl1
parameters:
repl: "1"
provisioner: io.openebs.csi-mayastor
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
labels:
argocd.argoproj.io/instance: ac-chart-mayastor
name: mayastor-repl2
parameters:
poolHasTopologyKey: |
node.k8s.toto/datacenter
repl: "2"
provisioner: io.openebs.csi-mayastor
reclaimPolicy: Delete
volumeBindingMode: Immediate
Thank you for your help.
No, this is on the control-plane.
Is there any more backtrace information which would point to what type is at fault?
Thanks for your reply. Here’s some clarification and the current status:
We were able to successfully create almost 220 PVCs without any issues, but the problem suddenly appeared and we've been unable to find a solution since.
Our etcd instances are running on data-plane nodes, not on control-plane nodes. → Our control-plane nodes are hosted on PPCe-based machines (PowerPC), which are not compatible with Mayastor’s requirements, so we run etcd on the kvm nodes (lvm) (x86_64 with large CPU and RAM capacity).
In another environment where we also run Mayastor (same version), we don’t use any topology or node/pool labels at all — no affinity filters, no inclusion, no exclusion.
In this environment, everything is working perfectly, including: Provisioning with replica count = 2 Fast and stable volume creation No panics, no sorting-related errors, no delays
In production env
We are using label and topology , we can see on the etcd pod the datacenter label is empty
"Created":"Online"},"policy":{"self_heal":true},"topology":{"node":{"Labelled":{"exclusion":{},"inclusion":{},"creation_affinity_keys":[]}},"pool":{"Labelled":{"exclusion":{},"inclusion":{"openebs.io/created-by":"operator-diskpool","node.k8s.toto/datacenter":""}
"node.k8s.toto/datacenter":""
k get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
agent-core-7cd7969d95-4vkrf 2/2 Running 0 114m 10.233.71.149 prd-app-kub-wsl11 <none> <none>
agent-ha-node-48qhl 1/1 Running 0 109m xxx.xx.xxx.225 prd-app-kub-wsf25 <none> <none>
agent-ha-node-89ft4 1/1 Running 0 114m xxx.xx.xxx.231 prd-app-kub-wsf31 <none> <none>
agent-ha-node-8scxk 1/1 Running 0 108m xxx.xx.xxx.222 prd-app-kub-wsf22 <none> <none>
agent-ha-node-c9mr5 1/1 Running 0 107m xxx.xx.xxx.234 prd-app-kub-wsf34 <none> <none>
agent-ha-node-cqdcb 1/1 Running 0 111m xxx.xx.xxx.224 prd-app-kub-wsf24 <none> <none>
agent-ha-node-d2r7n 1/1 Running 0 108m xxx.xx.xxx.215 prd-app-kub-wsf15 <none> <none>
agent-ha-node-d9khw 1/1 Running 0 113m xxx.xx.xxx.226 prd-app-kub-wsf26 <none> <none>
agent-ha-node-ffm7w 1/1 Running 0 114m xxx.xx.xxx.223 prd-app-kub-wsf23 <none> <none>
agent-ha-node-fmshc 1/1 Running 0 106m xxx.xx.xxx.217 prd-app-kub-wsf17 <none> <none>
agent-ha-node-fphb8 1/1 Running 0 112m xxx.xx.xxx.211 prd-app-kub-wsf11 <none> <none>
agent-ha-node-g74l4 1/1 Running 0 106m xxx.xx.xxx.229 prd-app-kub-wsf29 <none> <none>
agent-ha-node-hth6c 1/1 Running 0 107m xxx.xx.xxx.230 prd-app-kub-wsf30 <none> <none>
agent-ha-node-klpnw 1/1 Running 0 110m xxx.xx.xxx.216 prd-app-kub-wsf16 <none> <none>
agent-ha-node-kswlb 1/1 Running 0 110m xxx.xx.xxx.228 prd-app-kub-wsf28 <none> <none>
agent-ha-node-nk7dg 1/1 Running 0 105m xxx.xx.xxx.219 prd-app-kub-wsf19 <none> <none>
agent-ha-node-nl5n7 1/1 Running 0 110m xxx.xx.xxx.232 prd-app-kub-wsf32 <none> <none>
agent-ha-node-pgm94 1/1 Running 0 106m xxx.xx.xxx.221 prd-app-kub-wsf21 <none> <none>
agent-ha-node-ppmm6 1/1 Running 0 108m xxx.xx.xxx.213 prd-app-kub-wsf13 <none> <none>
agent-ha-node-qwlnp 1/1 Running 0 113m xxx.xx.xxx.227 prd-app-kub-wsf27 <none> <none>
agent-ha-node-s8zl9 1/1 Running 0 108m xxx.xx.xxx.233 prd-app-kub-wsf33 <none> <none>
agent-ha-node-snnlt 1/1 Running 0 113m xxx.xx.xxx.214 prd-app-kub-wsf14 <none> <none>
agent-ha-node-vrbzg 1/1 Running 0 111m xxx.xx.xxx.212 prd-app-kub-wsf12 <none> <none>
agent-ha-node-w69xj 1/1 Running 0 108m xxx.xx.xxx.218 prd-app-kub-wsf18 <none> <none>
agent-ha-node-wqw4n 1/1 Running 0 113m xxx.xx.xxx.220 prd-app-kub-wsf20 <none> <none>
api-rest-6654667675-ntl69 1/1 Running 0 114m 10.233.97.0 prd-app-kub-wsl30 <none> <none>
csi-controller-7cd9c66c69-frw7f 6/6 Running 0 114m xxx.xx.xxx.125 prd-app-kub-wsl25 <none> <none>
csi-node-2vb65 2/2 Running 0 6h14m xxx.xx.xxx.228 prd-app-kub-wsf28 <none> <none>
csi-node-42xgc 2/2 Running 0 6h13m xxx.xx.xxx.224 prd-app-kub-wsf24 <none> <none>
csi-node-479wn 2/2 Running 0 6h19m xxx.xx.xxx.211 prd-app-kub-wsf11 <none> <none>
csi-node-55hg4 2/2 Running 0 6h12m xxx.xx.xxx.212 prd-app-kub-wsf12 <none> <none>
csi-node-6j45r 2/2 Running 0 6h21m xxx.xx.xxx.218 prd-app-kub-wsf18 <none> <none>
csi-node-9srbx 2/2 Running 0 4h58m xxx.xx.xxx.216 prd-app-kub-wsf16 <none> <none>
csi-node-c57sj 2/2 Running 0 112m xxx.xx.xxx.231 prd-app-kub-wsf31 <none> <none>
csi-node-djbb8 2/2 Running 0 6h20m xxx.xx.xxx.227 prd-app-kub-wsf27 <none> <none>
csi-node-dwknp 2/2 Running 0 6h12m xxx.xx.xxx.226 prd-app-kub-wsf26 <none> <none>
csi-node-f7hqr 2/2 Running 0 6h15m xxx.xx.xxx.217 prd-app-kub-wsf17 <none> <none>
csi-node-gjtbz 2/2 Running 0 6h12m xxx.xx.xxx.232 prd-app-kub-wsf32 <none> <none>
csi-node-gw2tt 2/2 Running 0 6h18m xxx.xx.xxx.230 prd-app-kub-wsf30 <none> <none>
csi-node-gwq6b 2/2 Running 0 6h18m xxx.xx.xxx.233 prd-app-kub-wsf33 <none> <none>
csi-node-jz87z 2/2 Running 0 6h15m xxx.xx.xxx.214 prd-app-kub-wsf14 <none> <none>
csi-node-kdg47 2/2 Running 0 4h31m xxx.xx.xxx.215 prd-app-kub-wsf15 <none> <none>
csi-node-kq7rb 2/2 Running 0 6h12m xxx.xx.xxx.219 prd-app-kub-wsf19 <none> <none>
csi-node-lmp5w 2/2 Running 0 6h15m xxx.xx.xxx.213 prd-app-kub-wsf13 <none> <none>
csi-node-msrw5 2/2 Running 0 6h21m xxx.xx.xxx.225 prd-app-kub-wsf25 <none> <none>
csi-node-pkckj 2/2 Running 0 6h16m xxx.xx.xxx.220 prd-app-kub-wsf20 <none> <none>
csi-node-psqwx 2/2 Running 0 6h14m xxx.xx.xxx.234 prd-app-kub-wsf34 <none> <none>
csi-node-r4tlh 2/2 Running 0 4h15m xxx.xx.xxx.229 prd-app-kub-wsf29 <none> <none>
csi-node-tbtdf 2/2 Running 0 6h14m xxx.xx.xxx.221 prd-app-kub-wsf21 <none> <none>
csi-node-tl692 2/2 Running 0 6h16m xxx.xx.xxx.223 prd-app-kub-wsf23 <none> <none>
csi-node-tzhdv 2/2 Running 0 6h18m xxx.xx.xxx.222 prd-app-kub-wsf22 <none> <none>
etcd-0 1/1 Running 0 110m 10.233.114.148 prd-app-kub-wsl20 <none> <none>
etcd-1 1/1 Running 0 111m 10.233.89.239 prd-app-kub-wsl22 <none> <none>
etcd-2 1/1 Running 0 112m 10.233.116.166 prd-app-kub-wsl34 <none> <none>
io-engine-42c4x 2/2 Running 0 7h55m xxx.xx.xxx.234 prd-app-kub-wsf34 <none> <none>
io-engine-4t7h7 2/2 Running 0 8h xxx.xx.xxx.218 prd-app-kub-wsf18 <none> <none>
io-engine-4x8r2 2/2 Running 0 8h xxx.xx.xxx.227 prd-app-kub-wsf27 <none> <none>
io-engine-5lvm7 2/2 Running 0 7h57m xxx.xx.xxx.230 prd-app-kub-wsf30 <none> <none>
io-engine-5qxgw 2/2 Running 0 8h xxx.xx.xxx.221 prd-app-kub-wsf21 <none> <none>
io-engine-75c7j 2/2 Running 0 7h56m xxx.xx.xxx.232 prd-app-kub-wsf32 <none> <none>
io-engine-7jskm 2/2 Running 0 4h31m xxx.xx.xxx.215 prd-app-kub-wsf15 <none> <none>
io-engine-8brfn 2/2 Running 0 8h xxx.xx.xxx.223 prd-app-kub-wsf23 <none> <none>
io-engine-9sh6t 2/2 Running 0 8h xxx.xx.xxx.217 prd-app-kub-wsf17 <none> <none>
io-engine-bgzxv 2/2 Running 0 5h14m xxx.xx.xxx.212 prd-app-kub-wsf12 <none> <none>
io-engine-c58gb 2/2 Running 0 8h xxx.xx.xxx.219 prd-app-kub-wsf19 <none> <none>
io-engine-cbfjg 2/2 Running 0 4h16m xxx.xx.xxx.229 prd-app-kub-wsf29 <none> <none>
io-engine-czprx 2/2 Running 0 8h xxx.xx.xxx.225 prd-app-kub-wsf25 <none> <none>
io-engine-f45zh 2/2 Running 0 7h55m xxx.xx.xxx.233 prd-app-kub-wsf33 <none> <none>
io-engine-fh28x 2/2 Running 0 8h xxx.xx.xxx.222 prd-app-kub-wsf22 <none> <none>
io-engine-jgx4t 2/2 Running 0 8h xxx.xx.xxx.224 prd-app-kub-wsf24 <none> <none>
io-engine-jp999 2/2 Running 0 8h xxx.xx.xxx.228 prd-app-kub-wsf28 <none> <none>
io-engine-pk59x 2/2 Running 0 5h12m xxx.xx.xxx.214 prd-app-kub-wsf14 <none> <none>
io-engine-pnf4h 2/2 Running 0 5h13m xxx.xx.xxx.213 prd-app-kub-wsf13 <none> <none>
io-engine-pstnd 2/2 Running 0 7h56m xxx.xx.xxx.231 prd-app-kub-wsf31 <none> <none>
io-engine-t4s9b 2/2 Running 0 5h xxx.xx.xxx.216 prd-app-kub-wsf16 <none> <none>
io-engine-vb92z 2/2 Running 0 5h27m xxx.xx.xxx.211 prd-app-kub-wsf11 <none> <none>
io-engine-wlh9m 2/2 Running 0 8h xxx.xx.xxx.226 prd-app-kub-wsf26 <none> <none>
io-engine-znlvj 2/2 Running 0 8h xxx.xx.xxx.220 prd-app-kub-wsf20 <none> <none>
localpv-provisioner-55666776f8-pf94d 1/1 Running 0 4d7h 10.233.98.251 prd-app-kub-wsl32 <none> <none>
obs-callhome-7f95bd4d44-nvvjx 1/1 Running 0 114m 10.233.77.54 prd-app-kub-wsl27 <none> <none>
operator-diskpool-854f766ccd-6zdv2 1/1 Running 0 6h21m 10.233.107.51 prd-app-kub-wsf16 <none> <none>
agent-core
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 0: 0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 3: 0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 4: 0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 5: 0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 6: 0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 7: 0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 9: 0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 10: 0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 31: 0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
csi-controller
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller 2025-07-22T16:55:18.096743Z INFO csi_driver::trace: [ CSI ] Create Volume Request started
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller at control-plane/csi-driver/src/trace.rs:10
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller in csi_controller::controller::create_volume with request: Request { metadata: MetadataMap { headers: {"content-type": "application/grpc", "user-agent": "grpc-go/1.69.2", "te": "trailers", "grpc-timeout": "35999957u"} }, message: CreateVolumeRequest { name: "pvc-dbafed96-a78f-4d48-9a0a-9e190e081353", capacity_range: Some(CapacityRange { required_bytes: 2000000000, limit_bytes: 0 }), volume_capabilities: [VolumeCapability { access_mode: Some(AccessMode { mode: SingleNodeWriter }), access_type: Some(Mount(MountVolume { fs_type: "ext4", mount_flags: [], volume_mount_group: "" })) }], parameters: {"protocol": "nvmf", "ioTimeout": "30", "csi.storage.k8s.io/pvc/namespace": "toto-cloud--interne--prd--glpi-nginx--arch", "csi.storage.k8s.io/pv/name": "pvc-dbafed96-a78f-4d48-9a0a-9e190e081353", "poolHasTopologyKey": "node.k8s.toto/datacenter", "csi.storage.k8s.io/pvc/name": "data-mariadb-0", "repl": "2"}, secrets: {}, volume_content_source: None, accessibility_requirements: Some(TopologyRequirement { requisite: [Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf11", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf12"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf13", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf14"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf15"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf16", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf17"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf18"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf19"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf20", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf21"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf22", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf23", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf24", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf25", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf26"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf27"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf28", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf29"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf30", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf31"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf32", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf33"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf34"} }], preferred: [Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf22", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf23"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf24"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf25"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf26", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf27"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf28"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf29"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf30", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf31", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf32", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf33"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf34", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf11", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf12"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf13", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf14", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf15", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf16", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf17"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf18", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf19"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf20"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf21"} }] }), mutable_parameters: {} }, extensions: Extensions }, volume.uuid: "dbafed96-a78f-4d48-9a0a-9e190e081353"
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller 2025-07-22T16:55:18.100878Z ERROR csi_controller::controller: error: status: AlreadyExists, message: "Existing volume dbafed96-a78f-4d48-9a0a-9e190e081353 is in insufficient state: Deleting", details: [], metadata: MetadataMap { headers: {} }
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller at control-plane/csi-driver/src/bin/controller/controller.rs:299
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller in csi_controller::controller::create_volume with request: Request { metadata: MetadataMap { headers: {"content-type": "application/grpc", "user-agent": "grpc-go/1.69.2", "te": "trailers", "grpc-timeout": "35999957u"} }, message: CreateVolumeRequest { name: "pvc-dbafed96-a78f-4d48-9a0a-9e190e081353", capacity_range: Some(CapacityRange { required_bytes: 2000000000, limit_bytes: 0 }), volume_capabilities: [VolumeCapability { access_mode: Some(AccessMode { mode: SingleNodeWriter }), access_type: Some(Mount(MountVolume { fs_type: "ext4", mount_flags: [], volume_mount_group: "" })) }], parameters: {"protocol": "nvmf", "ioTimeout": "30", "csi.storage.k8s.io/pvc/namespace": "toto-cloud--interne--prd--glpi-nginx--arch", "csi.storage.k8s.io/pv/name": "pvc-dbafed96-a78f-4d48-9a0a-9e190e081353", "poolHasTopologyKey": "node.k8s.toto/datacenter", "csi.storage.k8s.io/pvc/name": "data-mariadb-0", "repl": "2"}, secrets: {}, volume_content_source: None, accessibility_requirements: Some(TopologyRequirement { requisite: [Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf11", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf12"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf13", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf14"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf15"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf16", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf17"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf18"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf19"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf20", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf21"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf22", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf23", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf24", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf25", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf26"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf27"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf28", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf29"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf30", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf31"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf32", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf33"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf34"} }], preferred: [Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf22", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf23"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf24"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf25"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf26", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf27"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf28"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf29"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf30", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf31", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf32", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf33"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf34", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf11", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf12"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf13", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf14", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf15", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf16", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf17"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf18", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf19"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf20"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf21"} }] }), mutable_parameters: {} }, extensions: Extensions }, volume.uuid: "dbafed96-a78f-4d48-9a0a-9e190e081353"
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller
csi-controller
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller 2025-07-22T16:56:49.798146Z ERROR csi_controller::controller: error: status: Unavailable, message: "error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: \"Failed to wait for thread\", message: \"task 124905 panicked with message \\\"user-provided comparison function does not correctly implement a total order\\\"\", kind: Aborted }'", details: [], metadata: MetadataMap { headers: {} }
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller at control-plane/csi-driver/src/bin/controller/controller.rs:299
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-provisioner I0722 16:56:49.798508 1 controller.go:1115] "Temporary error received, adding PVC to claims in progress" claimUID="6b82173a-5d24-4358-8b0e-303cd0f58f08"
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-provisioner I0722 16:56:49.798529 1 controller.go:965] "Retrying syncing claim" key="6b82173a-5d24-4358-8b0e-303cd0f58f08" failures=28
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-provisioner E0722 16:56:49.798558 1 controller.go:988] "Unhandled Error" err="error syncing claim \"6b82173a-5d24-4358-8b0e-303cd0f58f08\": failed to provision volume with StorageClass \"mayastor-repl2\": rpc error: code = Unavailable desc = error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: \"Failed to wait for thread\", message: \"task 124905 panicked with message \\\"user-provided comparison function does not correctly implement a total order\\\"\", kind: Aborted }'" logger="UnhandledError"
csi-controller
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller 2025-07-22T16:55:18.100878Z ERROR csi_controller::controller: error: status: AlreadyExists, message: "Existing volume dbafed96-a78f-4d48-9a0a-9e190e081353 is in insufficient state: Deleting", details: [], metadata: MetadataMap { headers: {} }
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller at control-plane/csi-driver/src/bin/controller/controller.rs:299
describe pvc
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 3s (x3 over 9s) persistentvolume-controller Waiting for a volume to be created either by the external provisioner 'io.openebs.csi-mayastor' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
Warning ProvisioningFailed <invalid> io.openebs.csi-mayastor_prd-app-kub-wsl25_7e569ec3-7c63-4d71-aeb9-5d056cd0c01f failed to provision volume with StorageClass "mayastor-repl1": rpc error: code = Unavailable desc = error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: "Failed to wait for thread", message: "task 129518 panicked with message \"user-provided comparison function does not correctly implement a total order\"", kind: Aborted }'
Normal Provisioning <invalid> (x4 over <invalid>) io.openebs.csi-mayastor_prd-app-kub-wsl25_7e569ec3-7c63-4d71-aeb9-5d056cd0c01f External provisioner is provisioning volume for claim "mayastor/pvc-test-1"
Warning ProvisioningFailed <invalid> (x3 over <invalid>) io.openebs.csi-mayastor_prd-app-kub-wsl25_7e569ec3-7c63-4d71-aeb9-5d056cd0c01f failed to provision volume with StorageClass "mayastor-repl1": rpc error: code = Aborted desc = Existing volume 5bba40cd-d8b7-41bf-8e98-b80279be976f is in insufficient state: Creating
debian:~ (admin_apps:mayastor) % kmay delete volume 6b82173a-5d24-4358-8b0e-303cd0f58f08
> Are you sure you want to delete the resource? Yes
Failed to get volume 6b82173a-5d24-4358-8b0e-303cd0f58f08. Error error in response: status code '507 Insufficient Storage', content: 'RestJsonError { details: "", message: "SvcError :: StoreDirty: Storage Error: Volume Config for Resource id 6b82173a-5d24-4358-8b0e-303cd0f58f08 not committed to the store", kind: FailedPersist }'
on etcd pod
Replicas 1
kubectl exec -n mayastor -it ac-chart-mayastor-etcd-0 -- sh
I have no name!@etcd-0:/opt/bitnami/etcd$ etcdctl get --prefix /openebs.io/mayastor/apis/v0/clusters/0c01be95-e134-4770-b7a1-7c1f13400acc/namespaces/mayastor/ | grep 072bff26
{"uuid":"5bad0bc9-9ca4-466b-8dad-12ee0798cc7b","name":"072bff26-78f5-414a-ba2d-ba044d677dc9","node":"prd-app-kub-wsf26","children":[{"Replica":{"uuid":"e04d79cb-af13-4473-a706-b741260ed09c","share_uri":"bdev:///e04d79cb-af13-4473-a706-b741260ed09c?uuid=e04d79cb-af13-4473-a706-b741260ed09c"}}],"size":100000000,"spec_status":{"Created":"Online"},"share":"nvmf","managed":true,"owner":"072bff26-78f5-414a-ba2d-ba044d677dc9","operation":null,"nvmf_config":{"controllerIdRange":{"start":1,"end":2},"reservationKey":10208836743857753211,"reservationType":"ExclusiveAccess","preemptPolicy":"Holder"},"status_info":{"shutdown_failed":false},"allowed_hosts":["nqn.2019-05.io.openebs:node-name:prd-app-kub-wsf24"]}
{"name":"e04d79cb-af13-4473-a706-b741260ed09c","uuid":"e04d79cb-af13-4473-a706-b741260ed09c","size":100000000,"pool":"pool-prd-app-kub-wsf26","share":"none","thin":false,"status":{"Created":"online"},"managed":true,"owners":{"volume":"072bff26-78f5-414a-ba2d-ba044d677dc9","disown_all":false},"operation":null,"allowed_hosts":[],"encrypted":false}
/openebs.io/mayastor/apis/v0/clusters/0c01be95-e134-4770-b7a1-7c1f13400acc/namespaces/mayastor/VolumeSpec/072bff26-78f5-414a-ba2d-ba044d677dc9
{"uuid":"072bff26-78f5-414a-ba2d-ba044d677dc9","size":100000000,"labels":null,"num_replicas":1,"status":{"Created":"Online"},"policy":{"self_heal":true},"topology":{"node":{"Labelled":{"exclusion":{},"inclusion":{},"creation_affinity_keys":[]}},"pool":{"Labelled":{"exclusion":{},"inclusion":{"openebs.io/created-by":"operator-diskpool"},"creation_affinity_keys":[]}}},"last_nexus_id":null,"operation":null,"thin":false,"target":{"node":"prd-app-kub-wsf26","nexus":"5bad0bc9-9ca4-466b-8dad-12ee0798cc7b","protocol":"nvmf","active":true,"config":{"controllerIdRange":{"start":1,"end":2},"reservationKey":10208836743857753211,"reservationType":"ExclusiveAccess","preemptPolicy":"Holder"},"frontend":{"host_acl":[{"node_name":"prd-app-kub-wsf24","node_nqn":"nqn.2019-05.io.openebs:node-name:prd-app-kub-wsf24"}]}},"publish_context":{},"affinity_group":null,"encrypted":false}
/openebs.io/mayastor/apis/v0/clusters/0c01be95-e134-4770-b7a1-7c1f13400acc/namespaces/mayastor/volume/072bff26-78f5-414a-ba2d-ba044d677dc9/nexus/5bad0bc9-9ca4-466b-8dad-12ee0798cc7b/info
Replicas 2
kubectl exec -n mayastor -it ac-chart-mayastor-etcd-0 -- sh
etcdctl get --prefix /openebs.io/mayastor/apis/v0/clusters/0c01be95-e134-4770-b7a1-7c1f13400acc/namespaces/mayastor/ | grep f3a996b6
I have no name!@etcd-0:/opt/bitnami/etcd$ etcdctl get --prefix /openebs.io/mayastor/apis/v0/clusters/0c01be95-e134-4770-b7a1-7c1f13400acc/namespaces/mayastor/ | grep f3a996b6
{"uuid":"33f6939c-1ac6-41f7-aa0c-19a388babf00","name":"f3a996b6-af76-4bbd-9bcd-bdb0648bd313","node":"prd-app-kub-wsf23","children":[{"Replica":{"uuid":"b966bc4c-b8cb-4dc3-8cfd-3abed1d76421","share_uri":"nvmf://172.17.162.218:8420/nqn.2019-05.io.openebs:b966bc4c-b8cb-4dc3-8cfd-3abed1d76421?uuid=b966bc4c-b8cb-4dc3-8cfd-3abed1d76421"}},{"Replica":{"uuid":"f93de5d4-6b94-41ad-8591-865aaf65a44b","share_uri":"bdev:///f93de5d4-6b94-41ad-8591-865aaf65a44b?uuid=f93de5d4-6b94-41ad-8591-865aaf65a44b"}}],"size":8589934592,"spec_status":{"Created":"Online"},"share":"nvmf","managed":true,"owner":"f3a996b6-af76-4bbd-9bcd-bdb0648bd313","operation":null,"nvmf_config":{"controllerIdRange":{"start":5,"end":6},"reservationKey":12253196876332580608,"reservationType":"ExclusiveAccess","preemptPolicy":"Holder"},"status_info":{"shutdown_failed":false},"allowed_hosts":["nqn.2019-05.io.openebs:node-name:prd-app-kub-wsf32"]}
{"name":"b966bc4c-b8cb-4dc3-8cfd-3abed1d76421","uuid":"b966bc4c-b8cb-4dc3-8cfd-3abed1d76421","size":8589934592,"pool":"pool-prd-app-kub-wsf18","share":"nvmf","thin":false,"status":{"Created":"online"},"managed":true,"owners":{"volume":"f3a996b6-af76-4bbd-9bcd-bdb0648bd313","disown_all":false},"operation":null,"allowed_hosts":["nqn.2019-05.io.openebs:node-name:prd-app-kub-wsf23"],"encrypted":false}
{"name":"f93de5d4-6b94-41ad-8591-865aaf65a44b","uuid":"f93de5d4-6b94-41ad-8591-865aaf65a44b","size":8589934592,"pool":"pool-prd-app-kub-wsf23","share":"none","thin":false,"status":{"Created":"online"},"managed":true,"owners":{"volume":"f3a996b6-af76-4bbd-9bcd-bdb0648bd313","disown_all":false},"operation":null,"allowed_hosts":["nqn.2019-05.io.openebs:node-name:prd-app-kub-wsf32"],"encrypted":false}
/openebs.io/mayastor/apis/v0/clusters/0c01be95-e134-4770-b7a1-7c1f13400acc/namespaces/mayastor/VolumeSpec/f3a996b6-af76-4bbd-9bcd-bdb0648bd313
{"uuid":"f3a996b6-af76-4bbd-9bcd-bdb0648bd313","size":8589934592,"labels":null,"num_replicas":2,"status":{"Created":"Online"},"policy":{"self_heal":true},"topology":{"node":{"Labelled":{"exclusion":{},"inclusion":{},"creation_affinity_keys":[]}},"pool":{"Labelled":{"exclusion":{},"inclusion":{"openebs.io/created-by":"operator-diskpool","node.k8s.toto/datacenter":""},"creation_affinity_keys":[]}}},"last_nexus_id":null,"operation":null,"thin":false,"target":{"node":"prd-app-kub-wsf23","nexus":"33f6939c-1ac6-41f7-aa0c-19a388babf00","protocol":"nvmf","active":true,"config":{"controllerIdRange":{"start":5,"end":6},"reservationKey":12253196876332580608,"reservationType":"ExclusiveAccess","preemptPolicy":"Holder"},"frontend":{"host_acl":[{"node_name":"prd-app-kub-wsf32","node_nqn":"nqn.2019-05.io.openebs:node-name:prd-app-kub-wsf32"}]}},"publish_context":{},"affinity_group":null,"encrypted":false}
/openebs.io/mayastor/apis/v0/clusters/0c01be95-e134-4770-b7a1-7c1f13400acc/namespaces/mayastor/volume/f3a996b6-af76-4bbd-9bcd-bdb0648bd313/nexus/33f6939c-1ac6-41f7-aa0c-19a388babf00/info
Thank you for detailed information!
With that I would like to point the finger to the topology, but the topology would only be used here for filtering pools, not for sorting.
On the agent-core logs, is that the entire backtrace? Seems to be missing the entire call stack, which would help point to where in our code we're hitting it
Agent core: this is the only the same stack output I get when I try to kubectl apply -f pvc.yaml. I really don’t see anything else — just that small snippet. Same thing happens when I try to delete the PVC.
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 0: 0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 3: 0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 4: 0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 5: 0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 6: 0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 7: 0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 9: 0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 10: 0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 31: 0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 0: 0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 3: 0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 4: 0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 5: 0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 6: 0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 7: 0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 9: 0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 10: 0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 31: 0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 2025-07-22T18:32:55.545216Z ERROR core::controller::reconciler::volume::garbage_collector: Failed to destroy volume, error: Storage Error: Volume Config for Resource id dbe26fcf-d022-455e-9a80-0478ef83c5c6 not committed to the store
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 0: 0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 3: 0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 4: 0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 5: 0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 6: 0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 7: 0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 9: 0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 10: 0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 31: 0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 0: 0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 3: 0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 4: 0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 5: 0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 6: 0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 7: 0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 9: 0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 10: 0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 31: 0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 0: 0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 3: 0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 4: 0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 5: 0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 6: 0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 7: 0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 9: 0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 10: 0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 31: 0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
It always follows the same pattern: it logs INFO csi_driver::trace: [CSI] Create Volume Request started, then it fails, retries, says the volume already exists, and retries again — as you can see in the logs.
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-provisioner I0722 18:33:11.205838 1 event.go:389] "Event occurred" object="mayastor/pvc-test-1" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"mayastor/pvc-test-1\""
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller 2025-07-22T18:33:11.208110Z INFO csi_driver::trace: [ CSI ] Create Volume Request started
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller at control-plane/csi-driver/src/trace.rs:10
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller in csi_controller::controller::create_volume with request: Request { metadata: MetadataMap { headers: {"content-type": "application/grpc", "user-agent": "grpc-go/1.69.2", "te": "trailers", "grpc-timeout": "35999962u"} }, message: CreateVolumeRequest { name: "pvc-dbe26fcf-d022-455e-9a80-0478ef83c5c6", capacity_range: Some(CapacityRange { required_bytes: 7516192768, limit_bytes: 0 }), volume_capabilities: [VolumeCapability { access_mode: Some(AccessMode { mode: SingleNodeWriter }), access_type: Some(Mount(MountVolume { fs_type: "ext4", mount_flags: [], volume_mount_group: "" })) }], parameters: {"protocol": "nvmf", "ioTimeout": "30", "csi.storage.k8s.io/pvc/namespace": "mayastor", "csi.storage.k8s.io/pvc/name": "pvc-test-1", "repl": "1", "csi.storage.k8s.io/pv/name": "pvc-dbe26fcf-d022-455e-9a80-0478ef83c5c6"}, secrets: {}, volume_content_source: None, accessibility_requirements: Some(TopologyRequirement { requisite: [Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf11", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf12", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf13"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf14"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf15"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf16"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf17", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf18"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf19", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf20"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf21", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf22"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf23", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf24", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf25", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf26"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf27", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf28"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf29"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf30", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf31"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf32"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf33"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf34", "openebs.io/csi-node": "mayastor"} }], preferred: [Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf29"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf30", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf31"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf32"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf33"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf34", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf11", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf12"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf13"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf14"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf15", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf16", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf17"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf18"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf19"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf20", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf21", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf22", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf23", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf24"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf25", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf26", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf27"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf28", "openebs.io/csi-node": "mayastor"} }] }), mutable_parameters: {} }, extensions: Extensions }, volume.uuid: "dbe26fcf-d022-455e-9a80-0478ef83c5c6"
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller 2025-07-22T18:33:11.220502Z ERROR csi_controller::controller: error: status: Unavailable, message: "error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: \"Failed to wait for thread\", message: \"task 223637 panicked with message \\\"user-provided comparison function does not correctly implement a total order\\\"\", kind: Aborted }'", details: [], metadata: MetadataMap { headers: {} }
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller at control-plane/csi-driver/src/bin/controller/controller.rs:299
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller in csi_controller::controller::create_volume with request: Request { metadata: MetadataMap { headers: {"content-type": "application/grpc", "user-agent": "grpc-go/1.69.2", "te": "trailers", "grpc-timeout": "35999962u"} }, message: CreateVolumeRequest { name: "pvc-dbe26fcf-d022-455e-9a80-0478ef83c5c6", capacity_range: Some(CapacityRange { required_bytes: 7516192768, limit_bytes: 0 }), volume_capabilities: [VolumeCapability { access_mode: Some(AccessMode { mode: SingleNodeWriter }), access_type: Some(Mount(MountVolume { fs_type: "ext4", mount_flags: [], volume_mount_group: "" })) }], parameters: {"protocol": "nvmf", "ioTimeout": "30", "csi.storage.k8s.io/pvc/namespace": "mayastor", "csi.storage.k8s.io/pvc/name": "pvc-test-1", "repl": "1", "csi.storage.k8s.io/pv/name": "pvc-dbe26fcf-d022-455e-9a80-0478ef83c5c6"}, secrets: {}, volume_content_source: None, accessibility_requirements: Some(TopologyRequirement { requisite: [Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf11", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf12", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf13"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf14"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf15"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf16"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf17", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf18"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf19", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf20"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf21", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf22"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf23", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf24", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf25", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf26"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf27", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf28"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf29"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf30", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf31"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf32"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf33"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf34", "openebs.io/csi-node": "mayastor"} }], preferred: [Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf29"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf30", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf31"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf32"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf33"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf34", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf11", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf12"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf13"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf14"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf15", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf16", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf17"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf18"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf19"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf20", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf21", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf22", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf23", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf24"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf25", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf26", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf27"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf28", "openebs.io/csi-node": "mayastor"} }] }), mutable_parameters: {} }, extensions: Extensions }, volume.uuid: "dbe26fcf-d022-455e-9a80-0478ef83c5c6"
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-provisioner I0722 18:33:11.205838 1 event.go:389] "Event occurred" object="mayastor/pvc-test-1" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"mayastor/pvc-test-1\""
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller 2025-07-22T18:33:11.208110Z INFO csi_driver::trace: [ CSI ] Create Volume Request started
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller at control-plane/csi-driver/src/trace.rs:10
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller 2025-07-22T18:33:11.220502Z ERROR csi_controller::controller: error: status: Unavailable, message: "error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: \"Failed to wait for thread\", message: \"task 223637 panicked with message \\\"user-provided comparison function does not correctly implement a total order\\\"\", kind: Aborted }'", details: [], metadata: MetadataMap { headers: {} }
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller at control-plane/csi-driver/src/bin/controller/controller.rs:299
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller in csi_controller::controller::create_volume with request: Request { metadata: MetadataMap { headers: {"content-type": "application/grpc", "user-agent": "grpc-go/1.69.2", "te": "trailers", "grpc-timeout": "35999962u"} }, message: CreateVolumeRequest { name: "pvc-dbe26fcf-d022-455e-9a80-0478ef83c5c6", capacity_range: Some(CapacityRange { required_bytes: 7516192768, limit_bytes: 0 }), volume_capabilities: [VolumeCapability { access_mode: Some(AccessMode { mode: SingleNodeWriter }), access_type: Some(Mount(MountVolume { fs_type: "ext4", mount_flags: [], volume_mount_group: "" })) }], parameters: {"protocol": "nvmf", "ioTimeout": "30", "csi.storage.k8s.io/pvc/namespace": "mayastor", "csi.storage.k8s.io/pvc/name": "pvc-test-1", "repl": "1", "csi.storage.k8s.io/pv/name": "pvc-dbe26fcf-d022-455e-9a80-0478ef83c5c6"}, secrets: {}, volume_content_source: None, accessibility_requirements: Some(TopologyReq
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller 2025-07-22T18:35:18.257990Z ERROR csi_controller::controller: error: status: AlreadyExists, message: "Existing volume dbe26fcf-d022-455e-9a80-0478ef83c5c6 is in insufficient state: Deleting", details: [], metadata: MetadataMap { headers: {} }
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller 2025-07-22T18:34:14.247079Z INFO csi_driver::trace: [ CSI ] Create Volume Request started
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-provisioner I0722 18:37:26.281779 1 controller.go:1115] "Temporary error received, adding PVC to claims in progress" claimUID="dbe26fcf-d022-455e-9a80-0478ef83c5c6"
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-provisioner I0722 18:37:26.281797 1 controller.go:965] "Retrying syncing claim" key="dbe26fcf-d022-455e-9a80-0478ef83c5c6" failures=8
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-provisioner E0722 18:37:26.281820 1 controller.go:988] "Unhandled Error" err="error syncing claim \"dbe26fcf-d022-455e-9a80-0478ef83c5c6\": failed to provision volume with StorageClass \"mayastor-repl1\": rpc error: code = Unavailable desc = error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: \"Failed to wait for thread\", message: \"task 227927 panicked with message \\\"user-provided comparison function does not correctly implement a total order\\\"\", kind: Aborted }'" logger="UnhandledError"
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-provisioner I0722 18:37:26.281888 1 event.go:389] "Event occurred" object="mayastor/pvc-test-1" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"mayastor-repl1\": rpc error: code = Unavailable desc = error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: \"Failed to wait for thread\", message: \"task 227927 panicked with message \\\"user-provided comparison function does not correctly implement a total order\\\"\", kind: Aborted }'"
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller 2025-07-22T18:37:26.281552Z ERROR csi_controller::controller: error: status: Unavailable, message: "error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: \"Failed to wait for thread\", message: \"task 227927 panicked with message \\\"user-provided comparison function does not correctly implement a total order\\\"\", kind: Aborted }'", details: [], metadata: MetadataMap { headers: {} }
the complete stack with 3 tries on k delete pvc and k apply
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 0: 0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 3: 0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 4: 0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 5: 0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 6: 0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 7: 0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 9: 0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 10: 0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 31: 0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 0: 0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 3: 0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 4: 0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 5: 0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 6: 0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 7: 0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 9: 0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 10: 0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 31: 0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 2025-07-22T18:32:55.545216Z ERROR core::controller::reconciler::volume::garbage_collector: Failed to destroy volume, error: Storage Error: Volume Config for Resource id dbe26fcf-d022-455e-9a80-0478ef83c5c6 not committed to the store
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 0: 0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 3: 0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 4: 0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 5: 0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 6: 0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 7: 0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 9: 0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 10: 0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 31: 0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 0: 0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 3: 0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 4: 0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 5: 0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 6: 0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 7: 0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 9: 0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 10: 0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 31: 0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 0: 0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 3: 0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 4: 0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 5: 0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 6: 0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 7: 0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 9: 0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 10: 0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 31: 0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 0: 0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 3: 0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 4: 0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 5: 0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 6: 0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 7: 0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 9: 0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 10: 0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core 31: 0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
debian-kilian:~ (admin_apps:mayastor) % k get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
data-ac-chart-mayastor-etcd-0 Bound pvc-457c0a61-9cf1-4d11-beec-b35aaa70d673 2Gi RWO mayastor-etcd-localpv <unset> 18d
data-ac-chart-mayastor-etcd-1 Bound pvc-9f3a5aca-0e54-40a6-aa02-1ecacca818b0 2Gi RWO mayastor-etcd-localpv <unset> 18d
data-ac-chart-mayastor-etcd-2 Bound pvc-b0ff94c8-9266-4c8f-9352-1209b81c1b2e 2Gi RWO mayastor-etcd-localpv <unset> 18d
pvc-test-1 Pending mayastor-repl1 <unset> 75s
debian-kilian:~ (admin_apps:mayastor) % k get sc mayastor-repl2 -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"mayastor-repl2"},"parameters":{"ioTimeout":"30","poolHasTopologyKey":"node.k8s.asten/datacenter","protocol":"nvmf","repl":"2"},"provisioner":"io.openebs.csi-mayastor","reclaimPolicy":"Delete","volumeBindingMode":"Immediate"}
creationTimestamp: "2025-07-22T16:02:35Z"
name: mayastor-repl2
resourceVersion: "453750675"
uid: 76484fa3-da83-4340-bd3d-3d028babbed0
parameters:
ioTimeout: "30"
poolHasTopologyKey: node.k8s.asten/datacenter
protocol: nvmf
repl: "2"
provisioner: io.openebs.csi-mayastor
reclaimPolicy: Delete
volumeBindingMode: Immediate
debian-kilian:~ (admin_apps:mayastor) % k get sc mayastor-repl1 -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"mayastor-repl1"},"parameters":{"ioTimeout":"30","protocol":"nvmf","repl":"1"},"provisioner":"io.openebs.csi-mayastor","reclaimPolicy":"Delete","volumeBindingMode":"Immediate"}
creationTimestamp: "2025-07-22T16:02:17Z"
name: mayastor-repl1
resourceVersion: "453750284"
uid: 860fafb9-283f-49f6-884d-adb905a32b97
parameters:
ioTimeout: "30"
protocol: nvmf
repl: "1"
provisioner: io.openebs.csi-mayastor
reclaimPolicy: Delete
volumeBindingMode: Immediate
debian-kilian:~ (admin_apps:mayastor) % k get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
ceph-filesystem rook-ceph.cephfs.csi.ceph.com Delete Immediate true 193d
local-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 452d
mayastor-etcd-localpv openebs.io/local Delete WaitForFirstConsumer false 16d
mayastor-repl1 io.openebs.csi-mayastor Delete Immediate true 160m
mayastor-repl2 io.openebs.csi-mayastor Delete Immediate true 159m
nfs-csi nfs.csi.k8s.io Delete Immediate false 432d
openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 218d
openebs-locallvm local.csi.openebs.io Delete Immediate false 172d
Maybe the other StorageClass from OpenEBS is causing a conflict? We’re currently migrating from openebs-hostpath to mayastor, so both are still present in the cluster.
debian-kilian:~ (admin_apps:mayastor) % k get ds
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
ac-chart-mayastor-agent-ha-node 24 24 24 24 24 kubernetes.io/arch=amd64,node.k8s.asten/infra=v1,openebs.io/csi-node=mayastor 16d
ac-chart-mayastor-csi-node 24 24 24 24 24 kubernetes.io/arch=amd64,node.k8s.asten/infra=v1,openebs.io/csi-node=mayastor 16d
ac-chart-mayastor-io-engine 24 24 24 24 24 kubernetes.io/arch=amd64,openebs.io/engine=mayastor 16d
debian-kilian:~ (admin_apps:mayastor) % k get sts
NAME READY AGE
ac-chart-mayastor-etcd 3/3 16d
debian-kilian:~ (admin_apps:mayastor) % k get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
ac-chart-mayastor-agent-core 1/1 1 1 16d
ac-chart-mayastor-api-rest 1/1 1 1 16d
ac-chart-mayastor-csi-controller 1/1 1 1 16d
ac-chart-mayastor-localpv-provisioner 1/1 1 1 16d
ac-chart-mayastor-obs-callhome 1/1 1 1 16d
ac-chart-mayastor-operator-diskpool 1/1 1 1 16d
in values
node:
logLevel: info
topology:
segments:
openebs.io/csi-node: mayastor
# -- Add topology segments to the csi-node and agent-ha-node daemonset node selector
nodeSelector: True
Here’s the full backtrace I was able to capture. It shows a panic in AddVolumeReplica::with_default_policy during a slice sort operation:
user-provided comparison function does not correctly implement a total order
From what I understand, it seems like the custom sorting function used for selecting volume replicas might be violating Rust’s total ordering rules. This results in a panic that crashes the provisioning.
I’m not entirely sure what's
thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
user-provided comparison function does not correctly implement a total order
stack backtrace:
0: 0x55cd67df00ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
1: 0x55cd67e1e81b - core::fmt::write::hf5713710ce10ff22
2: 0x55cd67debce3 - std::io::Write::write_fmt::hda708db57927dacf
3: 0x55cd67df1bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
4: 0x55cd67df183e - std::panicking::default_hook::h81c8cd2e7c59ee33
5: 0x55cd67df24cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
6: 0x55cd67df2183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
7: 0x55cd67df05a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
8: 0x55cd67df1e44 - rust_begin_unwind
9: 0x55cd67e1c173 - core::panicking::panic_fmt::h3eea515d05f7a35e
10: 0x55cd67e20bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
11: 0x55cd66dfbac2 - core::slice::sort::stable::quicksort::quicksort::h891bee438e3ee28e
12: 0x55cd66dee558 - core::slice::sort::stable::drift::sort::h2840261b8def17e7
13: 0x55cd6706df7f - core::slice::sort::stable::driftsort_main::hce08a934693303a4
14: 0x55cd671f6217 - core::controller::scheduling::volume::AddVolumeReplica::with_default_policy::h97f5333af7d049c6
15: 0x55cd67221943 - <stor_port::types::v0::transport::volume::CreateVolume as core::volume::operations::CreateVolumeExe>::setup::{{closure}}::h6e4d72f70e22a13f
16: 0x55cd6721e6fa - core::volume::operations::CreateVolumeExe::run::{{closure}}::h49b64c3a9b167603
17: 0x55cd66bcef97 - core::volume::operations::<impl core::controller::resources::operations::ResourceLifecycleExt<core::volume::operations::CreateVolumeSource> for core::controller::resources::OperationGuard<core::controller::resources::ResourceMutex<stor_port::types::v0::store::volume::VolumeSpec>,stor_port::types::v0::store::volume::VolumeSpec>>::create_ext::{{closure}}::h9cbaa6de7b8c486a
18: 0x55cd66bae29a - core::volume::operations::<impl core::controller::resources::operations::ResourceLifecycle for core::controller::resources::OperationGuard<core::controller::resources::ResourceMutex<stor_port::types::v0::store::volume::VolumeSpec>,stor_port::types::v0::store::volume::VolumeSpec>>::create::{{closure}}::he284ae3b6c976fb6
19: 0x55cd66c54397 - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h5e8333050e1e7c49
20: 0x55cd6739ec8c - <opentelemetry::trace::context::WithContext<T> as core::future::future::Future>::poll::hc86ec8aeeb889947
21: 0x55cd67460cb6 - tokio::runtime::task::core::Core<T,S>::poll::heac7b23fa9c1f994
22: 0x55cd66d11555 - tokio::runtime::task::harness::Harness<T,S>::poll::hd5f5d2b78cc212db
23: 0x55cd67da50ab - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h9f947b4903e5e002
24: 0x55cd67da4061 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h64fad4113840e1a3
25: 0x55cd67da0354 - tokio::runtime::context::runtime::enter_runtime::ha9ceb71500dc86b3
26: 0x55cd67da3eca - tokio::runtime::scheduler::multi_thread::worker::run::h3505ec8fa10c7037
27: 0x55cd67d91a67 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::hfeee8786b776e854
28: 0x55cd67da7673 - tokio::runtime::task::core::Core<T,S>::poll::h4d8d9fe7b4aa06eb
29: 0x55cd67d89b84 - tokio::runtime::task::harness::Harness<T,S>::poll::h367a206a47f47d13
30: 0x55cd67d8d41f - tokio::runtime::blocking::pool::Inner::run::h799797b26549b35d
31: 0x55cd67d8f94e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
32: 0x55cd67d900c2 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h4eb31ab3cb5a138f
33: 0x55cd67df795b - std::sys::pal::unix::thread::Thread::new::thread_start::hcdbd1049068002f4
34: 0x7f70c3673272 - start_thread
35: 0x7f70c36eedec - clone3
36: 0x0 - <unknown>
https://releases.rs/docs/1.81.0/
The new sort implementations may panic if a type’s implementation of Ord (or the given comparison function) does not implement a total order as the trait requires. Ord’s supertraits (PartialOrd, Eq, and PartialEq) must also be consistent. The previous implementations would not “notice” any problem, but the new implementations have a good chance of detecting inconsistencies, throwing a panic rather than returning knowingly unsorted data.
Looks like this is our code:
13: 0x55cd6706df7f - core::slice::sort::stable::driftsort_main::hce08a934693303a4
14: 0x55cd671f6217 - core::controller::scheduling::volume::AddVolumeReplica::with_default_policy::h97f5333af7d049c6
Would you be able to try a debug image of the agent-core to see if it's capturing some more information helpful to pinpoint the issue?
Actually, i downgraded agent-core to version v2.7.0 and the issue completely disappeared — no more panics so far.
Do you have: A debug image ? Or a specific SHA/tag ?
Running a test with 600 volumes and 100 pools and using topology and not hitting this :/
I've pushed image openebs/mayastor-agent-core:v2.9.1-dev, which is exactly what it says in the tin, 2.9.1 build in debug mode
Also please ensure env RUST_BACKTRACE is set to full
Thank you
We've identified the main culprit, the dual weighted scoring, which by nature is not a total order :/
I think we should refactor that with a new system which normalizes the scores based on the entire range of values.
Meanwhile replacing sort with sort_unstable seems to reduce the panics in my testing, would you be able to test with the image openebs/mayastor-agent-core:v2.9.0-total-order ? (remember, not a fix, but seems to be more "resistent")