Pods stuck in ContainerCreating

[reitermarkus]

Describe the bug

I have two pods stuck in ContainerCreating.

One pod shows “No online replicas are available for Volume”.

  Warning  FailedAttachVolume  83s (x9 over 3m35s)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-6738a102-e39f-4413-baf9-8b5e697ebbaa" : rpc error: code = Internal desc = Operation failed: GenericOperation("error in response: status code '412 Precondition Failed', content: 'RestJsonError { details: \"No online replicas are available for Volume '6738a102-e39f-4413-baf9-8b5e697ebbaa'\", kind: FailedPrecondition }'")

The other shows “Nexus uuid already exists for nexus”.

  Warning  FailedAttachVolume  6s (x2 over 2m8s)      attachdetach-controller  (combined from similar events): AttachVolume.Attach failed for volume "pvc-134925ab-e047-4e8f-b0c9-d8c863347995" : rpc error: code = Internal desc = Operation failed: GenericOperation("error in response: status code '500 Internal Server Error', content: 'RestJsonError { details: \"create_nexus::status: Internal, message: \\\"Nexus uuid \\\\\\\"bc2d9042-4c1a-4450-baee-c91ce6c4d5d0\\\\\\\" already exists for nexus \\\\\\\"134925ab-e047-4e8f-b0c9-d8c863347995\\\\\\\"\\\", details: [], metadata: MetadataMap { headers: {\\\"content-type\\\": \\\"application/grpc\\\", \\\"date\\\": \\\"Mon, 21 Feb 2022 18:53:46 GMT\\\"} }\", kind: Internal }'")

To Reproduce

Steps to reproduce the behavior:

N/A

Expected behavior

Replicas should be available.

Screenshots

If applicable, add screenshots to help explain your problem.

** OS info (please complete the following information):**

• Distro: Talos v0.14.2
• Kernel version 5.15.23
• MayaStor revision or container image 1.0.0

Additional context

I think this happened after the node the pods were running on was rebooted. The storage class has 2 replicas. The cluster has 3 nodes and 3 Mayastor pools, all are ready/online.

[tiagolobocastro]

If you've restarted all mayastor pods you may have hit an issue where we were incorrectly setting all replicas an unusable. We've just release v1.0.1 which fixes the issue but at this point if you want to recover your data you'll have to take manual steps to do so.

[reitermarkus]

I have restored the volumes from a backup in the meantime.

I'm not sure if this is related to this issue, but kubectl mayastor get volumes shows one faulted volume:

 94d6f5c7-9a07-4c27-9cdb-8b499783571b  2         <none>       <none>         Faulted  8589934592 

This isn't referenced by any PV or PVC anymore. How can I delete this volume?

[tiagolobocastro]

Something like this: curl -X DELETE http://NODE_IP:30011/v0/volumes/94d6f5c7-9a07-4c27-9cdb-8b499783571b

[reitermarkus]

With that I get this:

{"details":"Storage Error: Volume Config for Resource id 94d6f5c7-9a07-4c27-9cdb-8b499783571b not committed to the store","kind":"FailedPersist"}

[tiagolobocastro]

Seems like we're not able to persist the config to etcd. Would you be able to share the core-agent logs? Otherwise you can try restarting the core-agent.

[reitermarkus]

core-agent logs show:

  Feb 22 23:03:41.621 ERROR core::volume::service: error: Storage Error: Volume Config for Resource id 94d6f5c7-9a07-4c27-9cdb-8b499783571b not committed to the store
    at control-plane/agents/core/src/volume/service.rs:59
    in core::volume::service::destroy_volume with request: DestroyVolume { uuid: VolumeId(94d6f5c7-9a07-4c27-9cdb-8b499783571b, "94d6f5c7-9a07-4c27-9cdb-8b499783571b") }, volume.uuid: 94d6f5c7-9a07-4c27-9cdb-8b499783571b
    in core::volume::destroy_volume with request.service: true

  Feb 22 23:03:41.621 ERROR common: Error processing message: Failed to handle message id 'v0/destroyVolume' on Channel 'v0/volume', details: Storage Error: Volume Config for Resource id 94d6f5c7-9a07-4c27-9cdb-8b499783571b not committed to the store
    at control-plane/agents/common/src/lib.rs:386

[tiagolobocastro]

So the spec is "dirty", as in not flushed to etcd. We should attempt to sync up, so either we have a bug in that logic or we can't connect to etcd. todo: we probably need a health probe so we can find out about the state of the etcd connection.. Could you please run: curl http://NODE_IP:30011/v0/volumes/94d6f5c7-9a07-4c27-9cdb-8b499783571b or kubectl mayastor get volume 94d6f5c7-9a07-4c27-9cdb-8b499783571b -o yaml

[reitermarkus]

---
spec:
  labels:
    local: "true"
  num_replicas: 2
  operation:
    operation: Destroy
  size: 8589934592
  status: Deleting
  uuid: 94d6f5c7-9a07-4c27-9cdb-8b499783571b
  topology:
    node_topology:
      explicit:
        allowed_nodes:
          - talos-cp-2
          - talos-cp-3
          - talos-cp-1
        preferred_nodes:
          - talos-cp-1
          - talos-cp-2
          - talos-cp-3
    pool_topology:
      labelled:
        exclusion: {}
        inclusion:
          openebs.io/created-by: msp-operator
  policy:
    self_heal: true
state:
  size: 8589934592
  status: Faulted
  uuid: 94d6f5c7-9a07-4c27-9cdb-8b499783571b
  replica_topology: {}

[reitermarkus]

Also tried restarting core-agent now, nothing changed.

[tiagolobocastro]

Yeah, think I found a bug... we're not deleting volumes in the deleting state when we fail the write to etcd, should be a simple fix but sadly it missed the hotfix window (was released yesterday).

[reitermarkus]

@tiagolobocastro, one of my nodes went into NotReady. After rebooting it, I have again one pod stuck in ContainerCreating with

  Warning  FailedAttachVolume  23s (x8 over 2m9s)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-9eef8a66-14b7-4a38-b49d-37db8ee7c787" : rpc error: code = Internal desc = Operation failed: GenericOperation("error in response: status code '412 Precondition Failed', content: 'RestJsonError { details: \"No online replicas are available for Volume '9eef8a66-14b7-4a38-b49d-37db8ee7c787'\", kind: FailedPrecondition }'")

kubectl mayastor get volume 9eef8a66-14b7-4a38-b49d-37db8ee7c787 -o yaml shows that both replicas for that volume are online though:

---
spec:
  labels:
    local: "true"
  num_replicas: 2
  size: 2147483648
  status: Created
  uuid: 9eef8a66-14b7-4a38-b49d-37db8ee7c787
  topology:
    node_topology:
      explicit:
        allowed_nodes:
          - talos-cp-2
          - talos-cp-3
          - talos-cp-1
        preferred_nodes:
          - talos-cp-2
          - talos-cp-3
          - talos-cp-1
    pool_topology:
      labelled:
        exclusion: {}
        inclusion:
          openebs.io/created-by: msp-operator
  policy:
    self_heal: true
state:
  size: 2147483648
  status: Online
  uuid: 9eef8a66-14b7-4a38-b49d-37db8ee7c787
  replica_topology:
    af5441f5-3141-410b-8a27-de871fcf7208:
      node: talos-cp-2
      pool: pool-on-talos-cp-2
      state: Online
    56c23152-c87c-4396-9d00-e09e58259336:
      node: talos-cp-1
      pool: pool-on-talos-cp-1
      state: Online

kubectl mayastor get volume 9eef8a66-14b7-4a38-b49d-37db8ee7c787 shows

 ID                                    REPLICAS  TARGET-NODE  ACCESSIBILITY  STATUS  SIZE 
 9eef8a66-14b7-4a38-b49d-37db8ee7c787  2         <none>       <none>         Online  2147483648 

I tried restarting all pods in the mayastor namespace, didn't seem to help.

Not sure if this is also related to the same bug.

[reitermarkus]

@tiagolobocastro, I have again encountered a problem with one of my volumes after a node reboot.

1 replica is online, 1 is degraded, and the corresponding pod is showing:

  Warning  FailedMount  38m (x123 over 23h)    kubelet  Unable to attach or mount volumes: unmounted volumes=[datadir], unattached volumes=[kube-api-access-4bhmw backupdir datadir]: timed out waiting for the condition
  Warning  FailedMount  14m (x376 over 23h)    kubelet  Unable to attach or mount volumes: unmounted volumes=[datadir], unattached volumes=[datadir kube-api-access-4bhmw backupdir]: timed out waiting for the condition
  Warning  FailedMount  3m31s (x695 over 23h)  kubelet  MountVolume.WaitForAttach failed for volume "pvc-5fafd0e8-a753-48cc-b628-bdc58448e5d8" : volume attachment is being deleted

I tried scaling down the volume to 1 replica with kubectl mayastor scale volume 5fafd0e8-a753-48cc-b628-bdc58448e5d8 1 to remove the degraded one, but it refuses with:

Failed to scale volume 5fafd0e8-a753-48cc-b628-bdc58448e5d8. Error error in response: status code '507 Insufficient Storage', content: 'RestJsonError { details: "Storage Error: Nexus Config for Resource id f2a058e9-987b-4140-bc26-06b6a13da4d9 not committed to the store", kind: FailedPersist }'

Why is there insufficient storage for removing a replica?

After shutting down the node hosting the degraded replica, running kubectl mayastor scale volume 5fafd0e8-a753-48cc-b628-bdc58448e5d8 1 produces:

Failed to scale volume 5fafd0e8-a753-48cc-b628-bdc58448e5d8. Error error in response: status code '500 Internal Server Error', content: 'RestJsonError { details: "Internal error: Failed to find the node where a replica lives", kind: Internal }'

[reitermarkus]

I forgot to mention: Even after the node with the degraded replica is rebooted, the volume's rebuildProgress is still stuck at 40.

[tiagolobocastro]

Yeah, think I found a bug... we're not deleting volumes in the deleting state when we fail the write to etcd, should be a simple fix but sadly it missed the hotfix window (was released yesterday).

Please reopen if this is still an issue