carina 腾讯云裸盘自动发现时候pod crash

容器日志 container.log

机器fdisk -l

csi-config

Jul 16 '22 06:07 guolong123

这可能是个问题，等我们在观察测试一下

Jul 19 '22 03:07 antmoveh

这块逻辑还在看，我们找了个腾讯云主机在测试

Jul 21 '22 01:07 antmoveh

这块有进展了吗，啥时候能出新版本啊？

Jul 29 '22 06:07 guolong123

这块有进展了吗，啥时候能出新版本啊？

下周我看一下，修复一下这块逻辑。原先负责这个裸盘的开发出差了。没有意外的话下周一应该能修复，至于发新版本还忒延后，到是会编译出新的可用镜像

Jul 29 '22 06:07 antmoveh

好的，感谢。目前有啥办法能跳过这个问题吗？我现在只能用loop盘

Jul 29 '22 06:07 guolong123

好的，感谢。目前有啥办法能跳过这个问题吗？我现在只能用loop盘

你是一定要用裸盘吗用lvm的方式应该无此问题

Jul 29 '22 06:07 antmoveh

我可能不太能理解lvm的方式要怎样操作？不是自己建一些loop盘来用吗？有相关文档吗

Jul 29 '22 06:07 guolong123

我可能不太能理解lvm的方式要怎样操作？不是自己建一些loop盘来用吗？有相关文档吗

https://github.com/carina-io/carina/blob/main/debug/hack/config.json 有个configmap的配置文件，policy配置为Lvm就是用lvm的方式管理磁盘。其他的就是普通的pvc sc使用方式了

Jul 29 '22 06:07 antmoveh

我发现我目前用的是lvm的形式，但看日志有报错：是因为这个分区我已经挂载的原因吗？

Jul 29 '22 06:07 guolong123

我发现我目前用的是lvm的形式，但看日志有报错：是因为这个分区我已经挂载的原因吗？

这个磁盘必须是裸盘。上边不能用任何数据和分区也不能被挂载

Jul 29 '22 06:07 antmoveh

好吧，那我后面弄个新的机器用lvm的形式试试。

Jul 29 '22 06:07 guolong123

好吧，那我后面弄个新的机器用lvm的形式试试。

多创建几个新的loop设备也可以。

Jul 29 '22 06:07 antmoveh

好的，我试过了，loop设备是ok的，目前集群中的环境就是通过这种方式来做存储的。就是机器机器重启后需要重新执行losetup -f操作。比较麻烦。

Jul 29 '22 07:07 guolong123

好的，我试过了，loop设备是ok的，目前集群中的环境就是通过这种方式来做存储的。就是机器机器重启后需要重新执行losetup -f操作。比较麻烦。

腾讯云的云盘，几块钱几十G 能用一个月

Jul 29 '22 07:07 antmoveh

它要是直接有对外的sc提供就好了。我看了好像只能在它的平台建集群才能用它的sc

Jul 29 '22 07:07 guolong123

它要是直接有对外的sc提供就好了。我看了好像只能在它的平台建集群才能用它的sc

不是用它的sc 是直接购买他的云盘挂上去。挂上去后就是个普通磁盘

Jul 29 '22 07:07 antmoveh

1 node(s) didn't find available persistent volumes to bind 我新加的裸盘，然后调度到主机的时候报这个错误是为啥啊

Jul 29 '22 08:07 guolong123

1 node(s) didn't find available persistent volumes to bind 我新加的裸盘，然后调度到主机的时候报这个错误是为啥啊

看看pvc状态是不是没bound pv

Jul 29 '22 08:07 antmoveh

Jul 29 '22 08:07 guolong123

新加的节点上没有carina-node服务？ kubectl get pvc 看看他需要的pvc是啥状态

Jul 29 '22 08:07 antmoveh

节点上是有carina-node服务的

Jul 29 '22 08:07 guolong123

看着它的日志也是正常的

Jul 29 '22 08:07 guolong123

节点上是有carina-node服务的

kubectl describe pvc xxx kubectl get lv kubectl describe lv xxx 看看有啥报错信息没有

Jul 29 '22 08:07 antmoveh

(venv) PS D:\01Work\01GitProject\04Xishu\phoenix\distribution\helm> kubectl describe pvc mysql-data-pvc
Name:          mysql-data-pvc
Namespace:     default
StorageClass:  csi-carina-lvm
Status:        Bound
Volume:        pvc-8ad586fa-3531-44c8-86c0-02cd796a9070
Labels:        app.kubernetes.io/managed-by=Helm
Annotations:   meta.helm.sh/release-name: mysql
               meta.helm.sh/release-namespace: default
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: carina.storage.io
               volume.kubernetes.io/selected-node: vm-0-10-ubuntu
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      8Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       <none>
Name:          mysql-data-pvc
Namespace:     keta-presentation
StorageClass:  csi-carina-raw
Status:        Pending
Volume:
Labels:        app.kubernetes.io/managed-by=Helm
Annotations:   meta.helm.sh/release-name: mysql
               meta.helm.sh/release-namespace: keta-presentation
               volume.beta.kubernetes.io/storage-provisioner: carina.storage.io
               volume.kubernetes.io/selected-node: vm-0-9-ubuntu
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       mysql-c877dfb4b-bhrg6
Events:
  Type     Reason                Age                   From                                                                                           Message
  ----     ------                ----                  ----                                                                                           -------
  Normal   WaitForFirstConsumer  19m                   persistentvolume-controller                                                                    waiting for first consumer to be created before binding  
  Warning  ProvisioningFailed    13m                   carina.storage.io_csi-carina-controller-7d84985df8-xngtq_c1d3151f-4e17-4121-9c8b-f439a4e4f809  failed to provision volume with StorageClass "csi-carina-raw": error generating accessibility requirements: no topology key found on CSINode vm-0-9-ubuntu
  Warning  ProvisioningFailed    9m22s (x7 over 19m)   carina.storage.io_csi-carina-controller-7d84985df8-xngtq_c1d3151f-4e17-4121-9c8b-f439a4e4f809  failed to provision volume with StorageClass "csi-carina-raw": rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Normal   ExternalProvisioning  4m44s (x62 over 19m)  persistentvolume-controller                                                                    waiting for a volume to be created, either by external provisioner "carina.storage.io" or manually created by system administrator
  Normal   Provisioning          4m22s (x11 over 19m)  carina.storage.io_csi-carina-controller-7d84985df8-xngtq_c1d3151f-4e17-4121-9c8b-f439a4e4f809  External provisioner is provisioning volume for claim "keta-presentation/mysql-data-pvc"
  Warning  ProvisioningFailed    4m21s (x3 over 18m)   carina.storage.io_csi-carina-controller-7d84985df8-xngtq_c1d3151f-4e17-4121-9c8b-f439a4e4f809  failed to provision volume with StorageClass "csi-carina-raw": rpc error: code = Internal desc = exit status 3

Jul 29 '22 08:07 guolong123

(venv) PS D:\01Work\01GitProject\04Xishu\phoenix\distribution\helm> kubectl describe pvc mysql-data-pvc
Name:          mysql-data-pvc
Namespace:     default
StorageClass:  csi-carina-lvm
Status:        Bound
Volume:        pvc-8ad586fa-3531-44c8-86c0-02cd796a9070
Labels:        app.kubernetes.io/managed-by=Helm
Annotations:   meta.helm.sh/release-name: mysql
               meta.helm.sh/release-namespace: default
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: carina.storage.io
               volume.kubernetes.io/selected-node: vm-0-10-ubuntu
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      8Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       <none>
Name:          mysql-data-pvc
Namespace:     keta-presentation
StorageClass:  csi-carina-raw
Status:        Pending
Volume:
Labels:        app.kubernetes.io/managed-by=Helm
Annotations:   meta.helm.sh/release-name: mysql
               meta.helm.sh/release-namespace: keta-presentation
               volume.beta.kubernetes.io/storage-provisioner: carina.storage.io
               volume.kubernetes.io/selected-node: vm-0-9-ubuntu
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       mysql-c877dfb4b-bhrg6
Events:
  Type     Reason                Age                   From                                                                                           Message
  ----     ------                ----                  ----                                                                                           -------
  Normal   WaitForFirstConsumer  19m                   persistentvolume-controller                                                                    waiting for first consumer to be created before binding  
  Warning  ProvisioningFailed    13m                   carina.storage.io_csi-carina-controller-7d84985df8-xngtq_c1d3151f-4e17-4121-9c8b-f439a4e4f809  failed to provision volume with StorageClass "csi-carina-raw": error generating accessibility requirements: no topology key found on CSINode vm-0-9-ubuntu
  Warning  ProvisioningFailed    9m22s (x7 over 19m)   carina.storage.io_csi-carina-controller-7d84985df8-xngtq_c1d3151f-4e17-4121-9c8b-f439a4e4f809  failed to provision volume with StorageClass "csi-carina-raw": rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Normal   ExternalProvisioning  4m44s (x62 over 19m)  persistentvolume-controller                                                                    waiting for a volume to be created, either by external provisioner "carina.storage.io" or manually created by system administrator
  Normal   Provisioning          4m22s (x11 over 19m)  carina.storage.io_csi-carina-controller-7d84985df8-xngtq_c1d3151f-4e17-4121-9c8b-f439a4e4f809  External provisioner is provisioning volume for claim "keta-presentation/mysql-data-pvc"
  Warning  ProvisioningFailed    4m21s (x3 over 18m)   carina.storage.io_csi-carina-controller-7d84985df8-xngtq_c1d3151f-4e17-4121-9c8b-f439a4e4f809  failed to provision volume with StorageClass "csi-carina-raw": rpc error: code = Internal desc = exit status 3

kubectl get node vm-0-9-ubuntu -o wide

Jul 29 '22 08:07 antmoveh

(venv) PS D:\01Work\01GitProject\04Xishu\phoenix\distribution\helm> kubectl describe pvc mysql-data-pvc
Name:          mysql-data-pvc
Namespace:     default
StorageClass:  csi-carina-lvm
Status:        Bound
Volume:        pvc-8ad586fa-3531-44c8-86c0-02cd796a9070
Labels:        app.kubernetes.io/managed-by=Helm
Annotations:   meta.helm.sh/release-name: mysql
               meta.helm.sh/release-namespace: default
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: carina.storage.io
               volume.kubernetes.io/selected-node: vm-0-10-ubuntu
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      8Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       <none>
Name:          mysql-data-pvc
Namespace:     keta-presentation
StorageClass:  csi-carina-raw
Status:        Pending
Volume:
Labels:        app.kubernetes.io/managed-by=Helm
Annotations:   meta.helm.sh/release-name: mysql
               meta.helm.sh/release-namespace: keta-presentation
               volume.beta.kubernetes.io/storage-provisioner: carina.storage.io
               volume.kubernetes.io/selected-node: vm-0-9-ubuntu
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       mysql-c877dfb4b-bhrg6
Events:
  Type     Reason                Age                   From                                                                                           Message
  ----     ------                ----                  ----                                                                                           -------
  Normal   WaitForFirstConsumer  19m                   persistentvolume-controller                                                                    waiting for first consumer to be created before binding  
  Warning  ProvisioningFailed    13m                   carina.storage.io_csi-carina-controller-7d84985df8-xngtq_c1d3151f-4e17-4121-9c8b-f439a4e4f809  failed to provision volume with StorageClass "csi-carina-raw": error generating accessibility requirements: no topology key found on CSINode vm-0-9-ubuntu
  Warning  ProvisioningFailed    9m22s (x7 over 19m)   carina.storage.io_csi-carina-controller-7d84985df8-xngtq_c1d3151f-4e17-4121-9c8b-f439a4e4f809  failed to provision volume with StorageClass "csi-carina-raw": rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Normal   ExternalProvisioning  4m44s (x62 over 19m)  persistentvolume-controller                                                                    waiting for a volume to be created, either by external provisioner "carina.storage.io" or manually created by system administrator
  Normal   Provisioning          4m22s (x11 over 19m)  carina.storage.io_csi-carina-controller-7d84985df8-xngtq_c1d3151f-4e17-4121-9c8b-f439a4e4f809  External provisioner is provisioning volume for claim "keta-presentation/mysql-data-pvc"
  Warning  ProvisioningFailed    4m21s (x3 over 18m)   carina.storage.io_csi-carina-controller-7d84985df8-xngtq_c1d3151f-4e17-4121-9c8b-f439a4e4f809  failed to provision volume with StorageClass "csi-carina-raw": rpc error: code = Internal desc = exit status 3

kubectl get node vm-0-9-ubuntu -o wide

看着像topology调度约束，看看pod的topologykey是什么，那个节点上是否有这个topologykey

Jul 29 '22 09:07 antmoveh

Jul 29 '22 09:07 guolong123

kubectl get node xxx --show-label

Jul 29 '22 09:07 antmoveh

Jul 29 '22 09:07 guolong123

vm 15? 不是vm9吗

Jul 29 '22 09:07 antmoveh

carina carina copied to clipboard

腾讯云裸盘自动发现时候pod crash

carina
carina copied to clipboard