CDI clone target pvc hostpath sc return Unexpected status code 500

[Icedroid]

kubectl get deployments cdi-deployment -n cdi -o yaml | grep image

image: harbor.ctyuncdn.cn/ecf-edge/kubevirt/cdi-controller:v1.36.0
imagePullPolicy: IfNotPresent

kubectl get dv evm-win-dv

apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
      creationTimestamp: null
      name: dv-evm-win1
    spec:
      pvc:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi
        storageClassName: hostpath #openebs hostpath localpv
      source:
        pvc:
         # source PVC volume mode is block
          name: dv-win-server-2012-r2-block
          namespace: default

kubectl logs -f pods/d4852a0e-6c42-46f6-bf0b-293daa00d5e8-source-pod

VOLUME_MODE=filesystem
MOUNT_POINT=/var/run/cdi/clone/source
/var/run/cdi/clone/source /
UPLOAD_BYTES=20940664832
I0420 02:52:04.873265 11 clone-source.go:194] content-type is "filesystem-clone"
I0420 02:52:04.873324 11 clone-source.go:195] mount is "/var/run/cdi/clone/source"
I0420 02:52:04.873340 11 clone-source.go:196] upload-bytes is 20940664832
I0420 02:52:04.873356 11 clone-source.go:213] Starting cloner target
I0420 02:52:05.398727 11 clone-source.go:225] Set header to filesystem-clone
I0420 02:52:05.886401 11 prometheus.go:69] 0.02
I0420 02:52:06.886585 11 prometheus.go:69] 0.18
I0420 02:52:07.886791 11 prometheus.go:69] 0.24
I0420 02:52:08.887022 11 prometheus.go:69] 0.30
I0420 02:52:09.887173 11 prometheus.go:69] 0.57
I0420 02:52:10.888195 11 prometheus.go:69] 0.89
I0420 02:52:11.888577 11 prometheus.go:69] 0.93
I0420 02:52:12.888776 11 prometheus.go:69] 1.13
I0420 02:52:13.888860 11 prometheus.go:69] 1.36
I0420 02:52:14.888923 11 prometheus.go:69] 1.63
I0420 02:52:15.889075 11 prometheus.go:69] 1.91
I0420 02:52:16.889216 11 prometheus.go:69] 2.15
I0420 02:52:17.889361 11 prometheus.go:69] 2.44
I0420 02:52:18.889558 11 prometheus.go:69] 2.68
I0420 02:52:19.889839 11 prometheus.go:69] 2.95
I0420 02:52:20.890073 11 prometheus.go:69] 3.01
I0420 02:52:21.890184 11 prometheus.go:69] 3.23
I0420 02:52:22.890327 11 prometheus.go:69] 3.49
I0420 02:52:23.890751 11 prometheus.go:69] 3.85
I0420 02:52:24.890877 11 prometheus.go:69] 4.03
I0420 02:52:25.891039 11 prometheus.go:69] 4.33
I0420 02:52:26.891242 11 prometheus.go:69] 4.66
I0420 02:52:27.891406 11 prometheus.go:69] 4.92
I0420 02:52:28.891606 11 prometheus.go:69] 4.95
I0420 02:52:29.891692 11 prometheus.go:69] 5.18
I0420 02:52:30.891882 11 prometheus.go:69] 5.30
I0420 02:52:31.891979 11 prometheus.go:69] 5.56
I0420 02:52:32.892253 11 prometheus.go:69] 5.72
I0420 02:52:33.892413 11 prometheus.go:69] 6.00
I0420 02:52:34.892592 11 prometheus.go:69] 6.26
I0420 02:52:35.892656 11 prometheus.go:69] 6.43
I0420 02:52:36.892953 11 prometheus.go:69] 6.74
I0420 02:52:37.893124 11 prometheus.go:69] 6.96
I0420 02:52:38.905107 11 prometheus.go:69] 7.22
I0420 02:52:39.905363 11 prometheus.go:69] 7.50
I0420 02:52:40.905522 11 prometheus.go:69] 7.80
I0420 02:52:41.905684 11 prometheus.go:69] 7.94
I0420 02:52:42.905826 11 prometheus.go:69] 8.02
I0420 02:52:43.906044 11 prometheus.go:69] 8.20
I0420 02:52:44.906190 11 prometheus.go:69] 8.45
I0420 02:52:45.906344 11 prometheus.go:69] 8.72
I0420 02:52:46.906773 11 prometheus.go:69] 9.04
I0420 02:52:47.906912 11 prometheus.go:69] 9.30
I0420 02:52:48.906992 11 prometheus.go:69] 9.65
I0420 02:52:49.907153 11 prometheus.go:69] 9.79
I0420 02:52:50.907368 11 prometheus.go:69] 9.97
I0420 02:52:51.907517 11 prometheus.go:69] 10.14
I0420 02:52:52.907698 11 prometheus.go:69] 10.36
I0420 02:52:53.907835 11 prometheus.go:69] 10.63
I0420 02:52:54.955054 11 prometheus.go:69] 10.97
I0420 02:52:55.955143 11 prometheus.go:69] 11.30
I0420 02:52:56.955219 11 prometheus.go:69] 11.52
I0420 02:52:57.955330 11 prometheus.go:69] 11.80
I0420 02:52:58.955485 11 prometheus.go:69] 12.02
I0420 02:52:59.955546 11 prometheus.go:69] 12.35
I0420 02:53:00.955661 11 prometheus.go:69] 12.50
I0420 02:53:01.955794 11 prometheus.go:69] 12.80
I0420 02:53:02.956053 11 prometheus.go:69] 13.04
I0420 02:53:03.956117 11 prometheus.go:69] 13.36
I0420 02:53:04.956186 11 prometheus.go:69] 13.63
I0420 02:53:05.956233 11 prometheus.go:69] 13.86
I0420 02:53:06.956314 11 prometheus.go:69] 14.22
I0420 02:53:07.956464 11 prometheus.go:69] 14.57
I0420 02:53:08.956621 11 prometheus.go:69] 14.70
I0420 02:53:09.956673 11 prometheus.go:69] 15.12
I0420 02:53:10.956744 11 prometheus.go:69] 15.50
I0420 02:53:11.956857 11 prometheus.go:69] 15.74
I0420 02:53:12.956942 11 prometheus.go:69] 16.05
I0420 02:53:13.957112 11 prometheus.go:69] 16.30
I0420 02:53:14.957274 11 prometheus.go:69] 16.46
I0420 02:53:15.957412 11 prometheus.go:69] 16.59
I0420 02:53:16.957640 11 prometheus.go:69] 16.81
I0420 02:53:17.957739 11 prometheus.go:69] 17.14
I0420 02:53:18.957811 11 prometheus.go:69] 17.37
I0420 02:53:19.957885 11 prometheus.go:69] 17.71
I0420 02:53:20.957946 11 prometheus.go:69] 18.06
I0420 02:53:21.958040 11 prometheus.go:69] 18.41
I0420 02:53:22.958222 11 prometheus.go:69] 18.73
I0420 02:53:23.958280 11 prometheus.go:69] 18.89
I0420 02:53:24.958341 11 prometheus.go:69] 19.22
I0420 02:53:25.958423 11 prometheus.go:69] 19.56
I0420 02:53:26.958537 11 prometheus.go:69] 19.69
I0420 02:53:27.958692 11 prometheus.go:69] 19.96
I0420 02:53:28.958870 11 prometheus.go:69] 20.05
I0420 02:53:29.958990 11 prometheus.go:69] 20.24
I0420 02:53:30.959080 11 prometheus.go:69] 20.46
I0420 02:53:31.959215 11 prometheus.go:69] 20.79
I0420 02:53:32.959301 11 prometheus.go:69] 21.14
I0420 02:53:33.959432 11 prometheus.go:69] 21.43
I0420 02:53:34.959598 11 prometheus.go:69] 21.66
I0420 02:53:35.959784 11 prometheus.go:69] 22.01
I0420 02:53:36.959945 11 prometheus.go:69] 22.33
I0420 02:53:37.960533 11 prometheus.go:69] 22.57
I0420 02:53:38.960734 11 prometheus.go:69] 22.73
I0420 02:53:39.960796 11 prometheus.go:69] 23.07
I0420 02:53:40.960891 11 prometheus.go:69] 23.46
I0420 02:53:41.961010 11 prometheus.go:69] 23.61
I0420 02:53:42.961198 11 prometheus.go:69] 23.85
I0420 02:53:43.961282 11 prometheus.go:69] 24.16
I0420 02:53:44.961355 11 prometheus.go:69] 24.41
I0420 02:53:45.961429 11 prometheus.go:69] 24.63
I0420 02:53:46.961631 11 prometheus.go:69] 24.77
I0420 02:53:47.961901 11 prometheus.go:69] 25.11
I0420 02:53:48.962086 11 prometheus.go:69] 25.35
I0420 02:53:49.962299 11 prometheus.go:69] 25.41
I0420 02:53:50.962340 11 prometheus.go:69] 25.72
I0420 02:53:51.962497 11 prometheus.go:69] 26.04
I0420 02:53:52.962555 11 prometheus.go:69] 26.29
I0420 02:53:53.962603 11 prometheus.go:69] 26.49
I0420 02:53:54.962728 11 prometheus.go:69] 26.74
I0420 02:53:55.962776 11 prometheus.go:69] 27.03
I0420 02:53:56.962892 11 prometheus.go:69] 27.32
I0420 02:53:57.962934 11 prometheus.go:69] 27.48
I0420 02:53:58.963084 11 prometheus.go:69] 27.66
I0420 02:53:59.963147 11 prometheus.go:69] 27.80
I0420 02:54:00.963336 11 prometheus.go:69] 28.12
I0420 02:54:01.963478 11 prometheus.go:69] 28.18
I0420 02:54:02.963594 11 prometheus.go:69] 28.41
I0420 02:54:03.963757 11 prometheus.go:69] 28.73
I0420 02:54:04.963909 11 prometheus.go:69] 28.99
I0420 02:54:05.963968 11 prometheus.go:69] 29.24
I0420 02:54:06.964031 11 prometheus.go:69] 29.58
I0420 02:54:07.964114 11 prometheus.go:69] 29.82
I0420 02:54:08.964293 11 prometheus.go:69] 30.18
I0420 02:54:09.964377 11 prometheus.go:69] 30.51
I0420 02:54:10.964449 11 prometheus.go:69] 30.82
I0420 02:54:11.964495 11 prometheus.go:69] 31.11
I0420 02:54:12.964545 11 prometheus.go:69] 31.43
I0420 02:54:13.964739 11 prometheus.go:69] 31.57
I0420 02:54:14.964804 11 prometheus.go:69] 31.84
I0420 02:54:15.964875 11 prometheus.go:69] 32.04
I0420 02:54:16.965035 11 prometheus.go:69] 32.15
I0420 02:54:17.965085 11 prometheus.go:69] 32.37
I0420 02:54:18.965232 11 prometheus.go:69] 32.65
I0420 02:54:19.965341 11 prometheus.go:69] 32.91
I0420 02:54:20.966327 11 prometheus.go:69] 33.07
I0420 02:54:21.966540 11 prometheus.go:69] 33.39
I0420 02:54:22.966671 11 prometheus.go:69] 33.65
I0420 02:54:23.966737 11 prometheus.go:69] 33.99
I0420 02:54:24.966895 11 prometheus.go:69] 34.38
I0420 02:54:25.967107 11 prometheus.go:69] 34.63
I0420 02:54:26.967268 11 prometheus.go:69] 34.87
I0420 02:54:27.967416 11 prometheus.go:69] 35.09
I0420 02:54:28.967511 11 prometheus.go:69] 35.42
I0420 02:54:29.967674 11 prometheus.go:69] 35.68
I0420 02:54:30.967751 11 prometheus.go:69] 35.77
I0420 02:54:31.967834 11 prometheus.go:69] 36.07
I0420 02:54:32.968093 11 prometheus.go:69] 36.37
I0420 02:54:33.968264 11 prometheus.go:69] 36.68
I0420 02:54:34.968440 11 prometheus.go:69] 36.79
I0420 02:54:35.968531 11 prometheus.go:69] 37.07
I0420 02:54:36.968634 11 prometheus.go:69] 37.38
I0420 02:54:37.968726 11 prometheus.go:69] 37.69
I0420 02:54:38.968805 11 prometheus.go:69] 38.01
I0420 02:54:39.969031 11 prometheus.go:69] 38.36
I0420 02:54:40.969197 11 prometheus.go:69] 38.67
I0420 02:54:41.969277 11 prometheus.go:69] 38.88
I0420 02:54:42.969446 11 prometheus.go:69] 39.05
I0420 02:54:43.969640 11 prometheus.go:69] 39.15
I0420 02:54:44.969814 11 prometheus.go:69] 39.35
I0420 02:54:45.969973 11 prometheus.go:69] 39.53
I0420 02:54:46.970152 11 prometheus.go:69] 39.77
I0420 02:54:47.970214 11 prometheus.go:69] 40.12
I0420 02:54:48.970260 11 prometheus.go:69] 40.35
I0420 02:54:49.970308 11 prometheus.go:69] 40.60
I0420 02:54:50.970370 11 prometheus.go:69] 40.95
I0420 02:54:51.970419 11 prometheus.go:69] 41.25
I0420 02:54:52.970649 11 prometheus.go:69] 41.50
I0420 02:54:53.970697 11 prometheus.go:69] 41.85
I0420 02:54:54.970824 11 prometheus.go:69] 42.14
I0420 02:54:55.970877 11 prometheus.go:69] 42.49
I0420 02:54:56.970967 11 prometheus.go:69] 42.60
I0420 02:54:57.971235 11 prometheus.go:69] 42.81
I0420 02:54:58.971403 11 prometheus.go:69] 43.14
I0420 02:54:59.971459 11 prometheus.go:69] 43.50
I0420 02:55:00.971604 11 prometheus.go:69] 43.88
I0420 02:55:01.971778 11 prometheus.go:69] 44.06
I0420 02:55:02.971896 11 prometheus.go:69] 44.28
I0420 02:55:03.971942 11 prometheus.go:69] 44.49
I0420 02:55:04.972115 11 prometheus.go:69] 44.77
I0420 02:55:05.972302 11 prometheus.go:69] 44.89
I0420 02:55:06.972402 11 prometheus.go:69] 45.19
I0420 02:55:07.972568 11 prometheus.go:69] 45.32
I0420 02:55:08.972798 11 prometheus.go:69] 45.55
I0420 02:55:09.972924 11 prometheus.go:69] 45.88
I0420 02:55:10.973113 11 prometheus.go:69] 45.96
I0420 02:55:11.973338 11 prometheus.go:69] 45.99
I0420 02:55:12.973525 11 prometheus.go:69] 46.27
I0420 02:55:13.973598 11 prometheus.go:69] 46.41
I0420 02:55:14.973669 11 prometheus.go:69] 46.76
I0420 02:55:15.973747 11 prometheus.go:69] 47.06
I0420 02:55:16.973807 11 prometheus.go:69] 47.36
I0420 02:55:17.973858 11 prometheus.go:69] 47.63
I0420 02:55:18.974019 11 prometheus.go:69] 47.86
I0420 02:55:19.974241 11 prometheus.go:69] 48.20
I0420 02:55:20.974304 11 prometheus.go:69] 48.54
I0420 02:55:21.974493 11 prometheus.go:69] 48.83
I0420 02:55:22.974586 11 prometheus.go:69] 49.07
I0420 02:55:23.974650 11 prometheus.go:69] 49.30
I0420 02:55:24.974818 11 prometheus.go:69] 49.64
I0420 02:55:25.974969 11 prometheus.go:69] 49.80
I0420 02:55:26.975106 11 prometheus.go:69] 50.00
I0420 02:55:27.975252 11 prometheus.go:69] 50.06
I0420 02:55:28.975406 11 prometheus.go:69] 50.12
I0420 02:55:29.975493 11 prometheus.go:69] 50.45
I0420 02:55:30.975747 11 prometheus.go:69] 50.61
I0420 02:55:31.975798 11 prometheus.go:69] 50.75
I0420 02:55:32.975915 11 prometheus.go:69] 51.01
I0420 02:55:33.975993 11 prometheus.go:69] 51.25
I0420 02:55:34.976152 11 prometheus.go:69] 51.55
I0420 02:55:35.976832 11 prometheus.go:69] 51.82
I0420 02:55:36.977037 11 prometheus.go:69] 52.14
I0420 02:55:37.977121 11 prometheus.go:69] 52.42
I0420 02:55:38.977297 11 prometheus.go:69] 52.62
I0420 02:55:39.977482 11 prometheus.go:69] 52.96
I0420 02:55:40.977691 11 prometheus.go:69] 53.24
I0420 02:55:41.977854 11 prometheus.go:69] 53.57
I0420 02:55:42.978013 11 prometheus.go:69] 53.90
I0420 02:55:43.978080 11 prometheus.go:69] 54.23
I0420 02:55:44.978468 11 prometheus.go:69] 54.51
I0420 02:55:45.978548 11 prometheus.go:69] 54.72
I0420 02:55:46.978680 11 prometheus.go:69] 54.98
I0420 02:55:47.978801 11 prometheus.go:69] 55.24
I0420 02:55:48.978927 11 prometheus.go:69] 55.44
I0420 02:55:49.979075 11 prometheus.go:69] 55.57
I0420 02:55:50.979186 11 prometheus.go:69] 55.87
I0420 02:55:51.979232 11 prometheus.go:69] 56.13
I0420 02:55:52.979281 11 prometheus.go:69] 56.46
I0420 02:55:53.979344 11 prometheus.go:69] 56.71
I0420 02:55:54.079039 11 clone-source.go:127] Wrote 11878174720 bytes
I0420 02:55:54.979517 11 prometheus.go:69] 100.00
F0420 02:55:57.125769 11 clone-source.go:234] Unexpected status code 500
goroutine 1 [running]:
kubevirt.io/containerized-data-importer/vendor/k8s.io/klog/v2.stacks(0xc000010001, 0xc0002b8000, 0x4e, 0xa3)
vendor/k8s.io/klog/v2/klog.go:1026 +0xb8
kubevirt.io/containerized-data-importer/vendor/k8s.io/klog/v2.(*loggingT).output(0x154b4e0, 0xc000000003, 0x0, 0x0, 0xc0002610a0, 0x14e21cb, 0xf, 0xea, 0x0)
vendor/k8s.io/klog/v2/klog.go:975 +0x1a3
kubevirt.io/containerized-data-importer/vendor/k8s.io/klog/v2.(*loggingT).printf(0x154b4e0, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0xd8e23c, 0x19, 0xc0003cca00, 0x1, ...)
vendor/k8s.io/klog/v2/klog.go:750 +0x18b
kubevirt.io/containerized-data-importer/vendor/k8s.io/klog/v2.Fatalf(...)
vendor/k8s.io/klog/v2/klog.go:1502
main.main()
cmd/cdi-cloner/clone-source.go:234 +0x991

kubectl logs -f pods/cdi-upload-dv-win-server-2012

E0420 02:48:44.164399 1 util.go:201] signal: killed
E0420 02:48:44.164444 1 uploadserver.go:392] Saving stream failed: error unarchiving to /data: signal: killed
2022/04/20 02:48:46 http: TLS handshake error from [::1]:60808: remote error: tls: bad certificate
I0420 02:48:48.201727 1 uploadserver.go:379] Content type header is "filesystem-clone"

[awels]

E0420 02:48:44.164444 1 uploadserver.go:392] Saving stream failed: error unarchiving to /data: signal: killed is the offending line that is causing the problem. That shows up as the 500 in the source pod. Could it be that the memory or cpu limit is too low and kubernetes is killing the process?

[Icedroid]

@awels I have edited cdi config. also meet the same problem: ❯ kg cdi cdi -n cdi -o yaml

apiVersion: cdi.kubevirt.io/v1beta1
kind: CDI
metadata:
  annotations:
    cdi.kubevirt.io/configAuthority: ""
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"CDI","metadata":{"annotations":{},"name":"cdi"},"spec":{"config":{"filesystemOverhead":{"global":"0"}},"imagePullPolicy":"If
NotPresent","infra":{"nodeSelector":{"kubernetes.io/os":"linux","node-role.kubernetes.io/master":""},"tolerations":[{"key":"CriticalAddonsOnly","operator":"Exists"}]},"workload":
{"nodeSelector":{"kubernetes.io/os":"linux"}}}}
  creationTimestamp: "2021-08-02T09:26:14Z"
  finalizers:
  - operator.cdi.kubevirt.io
  generation: 78
  name: cdi
  resourceVersion: "173403216"
  selfLink: /apis/cdi.kubevirt.io/v1beta1/cdis/cdi
  uid: 11b8022f-0b10-47a6-9ccf-d9abddaa7d85
spec:
  config:
    filesystemOverhead:
      global: "0"
    podResourceRequirements:
      limits:
        cpu: "2"
        memory: 4096M
  imagePullPolicy: IfNotPresent
  infra:
    nodeSelector:
      kubernetes.io/os: linux
      node-role.kubernetes.io/master: ""
    tolerations:
    - key: CriticalAddonsOnly
      operator: Exists
  workload:
    nodeSelector:
      kubernetes.io/os: linux

[awels]

Is the target storage full? I can only make some guesses here without a lot more information.

[Icedroid]

Is the target storage full? I can only make some guesses here without a lot more information.

I confirm target storage is have avaliable space. The problem happen when copy source Ceph SC to target Hostpath SC.

[kubevirt-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[kubevirt-bot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[kubevirt-bot]

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

/close

[kubevirt-bot]

@kubevirt-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.