snc icon indicating copy to clipboard operation
snc copied to clipboard

Setup kubevirt-hostpath-provisioner

Open anjannath opened this issue 3 years ago • 10 comments

this sets up dynamic provisioner for pvs backed by hosts's filesystem

the CSI driver is part of the kubevirt project and the manifests used are available in their repo at: https://github.com/kubevirt/hostpath-provisioner/tree/main/deploy/csi

this also modifies the pvc manifest for the internal registry to set the StorageClass that uses the CSI based provisioner

when creating new pvc one should use the storageClassName "crc-csi-hostpath-provisioner"

Fixes #444

anjannath avatar Oct 19 '22 08:10 anjannath

about resource consumption in the cluster, with just the plain dynamic provisioner the cluster doesn't seem to consume more resource, while deploying the operator it takes around ~750mb more memory

# CRC with the hostpath-provisioner (deployed as an old provisioner)
Capacity:
  cpu:                5
  ephemeral-storage:  31970284Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             13999196Ki
  pods:               250

Allocatable:
  cpu:                4800m
  ephemeral-storage:  29463813686
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             13538396Ki
  pods:               250

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                2250m (46%)   0 (0%)
  memory             7392Mi (55%)  0 (0%)

# CRC with the hostpath-provisioner-operator (deployed as CSI driver managed by the operator)
Capacity:
  cpu:                5
  ephemeral-storage:  31970284Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             13999204Ki
  pods:               250

Allocatable:
  cpu:                4800m
  ephemeral-storage:  29463813686
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             13538404Ki
  pods:               250

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                2300m (47%)   0 (0%)
  memory             8142Mi (61%)  0 (0%)

anjannath avatar Oct 20 '22 08:10 anjannath

after much hit and trial, CSIDriver deployment is working now, finally :slightly_smiling_face: the provided manifests don't work ootb, resource consumption is minimal compared to the operator

anjannath avatar Oct 20 '22 12:10 anjannath

/ok-to-test

praveenkumar avatar Oct 21 '22 04:10 praveenkumar

/retest

praveenkumar avatar Oct 21 '22 05:10 praveenkumar

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: anjannath

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci[bot] avatar Oct 21 '22 06:10 openshift-ci[bot]

/retest

anjannath avatar Oct 21 '22 08:10 anjannath

Somehow the CI failed because it detect that hostpath-provisioner namespace already present. I think you should add the namespace resource to one of the yaml or create a separate one and use oc apply to create it so that if you run same apply multiple time it doesn't change the resource and also not exist with non-zero.

$ cat host.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: hostpath-provisioner

$ oc apply -f host.yaml 
namespace/hostpath-provisioner unchanged

praveenkumar avatar Oct 21 '22 10:10 praveenkumar

/retest

anjannath avatar Oct 21 '22 13:10 anjannath

/retest

anjannath avatar Oct 24 '22 08:10 anjannath

/retest

anjannath avatar Oct 26 '22 08:10 anjannath

downstream snc job passed with 4.11.10 not sure what is the reason for consistent failure with the openshift CI job :/

anjannath avatar Oct 26 '22 08:10 anjannath

the image registry pvc requests volume with capacity: 100Gi the CSI driver needs actual free disk space for this and since the VM created is only 30GB it fails to create the pv

as this was working previously when manually creating the pv with the right size, we might have to do that for the registry operator and the csi driver will be used for user workloads

Events:
  Type     Reason                Age                From                                                                                            Message
  ----     ------                ----               ----                                                                                            -------
  Normal   WaitForPodScheduled   81s (x2 over 83s)  persistentvolume-controller                                                                     waiting for pod image-registry-84489d56b9-5wzhh to be scheduled
  Normal   Provisioning          15s (x7 over 78s)  kubevirt.io.hostpath-provisioner_csi-hostpathplugin-2lknn_f802a19c-3b61-4624-ae77-348077c21b0b  External provisioner is provisioning volume for claim "openshift-image-registry/crc-image-registry-storage"
  Warning  ProvisioningFailed    15s (x7 over 78s)  kubevirt.io.hostpath-provisioner_csi-hostpathplugin-2lknn_f802a19c-3b61-4624-ae77-348077c21b0b  failed to provision volume with StorageClass "crc-csi-hostpath-provisioner": created volume capacity 32212254720 less than requested capacity 107374182400
  Normal   ExternalProvisioning  6s (x7 over 78s)   persistentvolume-controller                                                                     waiting for a volume to be created, either by external provisioner "kubevirt.io.hostpath-provisioner" or manually created by system administrator

anjannath avatar Nov 02 '22 13:11 anjannath

/retest

anjannath avatar Nov 03 '22 04:11 anjannath

/lgtm

praveenkumar avatar Nov 03 '22 08:11 praveenkumar

/cherry-pick release-4.12

praveenkumar avatar Nov 03 '22 08:11 praveenkumar

@praveenkumar: only crc-org org members may request cherry-picks. You can still do the cherry-pick manually.

In response to this:

/cherry-pick release-4.12

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

/cherry-pick release-4.12

praveenkumar avatar Nov 03 '22 13:11 praveenkumar

@praveenkumar: new pull request created: #594

In response to this:

/cherry-pick release-4.12

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.