[QE] CRC free PVs disappeared

[JiriHusak-lab]

General information

• OS: Linux
• Hypervisor: KVM / Hyper-V / hyperkit / vfkit
• Did you run crc setup before starting it (Yes)?
• Running CRC on: VM

CRC version

# Put `crc version` output here
CRC version: 2.27.0+71615e
OpenShift version: 4.13.12
Podman version: 4.4.4

CRC status

# Put `crc status --log-level debug` output here
DEBU CRC version: 2.27.0+71615e                   
DEBU OpenShift version: 4.13.12                   
DEBU Podman version: 4.4.4                        
DEBU Running 'crc status'                         
CRC VM:          Running
OpenShift:       Running (v4.13.12)
RAM Usage:       17.82GB of 56.1GB
Disk Usage:      109.4GB of 267.8GB (Inside the CRC VM)
Cache Usage:     20.57GB
Cache Directory: /var/crc/.crc/cache

CRC config

# Put `crc config view` output here
- consent-telemetry                     : no
- cpus                                  : 24
- disk-size                             : 250
- memory                                : 54576

Host Operating System

# Put the output of `cat /etc/os-release` in case of Linux
# put the output of `sw_vers` in case of Mac
# Put the output of `systeminfo` in case of Windows
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

Steps to reproduce

Hi,

please give me a hint.

I've CRC v2.27 with Openshift v4.13.12.

I've been used that there is predefined about 30 PVs available (correct me if I'm wrong with 30). There were many (~30) PVs - not bounded yet free and few which I've used by PVCs with status Bounded. I was able to see them all using oc get pv or in CRC Console.

Today I realized that I can see there only 9 PVs - all bounded. So it means that all other PVs have disappeared. I have ther 4 additional PVCs waiting to be bounded but there is just this 9 PVs already used.

Is there a change in PVs management in latest CRC versions? Are there no default PVs available and I have to create them manually? Or any hint, any clue how the rest of PVs could disappear? If I did it somehow, how I can did this? Of course I did not expicitelly delete free PVs. Could not be the case that when data are allocated , CRC autiomatically reduce number of free PVs? Which direction should I investigate? How can I recover missing PVs?

Thanks a lot for any hint, Jiri

Expected

oc get pv will show 30 PVs (free and bounded)

Actual

oc get pv shows only 9 bounded PVs

Logs

Before gather the logs try following if that fix your issue

$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug

Please consider posting the output of crc start --log-level debug on http://gist.github.com/ and post the link in the issue.

[JiriHusak-lab]

by "free" PV I mean "Available"

[JiriHusak-lab]

Just thinking loudly. I've noticed that PVs in my CRC versions created after installation have: persistentVolumeReclaimPolicy: Delete

I'm sure that in CRC version I previously used there was "Retain" value. That would explain my question. Isn't it that in CRC version, where persistentVolumeReclaimPolicy: Delete, once I delete PVC bounded with PV, PV is deleted as well. That would explain why I have now 8 PVs instead of 30 Available after CRC installation. I tried to create PVs using Create Persistent Volume, using default Storage class crc-csi-hostpath-provisioner, but I failed. There is an error:

Danger alert:An error occurred Error "Required value: must specify a volume type" for field "spec".

Here is my yaml for PV creation:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: example
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: crc-csi-hostpath-provisioner
  volumeMode: Filesystem

How can I create back PVs to be like those created by default by CRC installation?

Thanks, Jiri

[praveenkumar]

@JiriHusak-lab do you really need persistentVolumeReclaimPolicy: Retain ? what I tried is very basic which doesn't need to create the pv resource but directly pvc and pv is created automatic by using the storage provisioner.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: topolvm-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: crc-csi-hostpath-provisioner
---
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  namespace: default
  labels:
    app.kubernetes.io/name: my-pod
    app: example
spec:
  containers:
  - name: pause
    image: registry.k8s.io/pause
    volumeMounts:
    - mountPath: /test1
      name: my-volume
    securityContext:
      allowPrivilegeEscalation: false
      runAsNonRoot: true
      seccompProfile:
        type: RuntimeDefault
      capabilities:
        drop:
        - ALL
  volumes:
  - name: my-volume
    persistentVolumeClaim:
      claimName: topolvm-pvc

[JiriHusak-lab]

For me, there are PVCs in Pending state when there are no Available PVs. Pending forewer. I just created PVC using part of your yaml and it is in status Pending. It will be pending until I will create PV. Storage Provisioner should be enabled by default after CRC installation? Is it something new in recent versions? It doesn't work on my CRC I see.

But is there a way how to create back default pre-provisioned PVs pv001-pv030? Simple yaml code for such pre-provisioned PV creating for CRC.

Thanks, Jiri

[praveenkumar]

I am surprised that is it not working because https://github.com/crc-org/snc/blob/master/registry_pvc.yaml is used for registry pvc and it works. can you try to describe the pending pvc because it might be something else than provisioner.

Is it something new in recent versions? It doesn't work on my CRC I see.

We have this storage provisioner from 4.12 so it is not something new, what was the last version of CRC works for you?

is there a way how to create back default pre-provisioned PVs pv001-pv030?

https://github.com/crc-org/snc/blob/release-4.10/snc-library.sh#L128-L164 is what we used before to create those PV as part of bundle generation.

[JiriHusak-lab]

Hi, CRC version I've used before this one had predefined persistent volumes (with reclaim policy RETAIN). It was version v4.10.3. . I've solved my issue as I've created new default storage class nfs and now PVs are created automatically. But true is, that after CRC installation, having just one storage class - default crc-csi-hostpath-provisioner - automatic creation of PVs doesn't work. Is there expected some customization or some post installation steps to have it working?

Thx, Jiri

[anjannath]

But true is, that after CRC installation, having just one storage class - default crc-csi-hostpath-provisioner - automatic creation of PVs doesn't work. Is there expected some customization or some post installation steps to have it working?

no post installation are required, it should be working automatically.

I'm sure that in CRC version I previously used there was "Retain" value. That would explain my question. Isn't it that in CRC version, where persistentVolumeReclaimPolicy: Delete, once I delete PVC bounded with PV, PV is deleted as well. That would explain why I have now 8 PVs instead of 30 Available after CRC installation.

@JiriHusak-lab to understand the issue better, you only see pvcs in pending after 8 pvs are created, as in the 8 pvcs that were bound to pvs did you create the pvs manually or it was automatically created by the provisioner?

i am trying to understand if for some reason the storage provisioner is not able to create any more after creating a certain number of pvs (due to running out of storage space maybe)