CrunchyData
Repo and Instance Pods Restarting: EmptyDir /tmp Usage Exceeds 16Mi in CrunchyData Operator 5.8.2 on OpenShift 4.17
Description: We are running OpenShift Container Platform 4.17 and have deployed the certified CrunchyData PostgreSQL Operator (PGO) version 5.8.2 from the OperatorHub.
After configuring a PostgresCluster custom resource, we observe repeated restarts of the instance and repo pods with the following event:
11s Warning Evicted Pod/pgo-cluster-instance1-8wqf-0 Usage of EmptyDir volume "tmp" exceeds the limit "16Mi". 10s Warning Evicted Pod/pgo-cluster-instance1-pw46-0 Usage of EmptyDir volume "tmp" exceeds the limit "16Mi". 7s Warning Evicted Pod/pgo-cluster-instance1-h597-0 Usage of EmptyDir volume "tmp" exceeds the limit "16Mi".
EXPECTED
This suggests that the emptyDir volume used for /tmp has a sizeLimit of only 16Mi, which is too small under normal operational load—especially during pgBackRest operations.
We would like to increase the emptyDir size limit for the /tmp volume used in the repo and instance pods. However, we do not find any configuration option in the CR (PostgresCluster) to control this size.
Is there a supported way to override or increase the emptyDir sizeLimit (e.g., to 512Mi)? If not currently supported, can this be added as a configurable field in the CRD? Is there a workaround we can use safely (e.g., patching StatefulSet or customizing templates)?
pgo-cluster-instance1-8wqf-0.log pgo-cluster-repo-host-0.log
Environment
Please provide the following details:
- Platform: (
Kubernetes,OpenShift,Rancher,GKE,EKS,AKSetc.) - Platform Version: (e.g.
1.20.3,4.7.0) - PGO Image Tag: (e.g.
ubi8-5.x.y-0) - Postgres Version (e.g.
15) - Storage: (e.g.
hostpath,nfs, or the name of your storage class)
Provide steps to get to the error condition:
[root@vmlcasocppmd001 logs]# oc get pod NAME READY STATUS RESTARTS AGE pgo-cluster-instance1-8wqf-0 0/4 Init:0/2 0 1s pgo-cluster-instance1-h597-0 3/4 Running 0 7s pgo-cluster-instance1-pw46-0 0/4 Init:1/2 0 4s pgo-cluster-repo-host-0 0/2 Init:0/2 0 6s
Logs
pod logs attached
Additional Information
Please provide any additional information that may be helpful.
@rakeshdeshmukh2018 are you able to see what is consuming all of the space in /tmp? Maybe logs, or something like that?
