Unable to create manual projects on 2.13.1

[fopson]

Please confirm the following

• [X] I agree to follow this project's code of conduct.
• [X] I have checked the current issues for duplicates.
• [X] I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

Unable to create manual project on AWX deployed through Operator. I have attempted to make /var/lib/awx/projects persistent and link it to /op/awx/projects locally on the node. I have followed instructions given in the community forum https://forum.ansible.com/t/awx-change-project-base-path-kubernetes/2473/8 but I still get the same error message.

I have been googling solutions for the past 2 days but no luck.

AWX Operator version

2.13.1

AWX version

24

Kubernetes platform

minikube

Kubernetes/Platform version

v1.32.0

Modifications

no

Steps to reproduce

Install AWX Operator 2.13.1

Setup Projects Dir

mkdir /opt/awx/projects
sudo chmod 775 /opt/awx/projects
sudo chown 1000:0 /opt/awx/projects

Create Persistent Volume

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: awx-projects-volume
spec:
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  capacity:
    storage: 2Gi
  storageClassName: awx-projects-volume
  hostPath:
    path: /opt/awx/projects

Create Persistent Volume Claim

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: awx-projects-claim
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 2Gi
  storageClassName: awx-projects-volume

Checks

Create TLS Secret

---
apiVersion: v1
kind: Secret
metadata:
  name: awx-tls-cert
  namespace: certificate
type: kubernetes.io/tls
data:
  tls.crt: <Base64-of-the-Cert>
  tls.key: <Base64-of-the-Key>

Deploy AWX

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
  namespace: awx
spec:
  # Service
  service_type: clusterip
  
  # Persisten Storage
  projects_persistence: true
  projects_existing_claim: awx-projects-claim

  # HTTPS
  ingress_type: ingress
  ingress_hosts:
    - hostname: fqdn-1
    - hostname: fqdn-2
      tls_secret: awx-tls-cert

Checks

Create a few projects under /opt/awx/projects/

Check

Expected results

Being able to create manual/persistent projects.

Actual results

AWX /var/lib/awx/projects/ doesn't seem to be linked to /opt/awx/projects/ on the local host.

Additional information

No response

Operator Logs

No response

[fosterseth]

your steps look correct to me

can you shell into your awx-task pod (awx-task container) and navigate to /var/lib/awx/projects? does the ls -la output look correct to you? if you touch foo in that directory do you see it on /opt/awx/projects?

[fopson]

Thanks for your comments @fosterseth.

Unfortunately, it looks like the linking of the volume isn't working as expected. A file listing of the /var/lib/awx/projects directory looks different than the file listing of the /opt/awx/projects directory.

I am not able to see the file created from within the pod.

[kurokobo]

The hostPath based PV on minikube is quite different from other distributions. Your PV refers /opt/awx/projects of your minikube VM (or docker container) instead of your minikube host. Please investigate inside VM (or docker container) that is used as minikube instance.

Also, /opt on minikube instance is not persisted. Refer to: https://minikube.sigs.k8s.io/docs/handbook/persistent_volumes/