awx-operator icon indicating copy to clipboard operation
awx-operator copied to clipboard

Published 20 hours ago • verified publisher icon ansible

GUI not working through nginx proxy

Open AndreasDavour opened this issue 3 years ago • 9 comments

ISSUE TYPE
  • Bug Report
SUMMARY

When accessing the GUI through a nginx proxy, I can't get to the GUI. I only get a page saying "Not Found the requested resource could not be found" and a Red Hat copyright message, which suggests that it is indeed awx that answers.

ENVIRONMENT

AWX version: 19.2.1 Operator version: 0.11.0 AWX install method: operator installation on rancher

STEPS TO REPRODUCE

Install kubernetes, install operator, install external nginx with provided config and try to access the gui at hostname.domain.com/awx

My awx environment and the exposed NodePort:

kubectl get all -n default

NAME READY STATUS RESTARTS AGE pod/awx-demo-8ff8f94d7-b5d6z 4/4 Running 0 10d pod/awx-demo-postgres-0 1/1 Running 0 10d pod/awx-operator-765db9c478-ww5bm 1/1 Running 1 10d

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/awx-demo-postgres ClusterIP None 5432/TCP 10d service/awx-demo-service NodePort 10.43.83.13 80:30468/TCP 10d service/awx-operator-metrics ClusterIP 10.43.88.15 8383/TCP,8686/TCP 10d service/kubernetes ClusterIP 10.43.0.1 443/TCP 13d

NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/awx-demo 1/1 1 1 10d deployment.apps/awx-operator 1/1 1 1 10d

NAME DESIRED CURRENT READY AGE replicaset.apps/awx-demo-54f8fcc8d5 0 0 0 10d replicaset.apps/awx-demo-578b7d9566 0 0 0 10d replicaset.apps/awx-demo-6ccb56b655 0 0 0 10d replicaset.apps/awx-demo-7d8469b85c 0 0 0 10d replicaset.apps/awx-demo-8ff8f94d7 1 1 1 10d replicaset.apps/awx-operator-765db9c478 1 1 1 10d

NAME READY AGE statefulset.apps/awx-demo-postgres 1/1 10d

My nginx config:

http { upstream awx { server 10.104.129.171:30468; } server { listen 80; location /awx {
proxy_pass http://awx; } } }

EXPECTED RESULTS

To see the GUI

ACTUAL RESULTS

An error page indicating missing resources

ADDITIONAL INFORMATION

The log from the container running the web say this interesting tidbit, which to me indicate the web request is missing something:

2021-07-02 12:03:58,200 DEBUG [97eec0e818d64ec29c51c357103b83c3] awx.analytics.performance request: <WSGIRequest: GET '/awx/'>, response_time: 0.057s 2021-07-02 12:03:58,201 WARNING [97eec0e818d64ec29c51c357103b83c3] django.request Not Found: /awx/ 2021-07-02 12:03:58,201 WARNING [97eec0e818d64ec29c51c357103b83c3] django.request Not Found: /awx/ 10.42.3.0 - - [02/Jul/2021:12:03:58 +0000] "GET /awx/ HTTP/1.1" 404 3860 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "10.104.137.2" [pid: 28|app: 0|req: 337/1178] 10.42.3.0 () {52 vars in 854 bytes} [Fri Jul 2 12:03:58 2021] GET /awx/ => generated 3860 bytes in 58 msecs (HTTP/1.1 404) 8 headers in 387 bytes (1 switches on core 0) RESULT 2 OKREADY

To me it looks like nginx forwards a request that is missing something, but what? I guess this is more about nginx than awx, but if I ask someone who knows nginx and not awx they will have no idea what awx expects at this request...

AWX-OPERATOR LOGS

AndreasDavour avatar Jul 02 '21 12:07 AndreasDavour

can you share Nginx configuration, are you using ssl certs

Nitish0292 avatar Nov 11 '21 12:11 Nitish0292

also use serviceurl.com/ip:portnumber

Nitish0292 avatar Nov 11 '21 12:11 Nitish0292

As nobody was responding to this issue I tore it all up. I have no energy to try to reproduce this.

AndreasDavour avatar Nov 14 '21 20:11 AndreasDavour

Hi @AndreasDavour, I think @Nitish0292 did ask for some precisions on your setup to assist (I am interested in getting more info too).

vrubiolo avatar Nov 15 '21 09:11 vrubiolo

Sorry, I might have been unclear.

It was an installation attempt I tried out, and I did not manage to find the solution myself until any reply here.

It was time dependent, but not critical to get it to work, so it was nuked, and thus I no longer have the setup available for further debugging. We basically gave up on it, with the time frame we had.

You might note this was back in June, and no feedback until 5 days ago. Not a complaint, just an observation.

AndreasDavour avatar Nov 16 '21 11:11 AndreasDavour

Hi @AndreasDavour , thanks for the feedback here. No worries, I thought you had missed @Nitish0292's reply

vrubiolo avatar Nov 16 '21 13:11 vrubiolo

Hi @vrubiolo i have followed below configuration for ssl and reverse proxy and everything works well

server{ listen 443 ssl http2; server_name server.com; server_tokens off; ssl_certificate /etc/nginx/certs/ansible.crt; ssl_certificate_key /etc/nginx/certs/ansible.key; ssl on; ssl_session_cache builtin:1000 shared:SSL:10m; ssl_protocols TLSv1.3; ssl_ciphers "HIGH !aNULL !eNULL !EXPORT !CAMELLIA !DES !MD5 !PSK !RC4"; ssl_prefer_server_ciphers on; access_log /var/log/nginx/ansible.1og; location / { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme;

 proxy_pass       http://192.168.49.2:30687;
   }

if ($request_method !— ^(GET|HEAD|POST)$ ) { return 405; } }

Nitish0292 avatar Nov 16 '21 13:11 Nitish0292

I also have this problem.

I followed through with the installation as outlined in current README.md but with the slight difference of using microk8s instead of minikube. As far as my limited knowledge is concerned, there shouldn't be any difference.

I'm using version 0.17.0 of this repo and ran make deploy to install the operator.

The following pods are running (as they should):

root@awx:~/cfg# kubectl get pods
NAME                                              READY   STATUS    RESTARTS   AGE
awx-operator-controller-manager-98cb6f594-xhnkk   2/2     Running   0          71m
awx-demo-postgres-0                               1/1     Running   0          69m
awx-demo-7bc4f98579-n72pm                         4/4     Running   0          69m

The services are also present (as they should):

root@awx:~/cfg# kubectl get svc
NAME                                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
awx-operator-controller-manager-metrics-service   ClusterIP   10.152.183.213   <none>        8443/TCP       72m
awx-demo-postgres                                 ClusterIP   None             <none>        5432/TCP       70m
awx-demo-service                                  NodePort    10.152.183.75    <none>        80:32505/TCP   69m

I tried to verify my installation by running awx-manage version inside the awx-demo-task of pod awx-demo-7bc4f98579-n72pm:

root@awx:~/cfg# kubectl exec -it awx-demo-7bc4f98579-n72pm -c awx-demo-task -- /bin/bash
bash-5.1$ awx-manager version
bash: awx-manager: command not found

So to me there seems to be a problem with the image running in the awx-demo-task container and maybe also in the other awx-demo-* containers of my awx-demo-7bc4f98579-n72pm pod.

Maybe a quick kubectl describe can help debugging:

root@awx:~/cfg# kubectl describe pod awx-demo-7bc4f98579-n72pm
Name:         awx-demo-7bc4f98579-n72pm
Namespace:    awx
Priority:     0
Node:         awx/195.201.126.58
Start Time:   Wed, 16 Feb 2022 12:28:02 +0000
Labels:       app.kubernetes.io/component=awx
              app.kubernetes.io/managed-by=awx-operator
              app.kubernetes.io/name=awx-demo
              app.kubernetes.io/part-of=awx-demo
              app.kubernetes.io/version=20.0.0
              pod-template-hash=7bc4f98579
Annotations:  cni.projectcalico.org/podIP: 10.1.146.71/32
              cni.projectcalico.org/podIPs: 10.1.146.71/32
Status:       Running
IP:           10.1.146.71
IPs:
  IP:           10.1.146.71
Controlled By:  ReplicaSet/awx-demo-7bc4f98579
Init Containers:
  database-check:
    Container ID:  containerd://9dcf6ea7eee9bdf4fd4a70de2d13a47d86d9d6cc5bf05d3e96e3dc4af3d20997
    Image:         quay.io/centos/centos:8
    Image ID:      quay.io/centos/centos@sha256:ec16f510e8cd1cceeddc6888fcb37c6d6b7c6863fbc70bdbe660a8ccad4bab6a
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      [[ -d /check-db/pgsql/data ]] && rm -rf /check-db/data && mv /check-db/pgsql/data/ /check-db/data/ && rm -rf /check-db/pgsql || true

    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 16 Feb 2022 12:28:09 +0000
      Finished:     Wed, 16 Feb 2022 12:28:09 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /check-db from check-db-pvc (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bzr7d (ro)
Containers:
  redis:
    Container ID:  containerd://e0c8eb1d7a90282b2a48defafc240881a72deaa99bed3bf69bf96fef02041770
    Image:         docker.io/redis:latest
    Image ID:      docker.io/library/redis@sha256:0d9c9aed1eb385336db0bc9b976b6b49774aee3d2b9c2788a0d0d9e239986cb3
    Port:          <none>
    Host Port:     <none>
    Args:
      redis-server
      /etc/redis.conf
    State:          Running
      Started:      Wed, 16 Feb 2022 12:28:14 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /data from awx-demo-redis-data (rw)
      /etc/redis.conf from awx-demo-redis-config (ro,path="redis.conf")
      /var/run/redis from awx-demo-redis-socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bzr7d (ro)
  awx-demo-web:
    Container ID:   containerd://44ca5ba446b2cf574656c5c1c49fec460fff87bad2d66996f9b6148ab550b991
    Image:          quay.io/ansible/awx:20.0.0
    Image ID:       quay.io/ansible/awx@sha256:1af0d56bda7ae6e8ff69046d5833ea14669ea8c12ac1a3bef88c98722ff1afda
    Port:           8052/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Wed, 16 Feb 2022 12:28:30 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     1
      memory:  2Gi
    Environment:
      MY_POD_NAMESPACE:  awx (v1:metadata.namespace)
      UWSGI_MOUNT_PATH:  /
    Mounts:
      /etc/nginx/nginx.conf from awx-demo-nginx-conf (ro,path="nginx.conf")
      /etc/tower/SECRET_KEY from awx-demo-secret-key (ro,path="SECRET_KEY")
      /etc/tower/conf.d/credentials.py from awx-demo-application-credentials (ro,path="credentials.py")
      /etc/tower/conf.d/execution_environments.py from awx-demo-application-credentials (ro,path="execution_environments.py")
      /etc/tower/conf.d/ldap.py from awx-demo-application-credentials (ro,path="ldap.py")
      /etc/tower/settings.py from awx-demo-settings (ro,path="settings.py")
      /var/lib/awx/projects from awx-demo-projects (rw)
      /var/lib/awx/rsyslog from rsyslog-dir (rw)
      /var/run/awx-rsyslog from rsyslog-socket (rw)
      /var/run/redis from awx-demo-redis-socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bzr7d (ro)
      /var/run/supervisor from supervisor-socket (rw)
  awx-demo-task:
    Container ID:  containerd://e08b18010aa1d2d9ac8d3f1569ffa779db9d618b78ecd2c37d17a43ca9145260
    Image:         quay.io/ansible/awx:20.0.0
    Image ID:      quay.io/ansible/awx@sha256:1af0d56bda7ae6e8ff69046d5833ea14669ea8c12ac1a3bef88c98722ff1afda
    Port:          <none>
    Host Port:     <none>
    Args:
      /usr/bin/launch_awx_task.sh
    State:          Running
      Started:      Wed, 16 Feb 2022 12:28:31 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     500m
      memory:  1Gi
    Environment:
      SUPERVISOR_WEB_CONFIG_PATH:  /etc/supervisord.conf
      AWX_SKIP_MIGRATIONS:         1
      MY_POD_UID:                   (v1:metadata.uid)
      MY_POD_IP:                    (v1:status.podIP)
      MY_POD_NAMESPACE:            awx (v1:metadata.namespace)
    Mounts:
      /etc/receptor/receptor.conf from awx-demo-receptor-config (ro,path="receptor.conf")
      /etc/tower/SECRET_KEY from awx-demo-secret-key (ro,path="SECRET_KEY")
      /etc/tower/conf.d/credentials.py from awx-demo-application-credentials (ro,path="credentials.py")
      /etc/tower/conf.d/execution_environments.py from awx-demo-application-credentials (ro,path="execution_environments.py")
      /etc/tower/conf.d/ldap.py from awx-demo-application-credentials (ro,path="ldap.py")
      /etc/tower/settings.py from awx-demo-settings (ro,path="settings.py")
      /var/lib/awx/projects from awx-demo-projects (rw)
      /var/lib/awx/rsyslog from rsyslog-dir (rw)
      /var/run/awx-rsyslog from rsyslog-socket (rw)
      /var/run/receptor from receptor-socket (rw)
      /var/run/redis from awx-demo-redis-socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bzr7d (ro)
      /var/run/supervisor from supervisor-socket (rw)
  awx-demo-ee:
    Container ID:  containerd://a2257ba85417389b6db69e80a6aee357553e65a6a27179e3a2d1e53595ff53b1
    Image:         quay.io/ansible/awx-ee:latest
    Image ID:      quay.io/ansible/awx-ee@sha256:2b82b3dd55d6f7a6d2e1c727532c267d9b8ce20ec03120d3b617e259d42c185f
    Port:          <none>
    Host Port:     <none>
    Args:
      receptor
      --config
      /etc/receptor/receptor.conf
    State:          Running
      Started:      Wed, 16 Feb 2022 12:29:03 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        500m
      memory:     1Gi
    Environment:  <none>
    Mounts:
      /etc/receptor/receptor.conf from awx-demo-receptor-config (ro,path="receptor.conf")
      /var/lib/awx/projects from awx-demo-projects (rw)
      /var/run/receptor from receptor-socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bzr7d (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  check-db-pvc:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  postgres-awx-demo-postgres-0
    ReadOnly:   false
  awx-demo-application-credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  awx-demo-app-credentials
    Optional:    false
  awx-demo-secret-key:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  awx-demo-secret-key
    Optional:    false
  awx-demo-settings:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      awx-demo-awx-configmap
    Optional:  false
  awx-demo-nginx-conf:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      awx-demo-awx-configmap
    Optional:  false
  awx-demo-redis-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      awx-demo-awx-configmap
    Optional:  false
  awx-demo-redis-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  awx-demo-redis-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  supervisor-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  rsyslog-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  receptor-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  rsyslog-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  awx-demo-receptor-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      awx-demo-awx-configmap
    Optional:  false
  awx-demo-projects:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-bzr7d:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

This is the log of the manager obtained by running

kubectl logs deployments/awx-operator-controller-manager -c awx-manager > ~/awx-operator-controller-manager.log

awx-operator-controller-manager.log

jankal avatar Feb 16 '22 13:02 jankal

In my case, debugging container and service response yields a different response than the response of <host-ip>:80 even if it should be the same as the awx-demo-7bc4f98579-n72pm as a NodePort service should be listening there. But it seems, some kind of ingress of microk8s was listening there. I think the ingress was the source of the 404 error in my case.

The only remaining problem I seem to have is the django web-app not being aple to connect to it's database:

psycopg2.OperationalError: could not translate host name "awx-demo-postgres" to address: Name or service not known

I have to debug a little more.

jankal avatar Feb 16 '22 14:02 jankal