[BUG]: Issue while Configuring Authorization module with Powermax CSI Driver using Operator.

[Kaustubh12121]

Bug Description

config-yaml-file.txt

The Powermax CSI drivers pods with Authorization module enabled are going in CrashLoopBackoff State. Apparently, the bug is related to config map "powermax-reverseproxy-config".

The localhost endpoint i.e (https://localhost:9400) on which authorization sidecar listen on, is not working with CSI driver pods of powermax.

Sample config file of "powermax-reverseproxy-config" is pasted above.

Errors:-

[root@] oc get pods NAME READY STATUS RESTARTS AGE powermax-controller-65f5984df5-jd5cs 2/7 CrashLoopBackOff 5750 (20s ago) 4d9h powermax-controller-65f5984df5-pgsmt 2/7 CrashLoopBackOff 5749 (13s ago) 4d9h powermax-node-6z4gp 2/3 CrashLoopBackOff 1773 (80s ago) 4d9h powermax-node-c7gfc 2/3 CrashLoopBackOff 1771 (2m40s ago) 4d9h powermax-node-z2c2l 2/3 CrashLoopBackOff 1772 (27s ago) 4d9h

$ oc logs -f powermax-controller-65f5984df5-jd5cs -c driver

time="2024-04-07T19:58:11Z" level=error msg="Failed to fetch details for array: <storage_id>. [Bad Gateway]" time="2024-04-07T19:58:11Z" level=error msg="None of the managed arrays specified are locally connected"

$ oc logs -f powermax-controller-65f5984df5-jd5cs -c karavi-authorization-proxy

time="2024-04-03T10:49:30Z" level=info msg="main: starting sidecar-proxy" time="2024-04-03T10:49:30Z" level=info msg="configuration has been set" CSI_LOG_FORMAT=TEXT time="2024-04-03T10:49:30Z" level=info msg="configuration has been set" CSI_LOG_LEVEL=debug time="2024-04-03T10:49:30Z" level=info msg="configuration has been set" CSI_LOG_LEVEL=debug time="2024-04-03T10:49:31Z" level=info msg="main: config: " endpoint="https://localhost:9400" insecure=true intendedendpoint="<storage_endpoint>" isDefault=true password="********" systemID=<storage_id> username= time="2024-04-03T10:49:31Z" level=info msg="Listening on :9400"

Logs

bug_logs.txt ---> Full Logs csi-driver-yaml-file.txt ---> CSI driver yaml file config-yaml-file.txt ---> powermax-reverseproxy-config

Screenshots

No response

Additional Environment Information

No response

Steps to Reproduce

csi-driver-yaml-file.txt

$ cat powermax-creds.yaml

apiVersion: v1 data: username: <username_of_storage_array> password: <password_of_storage_array> kind: Secret metadata: name: powermax-creds namespace: powermax type: Opaque

$ oc create -f powermax-creds.yaml

$ cat config.yaml

mode: StandAlone port: 2222 logLevel: debug logFormat: text standAloneConfig: storageArrays: - storageArrayId: "<storage_id>" primaryURL: https://localhost:9400 proxyCredentialSecrets: - powermax-creds managementServers: - url: https://localhost:9400 arrayCredentialSecret: powermax-creds skipCertificateValidation: true

$ kubectl create configmap powermax-reverseproxy-config --from-file config.yaml -n powermax

$ openssl genrsa -out tls.key 2048 $ openssl req -new -x509 -sha256 -key tls.key -out tls.crt -days 3650 $ kubectl create secret -n powermax tls revproxy-certs --cert=tls.crt --key=tls.key $ kubectl create secret -n powermax tls csirevproxy-tls-secret --cert=tls.crt --key=tls.key

$ cat karavi-authorization-config.json [{"username":"","password":"","intendedEndpoint":"<storage_endpoint>","endpoint":"https://localhost:9400","systemID":"<storage_array_id>","insecure":true,"isDefault":true}]

$ kubectl -n powermax create secret generic karavi-authorization-config --from-file=config=karavi-authorization-config.json -o yaml --dry-run=client | kubectl apply -f -

$ kubectl -n powermax create secret generic proxy-server-root-certificate --from-literal=rootCertificate.pem= -o yaml --dry-run=client | kubectl apply -f -

$ oc apply -f token.yaml -n powermax

NOTE :- Driver Yaml has been pasted above.

$ oc create -f storage_csm_powermax_v2100.yaml

Expected Behavior

Expected Behavior should be that :-

1. Powermax Pods should be in up and Running State.
2. Authorization Sidecar container should listen on https://localhost:9400.

CSM Driver(s)

CSM Operator Version is 1.5 Config Version of Powermax CSI driver is v2.10.0 Authorization Module Version is v1.10.0

Installation Type

Operator v1.5

Container Storage Modules Enabled

Authorization module is enabled in CSI Driver yaml file.

Container Orchestrator

OpenShift v4.14.14

Operating System

RHEL 9.2

[csmbot]

@Kaustubh12121: Thank you for submitting this issue!

The issue is currently awaiting triage. Please make sure you have given us as much context as possible.

If the maintainers determine this is a relevant issue, they will remove the needs-triage label and respond appropriately.

---

We want your feedback! If you have any questions or suggestions regarding our contributing process/workflow, please reach out to us at [email protected].

[atye]

Hi @Kaustubh12121

The Authorization Server/Module seems to be good given the output in bug_logs.txt. All of the pods are running, services are available, and I see the ingress resources.

The karavi-authorization-proxy container logs will log every request it receives from the driver container so since we see no logs there, that means that requests are not reaching the proxy. Could we see the logs of the csipowermax-reverseproxy container? Perhaps there are errors there.

[Kaustubh12121]

Hi @Kaustubh12121

The Authorization Server/Module seems to be good given the output in bug_logs.txt. All of the pods are running, services are available, and I see the ingress resources.

The karavi-authorization-proxy container logs will log every request it receives from the driver container so since we see no logs there, that means that requests are not reaching the proxy. Could we see the logs of the csipowermax-reverseproxy container? Perhaps there are errors there.

Hi atye

This is the logs of csipowermax reverseproxy.

$ oc logs -f csipowermax-reverseproxy-c755bd6f8-gmrtk time="2024-04-08T13:54:03Z" level=info msg="Request ID: - Read Lock time: 24.69µs" time="2024-04-08T13:54:03Z" level=debug msg="Request ID: - Reporing server error to proxy health" time="2024-04-08T13:54:03Z" level=info msg="Request ID: - GET https://localhost:9400/univmax/restapi/version\n time: 484.405µs" time="2024-04-08T13:54:03Z" level=error msg="Authorisation step fails for: (000297901797) symID with error (Get "https://localhost:9400/univmax/restapi/version": dial tcp [::1]:9400: connect: connection refused)" time="2024-04-08T13:54:03Z" level=info msg="Lock: https://localhost:9400-Read, Active(0/5), Queued(0/50)" time="2024-04-08T13:54:03Z" level=info msg="Request ID: 40 - GET /univmax/restapi/100/system/symmetrix/000297901797" time="2024-04-08T13:54:03Z" level=info msg="Lock: https://localhost:9400-Read, Active(1/5), Queued(0/50)" time="2024-04-08T13:54:03Z" level=info msg="Request ID: 40 - Obtained Read lock" time="2024-04-08T13:54:03Z" level=info msg="Request ID: 40 - Read Lock time: 16.12µs" time="2024-04-08T13:54:03Z" level=debug msg="Request ID: 40 - Reporing server error to proxy health" 2024/04/08 13:54:03 http: proxy error: dial tcp [::1]:9400: connect: connection refused time="2024-04-08T13:54:03Z" level=info msg="Request ID: 40 - Unisphere RESTAPI response time: 413.984µs" time="2024-04-08T13:54:03Z" level=info msg="Request ID: 40 - Total time: 444.664µs" time="2024-04-08T13:54:03Z" level=info msg="Lock: https://localhost:9400-Read, Active(0/5), Queued(0/50)"

Here, I have given local endpoint as https://localhost:9400, as this was mentioned in the document.

[atye]

/sync

[Kaustubh12121]

Hi atye

If possible, can we connect over a remote session? Or Did you got something on this issue?

[atye]

For now, we have enough info to begin an investigation and reproduce the issue, investigate a fix, etc. The issue owner will reach out when we have an update.

[csmbot]

link: 22923

[boyamurthy]

link: 22771

[soudamsugit]

Hello Team,

is there any further udpate on this case?