ansible
[init-database] - Issues to connect to the database when workload is meshed using Istio
Please confirm the following
- [X] I agree to follow this project's code of conduct.
- [X] I have checked the current issues for duplicates.
- [X] I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.
Bug Summary
With the feature introduced by the https://github.com/ansible/awx-operator/commit/ffba1b4712a0b03f1faedfa70e3a9ef0d443e4a6, a InitContainer named init-database was introduced to validate the database status by calling the wait-for-migrations script. I really liked this approach though, so we have a dedicated task for it.
Therefore, if the workload/namespace is meshed with Istio, the initContainer requires an special UID to be captured by the Istio, otherwise the initContainer will not be able to reach out to the database. Note that this issue does not happen with a regular pod, as the sidecar will get injected as expected, whereas on the initContainer there is no sidecar yet.
To fix the issue, we need to expose to this deployment the ability to override the securityContext to runAsUser.
To address this issue, I stopped the operator (to avoid reconciliation) and manually patch the awx-task deployment which then works as expected.
❯ kubectl get deploy awx-toca-task -o json | jq '.spec.template.spec.initContainers.[0].securityContext'
{
"runAsUser": 1337
}
❯ kubectl images -n awx
[Summary]: 1 namespaces, 3 pods, 15 containers and 4 different images
+---------------------------------+-------------------------+--------------------------------+
| Pod | Container | Image |
+---------------------------------+-------------------------+--------------------------------+
| awx-toca-migration-24.2.0-kg4m6 | migration-job | quay.io/ansible/awx:24.2.0 |
+---------------------------------+-------------------------+--------------------------------+
| awx-toca-task-5bdc6ff596-xsj4z | redis | docker.io/redis:7 |
+ +-------------------------+--------------------------------+
| | awx-toca-task | quay.io/ansible/awx:24.2.0 |
+ +-------------------------+--------------------------------+
| | awx-toca-ee | quay.io/ansible/awx-ee:24.2.0 |
+ +-------------------------+--------------------------------+
| | awx-toca-rsyslog | quay.io/ansible/awx:24.2.0 |
+ +-------------------------+--------------------------------+
| | istio-proxy | docker.io/istio/proxyv2:1.21.1 |
+ +-------------------------+ +
| | (init) istio-validation | |
+ +-------------------------+--------------------------------+
| | (init) init-database | quay.io/ansible/awx:24.2.0 |
+ +-------------------------+--------------------------------+
| | (init) init-receptor | quay.io/ansible/awx-ee:24.2.0 |
+---------------------------------+-------------------------+--------------------------------+
| awx-toca-web-6f4657cbfb-gzhz5 | redis | docker.io/redis:7 |
+ +-------------------------+--------------------------------+
| | awx-toca-web | quay.io/ansible/awx:24.2.0 |
+ +-------------------------+ +
| | awx-toca-rsyslog | |
+ +-------------------------+--------------------------------+
| | istio-proxy | docker.io/istio/proxyv2:1.21.1 |
+ +-------------------------+ +
| | (init) istio-validation | |
+ +-------------------------+--------------------------------+
| | (init) init | quay.io/ansible/awx-ee:24.2.0 |
+---------------------------------+-------------------------+--------------------------------+
❯ kubectl get pods -n awx
NAME READY STATUS RESTARTS AGE
awx-toca-migration-24.2.0-kg4m6 0/1 Completed 0 61m
awx-toca-task-5bdc6ff596-xsj4z 5/5 Running 0 50m
awx-toca-web-6f4657cbfb-gzhz5 4/4 Running 0 53m
AWX Operator version
2.15.0
AWX version
24.2.0
Kubernetes platform
kubernetes
Kubernetes/Platform version
1.29
Modifications
yes
Steps to reproduce
- Mesh the namespace using Istio
- Deploy workload
Expected results
Work as expect
Actual results
Fails to connect to the database due to initContainer being unable to reach out to the network.
Additional information
N/A
Operator Logs
N/A
### Tasks
+1 just ran into this same issue today, task never comes online due to init containers firing before the istio sidecar is online.
Adding support for task_label to set sidecar.istio.io/inject to false could be a solution. There is a helm variable task_annotations for annotations but using annotation is deprecated https://istio.io/latest/docs/reference/config/annotations/#SidecarInject
I don't think thats a reasonable solution as our end goal here is to run istio successfully while using awx-operator, which we can do now we just have to pause controller enforcement to do so until a proper patch is released.
A more proper solution is to create inject.istio.io/templates in istiod that allows the injection of whatever you need in awx-task or any other object. (so you can inject securityContext even with controller enforcement)
And than use task_label to add the new custom label template to force the injection in awx-task pod.
https://github.com/istio/istio/blob/master/manifests/charts/istio-control/istio-discovery/values.yaml#L137-L162 https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/#custom-templates-experimental
