Consul inject side car fails for rabbitmq server in Openshift 4.11

[adaumann]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview

The consul inject sidecar init fails in Rabbit MQ. Using the RabbitMQ Cluster Operator in Openshift. If disabling the connectInject rabbitMQ works

Reproduction Steps

• Using Openshift 4.11 CRC
• Installing RabbitMQ Cluster Operator via Operator hub with security context information. https://www.rabbitmq.com/kubernetes/operator/using-on-openshift.html
• Rabbit MQ runs in privileged mode
• Installing Consul 1.12.0 and Consul operator 0.43 (Any version higher does not work at all in Openshift, see other issue)

Use following values:

´´´´ consul: global: name: consul datacenter: hcv-dc1 # Only this combination is working, 0.47.1 and 0.48 not working to start in insecure container. Secure container never possible image: registry.connect.redhat.com/hashicorp/consul:1.12.0-ubi imageK8S: registry.connect.redhat.com/hashicorp/consul-k8s-control-plane:0.43.0-ubi imagePullSecrets: - name: xxxxxxxxxxx-hcv-redhat-registry-pull-secret openshift: enabled: true

server: replicas: 1 bootstrapExpect: 1 disruptionBudget: enabled: true maxUnavailable: 0

client: enabled: true grpc: true

ui: enabled: true

connectInject: enabled: true default: true

controller: enabled: true ´´´´

Logs

• rabbitmq-cluster-server-0 : Failed to start because cluster-consul-connect-init not woking

´´´´

2022-09-16T08:37:25.697Z [INFO] Unable to find registered services; retrying 2022-09-16T08:37:25.697Z [INFO] Check to ensure a Kubernetes service has been created for this application. If your pod is not starting also check the connect-inject deployment logs. 2022-09-16T08:37:26.698Z [INFO] Unable to find registered services; retrying 2022-09-16T08:37:27.700Z [INFO] Unable to find registered services; retrying 2022-09-16T08:37:28.702Z [INFO] Unable to find registered services; retrying 2022-09-16T08:37:29.702Z [INFO] Unable to find registered services; retrying 2022-09-16T08:37:30.703Z [INFO] Unable to find registered services; retrying 2022-09-16T08:37:31.705Z [INFO] Unable to find registered services; retrying ´´´´

--->

Expected behavior

The consul connect init container should not fail in rabbitmq context

[david-yu]

Hi @adaumann thanks for filing this issue. It sounds like somehow RabbitMQ requires to be run in a privileged mode and maybe that's why you are not able to Consul K8s working on K8s? I also am curious does the RabbitMQ cluster-server run as a K8s service when using the Rabbit MQ Operator. I don't we're familiar enough with RabbitMQ to get into the details but from a Consul K8s perspective we do require a Service object for deployments you are trying to run on the mesh.

[david-yu]

I believe the other issue you will likely run into is that RabbitMQ servers currently require mluit-port services: https://www.rabbitmq.com/clustering.html#ports. Please see https://www.rabbitmq.com/clustering.html#ports about how to allow communication between services with multiple ports exposed.

[ishustava]

Hey @adaumann

Could you share the logs of the connect-inject deployment like the logs of the init container suggest?

[darkn3rd]

@adaumann RabbitMQ uses multiple ports, so likely the source of the problem is a single service resource that is listening on multiple ports. One way you determine this is is kubectl get svc --namespace rabbitmq (change the namespace if rabbitmq is deployed to another namespace). Without this, it will be hard for anyone to know what to advise.

@david-yu another multi-port use case. If this was pure helm, one could use kustomize to add needed complex patches required. However, with operators, this poses a unique challenges. Also, for multi-port scenarios, can anything be emitted in the logs to help the user more easily identify the issue?