connect: Communication between pods of the same consul service

[Poweranimal]

Question

Hi there,

How can two or more Pods that belong to the same consul connect service communicate between each other?

Example: StatefulSet with 2 replicas, connect-inject enabled and transparent proxy enabled. The consul registered port is 8000. pod-0 sends a request to pod-1 (e.g. wget -qO- http://pod-1.pod.default:8000).

Context: StatefulSet services like minio have a gossip communication between their Pods on the consul registered service port. For that reason they must be able to communicate between each other.

Example Setup

1. Deploy consul with connect enabled and transparent proxy enabled.
2. Deploy the k8s manifest defined below
3. Run: kubectl exec nginx-0 -- curl -v http://nginx-1.nginx.default:8000

Expected Result: The curl command should return Hello world!

Actual Result: The curl command returns an empty response.

k8s Manifest

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: nginx
spec:
  protocol: http
  transparentProxy:
    dialedDirectly: true
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 8000
      targetPort: 8000
  selector:
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/name: nginx
    component: nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx
data:
  default.conf.template: |
    server {
      listen  8000;
      location / {
        return 200 'Hello world!';
        add_header Content-Type text/plain;
      }
    }
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/instance: nginx
      app.kubernetes.io/name: nginx
      component: nginx
  serviceName: nginx
  replicas: 2
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      name: nginx
      annotations:
        consul.hashicorp.com/connect-inject: "true"
      labels:
        app.kubernetes.io/instance: nginx
        app.kubernetes.io/name: nginx
        component: nginx
    spec:
      serviceAccountName: nginx
      containers:
        - name: nginx
          image: docker.io/nginx:1.21.6
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8000
              name: http
              protocol: TCP
          volumeMounts:
            - mountPath: /etc/nginx/templates
              name: config
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: nginx

[david-yu]

Hi @Poweranimal, We do have a field within the the ServiceDefaults CRD called DialedDirectly (link) which allows you to dial the proxies directly for a given Service. Could you try setting the DialedDirectly config to true?

[Poweranimal]

Hi @david-yu , I enabled this mode. However, this only seems to allow direct IP address dialing from another consul service to the target consul service. It seems to have no effect when it comes to communication between Pods that belong to the same consul service.

[david-yu]

Apologies, I had neglected to glance at the first part of your k8s manifest as I did not expand it. One thing to try is also to use some annotations on the pods themselves to exclude traffic redirection on certain ports. In this it seems you may want to exclude traffic re-direction based on inbound request to port 8000.

https://www.consul.io/docs/connect/transparent-proxy#traffic-redirection-configuration

template:
    metadata:
      name: nginx
      annotations:
        consul.hashicorp.com/connect-inject: "true"
        consul.hashicorp.com/transparent-proxy-exclude-inbound-ports: "8000"

[Poweranimal]

Though this would also bypass features like mTLS and ACL. I’d like to keep my traffic secured by the mTLS of consul connect. I guess services like minio or redis in HA mode might share confidential data between each other that should be protected by TLS encryption.

[Poweranimal]

@david-yu I assume my presented use case is not supported by consul connect?

[whiskeysierra]

I've been told by our support manager that this isn't supported right now.

Our use case is actually HashiCorp Vault, which also needs to securely talk to itself (to proxy requests to the current master).

Same applies to Redis in sentinel mode.