K8SSAND-1783 ⁃ Encrypted setup not accesible from outside the cluster

[stdmje]

What happened? I followed the steps described here for encrypting the cluster and after facing some problems (see here and here) the encryption setup is working.

Now i tried to expose Stargate outside of the cluster using a IngressRouteTCP using the following example but i can't reach Stargate.

Did you expect to see something different? I was expecting to reach Stargate using cqlsh

How to reproduce it (as minimally and precisely as possible): Deploy the provided manifests.

Environment

• K8ssandra Operator version: 1.2.0
• Kubernetes version information: EKS GitVersion:"v1.22.11-eks-18ef993
• Manifests:

k8ssandracluster

apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
metadata:
  name: demo
spec:
  stargate:
    size: 1
    heapSize: 256M
  cassandra:
    serverEncryptionStores:
      keystoreSecretRef:
        name: server-encryption-stores
      truststoreSecretRef:
        name: server-encryption-stores
    clientEncryptionStores:
      keystoreSecretRef:
        name: client-encryption-stores
      truststoreSecretRef:
        name: client-encryption-stores
    serverVersion: "4.0.5"
    storageConfig:
      cassandraDataVolumeClaimSpec:
        storageClassName: gp3
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
    config:
      jvmOptions:
        heapSize: 1024M
        additionalOptions:
        - -Djavax.net.debug=all
      cassandraYaml:
        server_encryption_options:
          internode_encryption: all
          require_client_auth: true
        client_encryption_options:
          enabled: true
          require_client_auth: true
          protocol: TLS
    datacenters:
      - metadata:
          name: dc1
        size: 3
        racks:
        - name: eu-west-1a
          nodeAffinityLabels:
            topology.kubernetes.io/zone: eu-west-1a
        - name: eu-west-1b
          nodeAffinityLabels:
            topology.kubernetes.io/zone: eu-west-1b
        - name: eu-west-1c
          nodeAffinityLabels:
            topology.kubernetes.io/zone: eu-west-1c

Traefik values

service:
  enabled: true
  type: LoadBalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internal
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
ports:
  cql:
    expose: true
    port: 30942

IngressRouteTCP

    apiVersion: traefik.containo.us/v1alpha1
    kind: IngressRouteTCP
    metadata:
      name: demo-dc1-stargate-service-native-ingress
    spec:
      entryPoints:
      - cql
      routes:
      - match: HostSNI(`*`)
        services:
        - name: demo-dc1-stargate-service
          port: 9042
      tls:
        passthrough: true

cqlshrc

[authentication]
username = demo-superuser
password = changeme

[connection]
hostname = cassandra.private.tk.dev.infratest.example.com
port = 30942

[ssl]
validate = false
certfile = /Users/stdmje/git/cassandra-toolbox/generate_cluster_ssl_stores/ssl_artifacts_20220913_230745/cassandra_cert.pem

• Logs:

❯ cqlsh --ssl

/opt/homebrew/Cellar/cassandra/4.0.6/libexec/bin/cqlsh.py:460: DeprecationWarning: Legacy execution parameters will be removed in 4.0. Consider using execution profiles.
/opt/homebrew/Cellar/cassandra/4.0.6/libexec/bin/cqlsh.py:460: DeprecationWarning: Using ssl_options without ssl_context is deprecated and will result in an error in the next major release. Please use ssl_context to prepare for that release.
Connection error: ('Unable to connect to any servers', {'10.55.133.183:30942': PermissionError(1, "Tried connecting to [('10.55.133.183', 30942)]. Last error: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:997)"), '10.55.217.127:30942': PermissionError(1, "Tried connecting to [('10.55.217.127', 30942)]. Last error: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:997)"), '10.55.78.246:30942': PermissionError(1, "Tried connecting to [('10.55.78.246', 30942)]. Last error: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:997)")})

┆Issue is synchronized with this Jira Story by Unito