K8SSAND-1754 ⁃ encryption cannot be configured but disabled in C* 4.0.5

[mieslep]

What happened? Attempted to start a cluster using a manifest with encryption defined, but disabled; the cluster starts correctly when these are enabled but the first pod trying to start fails with a stacktrace, and the cluster does not come up:

ERROR [main] 2022-08-26 16:01:48,145 CassandraDaemon.java:911 - Exception encountered during startup
org.apache.cassandra.exceptions.ConfigurationException: Failed to initialize SSL
        at org.apache.cassandra.config.DatabaseDescriptor.applySslContext(DatabaseDescriptor.java:1014)
        at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:364)
        at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:178)
        at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:162)
        at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:818)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:754)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
Caused by: java.io.IOException: Failed to create SSL context using Internode messaging
        at org.apache.cassandra.security.SSLFactory.validateSslContext(SSLFactory.java:546)
        at org.apache.cassandra.config.DatabaseDescriptor.applySslContext(DatabaseDescriptor.java:1008)
        ... 6 common frames omitted
Caused by: java.io.IOException: failed to build key manager store for secure connections
        at org.apache.cassandra.security.SSLFactory.buildKeyManagerFactory(SSLFactory.java:252)
        at org.apache.cassandra.security.SSLFactory.createNettySslContext(SSLFactory.java:315)
        at org.apache.cassandra.security.SSLFactory.validateSslContext(SSLFactory.java:496)
        ... 7 common frames omitted
Caused by: java.nio.file.NoSuchFileException: conf/.keystore
        at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
        at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(Unknown Source)
        at java.base/java.nio.file.Files.newByteChannel(Unknown Source)
        at java.base/java.nio.file.Files.newByteChannel(Unknown Source)
        at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(Unknown Source)
        at java.base/java.nio.file.Files.newInputStream(Unknown Source)
        at org.apache.cassandra.security.SSLFactory.buildKeyManagerFactory(SSLFactory.java:227)
        ... 9 common frames omitted

Did you expect to see something different?

The cassandra.yaml on the pod should either have valid paths to keystore/truststore, or it should not.

client_encryption_options:
  enabled: false
  optional: true
  require_client_auth: true
  store_type: PKCS12
  keystore: conf/.keystore
  keystore_password: cassandra
  truststore_password: password
  truststore: conf/.truststore
server_encryption_options:
  internode_encryption: none
  optional: true
  require_client_auth: true
  store_type: PKCS12
  enable_legacy_ssl_storage_port: false

conf/.keystore and conf/.truststore do not exist.

How to reproduce it (as minimally and precisely as possible):

1. apply encryption-ready.yaml

Environment WSL2 Ubuntu 20.04 using kind

• K8ssandra Operator version:

v1.12.0

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:22:29Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.4 Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.7", GitCommit:"b56e432f2191419647a6a13b9f5867801850f969", GitTreeState:"clean", BuildDate:"2022-03-06T21:07:35Z", GoVersion:"go1.16.14", Compiler:"gc", Platform:"linux/amd64"} WARNING: version difference between client (1.24) and server (1.22) exceeds the supported minor version skew of +/-1

• Kubernetes cluster kind:

kind

• Manifests:

encryption-ready.yaml

kind: K8ssandraCluster
metadata:
  name: demo
spec:
  cassandra:
    serverVersion: "4.0.5"
    datacenters:
      - metadata:
          name: dc1
        size: 3
        storageConfig:
          cassandraDataVolumeClaimSpec:
            storageClassName: standard
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 5Gi
        config:
          jvmOptions:
            heapSize: 512M
          cassandraYaml:
            server_encryption_options:
              internode_encryption: none
              require_client_auth: true
              store_type: PKCS12
              optional: true
            client_encryption_options:
              enabled: false
              optional: true
              require_client_auth: true
              store_type: PKCS12
        stargate:
          size: 1
          heapSize: 256M
    serverEncryptionStores:
      keystoreSecretRef:
        name: server-encryption-stores
      truststoreSecretRef:
        name: server-encryption-stores
    clientEncryptionStores:
      keystoreSecretRef:
        name: client-encryption-stores
      truststoreSecretRef:
        name: client-encryption-stores

• K8ssandra Operator Logs:

n/a

Anything else we need to know?: I was chatting with adejanovski about this issue.

┆Issue is synchronized with this Jira Story by Unito