Enabled internal TLS between k8s pods by default

[jk464]

Continuing on from the theme in (https://github.com/StackStorm/stackstorm-k8s/pull/400) this PR is another attempt at improving the security hardening of StackStorm in k8s.

By default - all inter-pod communication between pods within a k8s deployment use non-encrypted protocols. (i.e. Between st2 and Mongo / RabbitMQ and Redis, and to the internal api, auth and stream endpoints within st2)

This is fine if your security model is hard (secure) on the outside (public facing), but crunchy (insecure) on the inside (private facing) - (like an Armadillo 😉).

However, ideally a modern approach would be zero-trust and secure communication everywhere.

This PR therefore, enables by default, Encryption/TLS everywhere*. It makes use of cert-manager to enabled automatic generation of certificates, either using a user provided CA (to allow use of a corporate CA) or if the user already has a cert issuer setup - one can provide that.

TLS is enabled/configured globally for st2 endpoints (api, auth, stream) under the st2 block in values.yaml

  tls:
    enabled: true
    secretName: "internal-tls"
    mountPath: "/etc/ssl/internal"
    certificate_issuer:
      existing: false
      name: stackstorm-issuer

And then the individual endpoints can be toggled on/off under their respective blocks in values.yaml.

Sadly the mongo and rabbitmq TLS can't be setup from the st2 config directly - so they're present (and have defaults configured) under their respective blocks as well.

The st2 templates have then been updated to automatically configure all the required settings to use these certificates to enabled TLS - and the cert/key/ca are also automatically mounted to all the containers.

Note since the api and stream endpoints don't support encryption, we have added a TLS proxy layer (using ghostunnel) infront of them to enable inter-pod encryption while allowing unencrypted communication on the intra-pod commuication between the st2 component and the proxy layer.

*Note that the OpenStack Tooz library, which st2 uses to talk to Redis, doesn't have support to configure the settings needed to enable TLS when using Sentinel - so Redis traffic is still unencrypted in this PR. However, we have an internal build of Tooz that adds this support, and we've also got the changes to the StackStorm helm template to support that working internally. Once we have our changes to Tooz merged upstream, we can provide a PR to stackstorm-k8s and st2 core library to enable this support as well. (You can get a sneak preview of those changes here https://github.com/jk464/stackstorm-k8s/tree/feature/redis-tls)

[jk464]

@cognifloyd thanks for the additional comments, I'll take a look at them on Monday, and I also have a go at updating st2api and stream to add TLS support then aswell 😊