Install redpanda console through operator

[pvsune]

Cover letter

Support installing Redpanda Console via the operator.

This PR supports the minimum configuration needed to install Console. The PR is huge to review already; other features (see below) will be done in follow up PRs to help in reviewing.

A minimal Console CR might look like:

apiVersion: redpanda.vectorized.io/v1alpha1
kind: Console
metadata:
  name: my-console
spec:
  # Defaults are set https://github.com/redpanda-data/redpanda/pull/5758/files#diff-1120641da9564909cc77411d89009ba02787493e8bde59d2cf6c56e4007c58c9
  server:
    listenPort: 5000
  schema:
    enabled: true
  clusterKeyRef:
    name: rp-abc1234
    namespace: rp-1-234-5678
  deployment:
    image: vectorized/console:master
  connect:
    enabled: true
    clusters:
    - name: connectors-cluster
      url:  http://rp-1-234-5678.rp-abc1234.svc.cluster.local:8083
      tls:
        enabled: true
        secretKeyRef:
          name: my-console-tls
          namespace: default

This PR supports Console configs listed here. The following configs though will be supported in a follow up PR:

• Redpanda which enables Console to get exact Redpanda/Kafka version
• Console which enables topic documentation through git
• Kafka.Protobuf Kafka.Messagepack

Support for mTLS Kafka API will also be in a follow up PR.

The Console controller will create the following resources:

• Kafka Service Account, ACLs
• ConfigMap
• ServiceAccount
• Deployment
• Service

And they all will be owned by the Console object. It also adds the following finalizers for cleanup:

• consoles.redpanda.vectorized.io/service-account
• consoles.redpanda.vectorized.io/acl

The controller implements a ThreadSafeStore which synchronize resources (mostly Secrets for TLS certs) across namespaces. This enables the controller to reference a Cluster in different namespace from the Console CR.

Most of the controller logic are found at pkg/console directory.

Ref

• #2142

[emaxerrno]

this is cool!

Let's make sure that the user should be able to pass all console configuration options somehow @weeco has really good product intuition here, i'd make him a required reviewer on this.

[vladoschreiner]

I opened the conversation about the missing configuration for the external access. Please resolve before finishing the review.

[alenkacz]

@pvsune looking at the minimal CR

tls:
        enabled: true
        secretKeyRef:
          name: my-console-tls
          namespace: default

should we rename secretKeyRef to nodeCertSecretKeyRef or something ... or how do you expect this to be extended if we ever need mutual TLS? Or does console not support mTLS?

Also wondering

  clusterKeyRef:
    name: rp-abc1234
    namespace: rp-1-234-5678

is this reference to redpanda?

[pvsune]

@alenkacz

does console not support mTLS?

Yes, I think this is the mTLS settings for Kafka Connect REF

how do you expect this to be extended if we ever need mutual TLS

For Kafka API mTLS settings though, I think we should just automatically detect it from the referenced Cluster if RequireClientAuth is enabled and set the correct certificates instead of asking in the spec REF

is this reference to redpanda?

Yes, this is the referenced Redpanda Cluster CR

[pvsune]

This fails on the entrypoint test because I mistakenly pushed this directly to upstream instead of in my fork. As discussed, I will create another PR on my fork, close this, and reference this PR on the new one so we can refer the conversations