Add possibility to add sidecar containers to the main-container `redpanda`

[GezimSejdiu]

Hi there,

I'm trying to attach another sidecar container to the redpanda container and based on the crd: https://github.com/redpanda-data/redpanda/blob/9ade95e13279ca8a8471586134af995269c7a017/src/go/k8s/config/crd/bases/redpanda.vectorized.io_clusters.yaml#L661-L662 it says Sidecars is a list of sidecars run alongside redpanda container -- but it seems it does restrict to only those which have been specified on the specification e.g. https://github.com/redpanda-data/redpanda/blob/9ade95e13279ca8a8471586134af995269c7a017/src/go/k8s/config/crd/bases/redpanda.vectorized.io_clusters.yaml#L664 .

I was wondering if we can e.g. provide it via a podTemplate (see this as an example of what Flink community did: https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/pod-template/ ) or similar approaches so that we can also expand the deployment which will allow us to attach e.g. sidecars containers (as an example could be to add a log aggregation, or metric collection).

I wasn't able to make it so that I can attach such a container alogside the rpkStatus to the redpanda main container or maybe I'm missing something. I do not want to patch the manifest after its deployment :( but rather want to have it deployet via the same manifest.

Thanks a lot for doing great work with redpanda.

Best regards,

[JEndler]

Adding a small example use-case:

This would, for example, enable the use-case to use the default Prometheus metrics which are exported from redpanda from <node ip>:9644/metrics and translate them to GCP stackdriver metrics using a prometheus-to-sd sidecar container.

Currently, one would have to deploy a separate Prometheus instance, collect all the metrics there, and then export them to GCP's Monitoring Suite somehow. This approach makes the deployment considerably more lightweight, because in this setup, there is no need to deploy a Prometheus instance at all, each pod simply has the metrics directly translated to Stackdriver and they are accumulated inside GCP's Monitoring Suite.

This could then simply be deployed as:

sidecars:
    - name: prometheus-to-sd
    image: gcr.io/google-containers/prometheus-to-sd:v0.9.0
    ports:
        - name: profiler
        containerPort: 6060
        - name: debug
        containerPort: 16061
    command:
        - /monitor
        - --stackdriver-prefix=custom.googleapis.com
        - --source=redpanda:http://localhost:9644/metrics
        - --pod-id=$(POD_NAME)
        - --namespace-id=$(POD_NAMESPACE)
    env:
        - name: POD_NAME
        valueFrom:
            fieldRef:
            fieldPath: metadata.name
        - name: POD_NAMESPACE
        valueFrom:
            fieldRef:
            fieldPath: metadata.namespace