Configure labels or annotations on child resources

[spencergilbert]

Proposal

Allow users to set labels on the "top level" resources created from the CRDs

Use case. Why is this important? Today users can label the pods created by the ECK operator, but the statefulset for elasticsearch doesn't have any means of being labelled. Presumably the kibana would have the same issue - we use labels in monitoring, logs, generic discoverability, etc.

It'd be very helpful to have specifications in the CRD to apply labels to the deployments/statefulsets created from them.

[spencergilbert]

I suppose also labelling the created services and all would be nice too

[sebgl]

Would it make sense to have all StatefulSets/Deployment inherit the labels applied to the Elasticsearch/Kibana resource?

Example:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
  labels:
    foo: bar
spec:
  version: 7.6.0
  nodeSets:
  - name: default
    count: 1

would label the underlying StatefulSet with foo: bar. Same for Kibana Deployment. And all other resources: services, secrets, etc.

It feels like an easy way of labeling everything at once. Would that cover your use case @spencergilbert? Or would you expect to be able to specify per-NodeSet/StatefulSet labels?

[spencergilbert]

That would definitely solve my use case :+1:

[charith-elastic]

After prototyping how this would work, I think that it would be better to let users decide which labels or annotations they want to propagate to child objects. This is because some labels/annotations are probably meant for the CR only and having them propagated to children could result in confusion and/or unexpected behaviour (e.g. deleting by label selector).

I am proposing that we introduce two annotations that can be applied to CRs:

eck.k8s.elastic.co/propagate-annotations: "comma,separated,list,of,annotations,to,propagate"
eck.k8s.elastic.co/propagate-labels: "comma,separated,list,of,labels,to,propagate"

Setting the annotation value to * would propagate all annotations/labels in the CR.

[sebgl]

Similar issue for setting headless services labels: https://github.com/elastic/cloud-on-k8s/issues/3990. As Charith suggested above, I think it makes sense to label all child resources, and not just StatefulSets.

[tschroeder-zendesk]

I am currently unable to use ECK due to this issue as my company requires me to label all resources with my team. It's been open for a long time. Any possibility of doing this soon?

[zpear]

Any chance of this happening soon? My team would benefit from this as well

[winterrobert]

We have a use case where we want to annotate the data and master services to sync them into Consul for our service discovery; as this can't be done with the operator currently we have to work around it by patching the service resources using Server-Side Apply in Argo CD which adds complexity and is inconsistent.

+1 for configurable labels or annotations on child resources.

[frivoire]

I would also be interested in the possibility to define labels on the StatefulSets (of ES nodesets) and on the Deployments (of Kibana). Because I need to have a team label on all K8S resources.

Maybe the solution to this issue is to create another structure: .spec.nodeSets[*].stsTemplate in the Elasticsearch resource in which we could fill .metadata.labels (the same way that we can already do it for Pods using Elasticsearch.spec.nodeSets[*].podTemplate.metadata.labels).

And similar in other resources: Kibana, etc...

[frivoire]

I would like to emphatize on one response:

After prototyping how this would work, I think that it would be better to let users decide which labels or annotations they want to propagate to child objects. This is because some labels/annotations are probably meant for the CR only and having them propagated to children could result in confusion and/or unexpected behaviour (e.g. deleting by label selector).

I totally agree with this "do not propagate all labels blindly, but let user choose" idea 👍 My main example where it's important is when the Elasticsearch/Kibana resources are created by an Helm Chart:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  annotations:
    meta.helm.sh/release-name: elastic-test
    meta.helm.sh/release-namespace: dev
  labels:
    app.kubernetes.io/managed-by: Helm
    helm.toolkit.fluxcd.io/name: elastic-test
    helm.toolkit.fluxcd.io/namespace: dev
  name: elastic-test

=> we do not want those helm labels to be put on the StatefulSet/Deployment resources, because they are not managed by Helm and Helm would mess them !

So, please do not propagate blindly the labels ! But a feature to be able to customize them would be very useful (cf my previous message) :)

[tomsozolins]

It would be helpful to add kubernetes recommended labels automatically on the pods -https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/

For example: app.kubernetes.io/name: name app.kubernetes.io/instance: instance

One use case where such labels are needed is network observability with cilium. Currently all eck pods are shown as Unknown App.

[JKrehling]

Is their any current workaround to being able to deploy the statefulsets with propagated labels? I liked the solution proposed in #3186 but it sounds like it never got merged due to not being able to clean up unused labels.
Since this is opt in anyways I'm not sure the cleanup is something that is really necessary since the labels that are required seldom change and without it you are hindered more than if you had dangling ones that you would have to clean up manually.

[davidegreenwald]

+1 on this request. This is preventing Datadog unified service tagging best practices, which look for a set of labels at both the StatefulSet and pod level.

[ebinsu]

+1 on this request.