scylla-operator
scylla-operator copied to clipboard
scylladb
Allow adding custom annotations to k8s services scylla operator cerates
- Feature Request
What should the feature do: Integration with external DNS ( https://github.com/kubernetes-sigs/external-dns )
What is the use case behind this feature: Automatically create DNS records for Scylla k8s services. For example the headless service This is useful in many situations, for example having a DNS record for the Headless service will allow easy integration with Scylla from outside the k8s cluster
For example,
- if a developer wants to connect to some Scylla node from his machine without explicitly using a Scylla node IP
- round robin load balancing of Scylla nodes ( outside of k8s )
Additional Information: Adding custom annotations to services ( and other k8s entities ) It is a common practice that allows integration with different tools Every bitnami chart allows adding custom annotations
What prevents you from adding the annotation on the service itself?
fwiw, v2 is planning to allow setting common labels and annotations for all child objects but this one seems to be needed to be set on the service only
it should keep the custom annotations https://github.com/scylladb/scylla-operator/blob/5257bd6da9bda02eca588ab20c7eea988681e5df/pkg/resourceapply/core.go#L75
@tnozicka I see. The issue is that we are operating the cluster with argocd and gitops semantics.
And this will require manually tagging the service.. Which is a bad thing.
But thanks for the workaround. Its still a good feature inmo since gitops is becoming the standard way to manage k8s resources. Other operators allow adding custom annotations, and inmo people will expect it.
🙏
I think adding meta is useful but with that level or granularity it becomes a bit hard to easily express it on the CRD. Not impossible though.
Other operators allow adding custom annotations, and inmo people will expect it.
Do you have examples you expect to be added to the CRD on how other operators add those meta fields with this level of granularity?
Why doesn't this work with git ops? if you create the object first, we'll follow the 3 laws of controllers and adopt it. If we create it first, git ops should apply your object and add that annotation. As none of them conflict, I'd assume this would work, wouldn't it?
Didn't know those laws. Can you please elaborate on the "adopt" part. I can create an empty service with no selectors and only an annotation And the operator will take it ( assuming since its name will be the same as he would have created ) And configure it further?
Regarding Examples.
Prometheus operator CRD
description: PodMetadata configures Labels and Annotations which are propagated to the prometheus pods.
properties:
annotations:
additionalProperties:
type: string
description: 'Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: http://kubernetes.io/docs/user-guide/annotations'
type: object
labels:
additionalProperties:
type: string
description: 'Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: http://kubernetes.io/docs/user-guide/labels'
type: object
name:
description: 'Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/identifiers#names'
type: string
type: object
It allows adding custom annotations to anything it creates IIRC.
The Three Laws of controllers are described here https://github.com/kubernetes/design-proposals-archive/blob/main/api-machinery/controller-ref.md#the-three-laws-of-controllers In your case that means pre-creating a service with your annotation and the same selector.
What I meant is that annotations for every child object are not that hard, but it's harder to do annotation that gets only applied to a service for node with index 6 or the headless service. I wonder whether the adoption would solve it as well though.
What if we have to set type LoadBalancer to allow external access?
Seems like it's not possible, unfortunately.
API changes are tracked in https://github.com/scylladb/scylla-operator/issues/1590
also the node services already have customizable annotations
$ kubectl explain scyllacluster.spec.exposeOptions.nodeService.annotations
GROUP: scylla.scylladb.com
KIND: ScyllaCluster
VERSION: v1
FIELD: annotations <map[string]string>
DESCRIPTION:
annotations is a custom key value map merged with every node Service
annotations.