Question configuration example for clustering Kubernetes/Redis locker?

[pboers1988]

In the documentation I can only find details about how to setup consul as a clustering locker. I was wondering if there are some working examples of how to setup Kubernetes as locker in the clustering configuration. I see there is code in the repository to support it, however there are no good examples in the tests or in the docs about how to configure it. The same can be said about redis. Does anyone have some pointers towards how to setup these lockers? Or is consul still the only locker that "really works"?

I would prefer to be able to use Kubernetes or redis as they are already part of the infrastructure that we are running. I could obviously try to reverse engineer, but maybe someone has a working example! Thanks!

[hellt]

I think there are no docs specifically about the k8s locker option. Maybe you can get what you need from the comments in the PR that added this capability https://github.com/karimra/gnmic/issues/560#issuecomment-1099762889

(or maybe wait till @karimra comes back online)

[karimra]

There are no dedicated doc pages for those two lockers, that needs to be added.

Meanwhile here is a short explanation of the config options with their defaults:

• kubernetes locker

clustering:
  # 
  locker:
    type: k8s
    # namespace where the leases will be created
    namespace: default
    # lease-duration, defines the duration of a lease
    lease-duration: 10s
    # defines how often leases are renewed.
    # If the set values is eq or higher than the `lease-duration`, lease-duration/2 applies.
    # Otherwise the set value applies.
    renew-period: 
    # defines a simple backoff duration in case of errors
    retry-timer: 2s
    # enable extra logging
    debug: false

• redis locker:

clustering:
  # 
  locker:
    type: redis
    # single address or a seed list of host:port addresses of cluster/sentinel nodes.
    # if more than one address is set, the locker uses a redis cluster client.
    servers:
      - 
    # specifies a master name to create a sentinel backed redis FailoverClient
    master-name:
    # a password that should match `requirepass` in redis server config (Redis 5.0 or lower)
    # or the user password (Redis 6.0 or higher)
    password:
    # sets the expiration time of a Redis mutex
    lease-duration: 10s
    # defines the frequency of Redis mutex expiration refresh.
    # If the set values is eq or higher than the `lease-duration`, lease-duration/2 applies.
    # Otherwise the set value applies.
    renew-period: 
    # defines a simple backoff duration in case of errors.
    retry-timer: 2s
    # poll timer of registered Redis services.
    poll-timer: 10s
    # enable extra logging
    debug: false

When writing this I realized that the redis locker could use some updates to add some extra config options.

[pboers1988]

Update 2 Found the missing RBAC in karimras issue https://github.com/karimra/gnmic/issues/560#issuecomment-1099762889

Update I'm mistaken, I'm still overlooking a setting. Getting the following error:

│ 2024/01/15 12:41:56.662043 /home/runner/work/gnmic/gnmic/pkg/lockers/k8s_locker/k8s_registration.go:49: [k8s_locker] watch ended with error: unknown (get endpoints)

Any ideas?

---

Thanks! I reverse-engineered the interface through the go files last week :) and now I'm attempting to lock down the service account. I've narrowed down the what the account needs by feeding it privledges until it stops erroring. Am I missing anything else?

kind: Role
metadata:
  annotations:
    meta.helm.sh/release-name: gnmic
  labels:
    app.kubernetes.io/instance: gnmic
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: gnmic
    app.kubernetes.io/version: 0.34.3
  name: gnmic-role
rules:
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - create
  - get
  - list
  - update
  - delete
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
   - get
   - list  

Thanks!