TimescaleDB Single ServiceAccount missing create:service permission

[theelderbeever]

What happened? Deployed timescaledb-single chart and was receiving multiple restarts of the primary node. Upon inspecting the logs I saw

2023-03-27 21:27:20,165 ERROR: create_config_service failed Traceback (most recent call last): File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 950, in _create_config_service if not self._api.create_namespaced_service(self._namespace, body): File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 483, in wrapper return getattr(self._core_v1_api, func)(*args, **kwargs) File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 419, in wrapper return self._api_client.call_api(method, path, headers, body, **kwargs) File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 388, in call_api return self._handle_server_response(response, _preload_content) File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 218, in _handle_server_response raise k8s_client.rest.ApiException(http_resp=response) patroni.dcs.kubernetes.K8sClient.rest.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': '6f61849e-2713-4cf9-960c-1b37f81a857b', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '07f5f817-c1f7-4ea4-9bde-892c11b4ecb0', 'X-Kubernetes-Pf-Prioritylevel-Uid': '9e745d7c-26e9-4dcb-8469-44fcdfbfa5da', 'Date': 'Mon, 27 Mar 2023 21:27:20 GMT', 'Content-Length': '335'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services is forbidden: User \"system:serviceaccount:billing-platform:billing-platform-timescaledb\" cannot create resource \"services\" in API group \"\" in the namespace \"billing-platform\"","reason":"Forbidden","details":{"kind":"services"},"code":403}\n'

I manually added privileges to the Role for services and the error went away.

Did you expect to see something different? That there shouldn't be an error.

How to reproduce it (as minimally and precisely as possible): Deploy the chart?

Environment

• Which helm chart and what version are you using? timescaledb-single 0.30.0

• What is in your values.yaml ?

timescaledb-single:
  replicaCount: 2
  secrets:
    credentialsSecretName: "billing-platform-timescaledb-patroni"
    pgbackrestSecretName: "billing-platform-timescaledb-pgbackrest"

  backup:
    enabled: true
    
  service:
    primary:
      type: LoadBalancer
      port: 5432

    replica:
      type: LoadBalancer
      port: 5432

  persistentVolumes:
    data:
      enabled: true
      size: 250Gi
      storageClass: gp3-iops3k
    wal:
      enabled: true
      size: 25Gi
      storageClass: gp3-iops3k
  resources:
    limits:
      cpu: 2000m
      memory: 8192Mi
    requests:
      cpu: 2000m
      memory: 8192Mi

  sharedMemory:
    useMount: true

  pgBouncer:
    enabled: true
    port: 6432
    config:
      max_client_conn: 5000
      default_pool_size: 25

  prometheus:
    enabled: true

• Kubernetes version information:

  kubectl version

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:33:11Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.10-eks-48e63af", GitCommit:"9176fb99b52f8d5ff73d67fea27f3a638f679f8a", GitTreeState:"clean", BuildDate:"2023-01-24T19:17:48Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

AWS EKS via terraform

Anything else we need to know?:

[abhinavDhulipala]

I'm having this problem as well. More specifically

kubectl logs tobs0-timescaledb-1
2023-04-06 05:18:27,510 ERROR: create_config_service failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 950, in _create_config_service
    if not self._api.create_namespaced_service(self._namespace, body):
  File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 483, in wrapper
    return getattr(self._core_v1_api, func)(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 419, in wrapper
    return self._api_client.call_api(method, path, headers, body, **kwargs)
  File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 388, in call_api
    return self._handle_server_response(response, _preload_content)
  File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 218, in _handle_server_response
    raise k8s_client.rest.ApiException(http_resp=response)
patroni.dcs.kubernetes.K8sClient.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'a173ae86-3dfd-4d3b-99bd-4d6b4b4060d7', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '26e83d07-726c-450e-ba7d-5196d84b714d', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e50744da-a79f-42da-bd7d-0dc68624af3e', 'Date': 'Thu, 06 Apr 2023 05:18:27 GMT', 'Content-Length': '318'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services is forbidden: User \\"system:serviceaccount:observability:tobs0-timescaledb\\" cannot create resource \\"services\\" in API group \\"\\" in the namespace \\"observability\\"","reason":"Forbidden","details":{"kind":"services"},"code":403}\n

[abhinavDhulipala]

Got by this particular bug via

kubectl edit role tobs0-timescaledb

rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - services # <-- added this
  verbs:
  - create
  - get
  - list
  - patch
  - update
  - watch
  - delete

[kopecmaciej]

If you have charts locally you can Just add in templates/role.timescaledb.yaml

rules:
- apiGroups: [""]
  resources: ["services"]
  verbs:
  - create
  - get
  - list
  - patch
  - update
  - watch
  - delete

[hongbo-miao]

I met same issue.

Thanks the information from @joschiwald at https://github.com/timescale/helm-charts/pull/601#issuecomment-1510381605 ! 😃

That issue is already fixed on patroni side. You could use the patroni-dcs-failsafe tags of the timescale/timescaledb-ha images, which uses a more current patroni version.

I succeed using new patroni-dcs-failsafe tag by

helm upgrade \
  timescale \
  timescaledb-single \
  --install \
  --repo=https://charts.timescale.com \
  --namespace=my-timescale \
  --create-namespace \
  --values=my-values.yaml

my-values.yaml

image:
  repository: timescale/timescaledb-ha
  tag: pg15.2-ts2.10.2-patroni-dcs-failsafe-p0

without any errors.

[Meet-S0ni]

image:
  repository: timescale/timescaledb-ha
  tag: pg15.2-ts2.10.1-p1

2023-08-28 11:55:09,173 ERROR: Permission denied
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/patroni/daemon.py", line 174, in abstract_main
    controller.run()
  File "/usr/lib/python3/dist-packages/patroni/__main__.py", line 133, in run
    super(Patroni, self).run()
  File "/usr/lib/python3/dist-packages/patroni/daemon.py", line 143, in run
    self._run_cycle()
  File "/usr/lib/python3/dist-packages/patroni/__main__.py", line 145, in _run_cycle
    self.schedule_next_run()
  File "/usr/lib/python3/dist-packages/patroni/__main__.py", line 127, in schedule_next_run
    elif self.ha.watch(nap_time):
  File "/usr/lib/python3/dist-packages/patroni/ha.py", line 1885, in watch
    return self.dcs.watch(leader_version, timeout)
  File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 1357, in watch
    return super(Kubernetes, self).watch(None, timeout)
  File "/usr/lib/python3/dist-packages/patroni/dcs/__init__.py", line 1134, in watch
    self.event.wait(timeout)
  File "/usr/lib/python3.10/threading.py", line 607, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/lib/python3.10/threading.py", line 324, in wait
    gotit = waiter.acquire(True, timeout)
  File "/usr/lib/python3/dist-packages/patroni/daemon.py", line 90, in sigterm_handler
    sys.exit()
SystemExit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 570, in _run_and_handle_exceptions
    return method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 1281, in touch_member
    ret = self._api.patch_namespaced_pod(self._name, self._namespace, body)
  File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 556, in wrapper
    return getattr(self._core_v1_api, func)(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 477, in wrapper
    return self._api_client.call_api(method, path, headers, body, **kwargs)
  File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 445, in call_api
    return self._handle_server_response(response, _preload_content)
  File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 264, in _handle_server_response
    raise k8s_client.rest.ApiException(http_resp=response)
patroni.dcs.kubernetes.K8sClient.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'aa8dd4b9-539d-404f-b9f0-a1ff9aec1168', 'Cache-Control': 'no-cache, private', 'Content-Type': 'applions': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'b7b72011-04fb-45e6-9c9b-f157c676089d', 'X-Kubernetes-Pf-Prioritylevel-Uid': '7f4f2292-6b31-4951-88 Aug 2023 11:55:09 GMT', 'Content-Length': '333'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \\"timescale-db-v15-0\\" is forbidden: User ale-db-v15\\" cannot patch resource \\"pods\\" in API group \\"\\" in the namespace \\"db\\"","reason":"Forbidden","details":{"name":"timescale-db-v15

[pfrydids]

Think this is related to #554

Any update as to when the default helm install will use these updated container images?