opentelemetry-helm-charts Webhook timeout when deploying collector

I have deployed the operator (chart version 0.21.4) using the default chart values into a namespace called application. The pod starts and seems healthy. When I try deploy the collector there is an error:

$ kubectl -n application apply -f kube-manifests/open-telemetry/collector.yaml
Error from server (InternalError): error when creating "kube-manifests/open-telemetry/collector.yaml": Internal error occurred: failed calling webhook "mopentelemetrycollector.kb.io": failed to call webhook: Post "https://opentelemetry-operator-webhook-service.application.svc:443/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector?timeout=10s": context deadline exceeded

collector.yaml:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel
  namespace: application
spec:
  mode: deployment
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:

    exporters:
      googlecloud:
      logging:

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [logging, googlecloud]

The firewall allows traffic on 443. Also, a test pod does not timeout when calling the webhook URL directly:

# curl -k -X POST https://opentelemetry-operator-webhook-service.application.svc:443/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector?timeout=30s
{"response":{"uid":"","allowed":false,"status":{"metadata":{},"message":"contentType=, expected application/json","code":400}}}

Feb 02 '23 14:02 chris-minka

@chris-minka hey! What is weird about your error is:

Error from server (InternalError): error when creating "kube-manifests/open-telemetry/collector.yaml": Internal error occurred: failed calling webhook "mopentelemetrycollector.kb.io"

the webhook is named mopentelemetrycollector.kb.io, see the weird m in the beginning of the name ?

I just did a fresh install and for me webhook is named:

opentelemetry-operator-mutating-webhook-configuration

and your example collector deployed without problems.

Any chance you have some webhook from old installation or smth like that?

Could you show us your webhook configs?

kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io

Feb 03 '23 10:02 povilasv

hi @povilasv! thanks for the reply.

$ kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io 
NAME                                                      WEBHOOKS   AGE
cert-manager-webhook                                      1          419d
gke-vpa-webhook-config                                    1          5h25m
gmp-operator.gmp-system.monitoring.googleapis.com         2          5h24m
neg-annotation.config.common-webhooks.networking.gke.io   1          2y51d
opentelemetry-operator-mutating-webhook-configuration     3          45h
pod-ready.config.common-webhooks.networking.gke.io        1          2y168d

$ kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io opentelemetry-operator-mutating-webhook-configuration -o yaml | head -n 62
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  annotations:
    cert-manager.io/inject-ca-from: application/opentelemetry-operator-serving-cert
    meta.helm.sh/release-name: opentelemetry-operator
    meta.helm.sh/release-namespace: application
  creationTimestamp: "2023-02-01T15:30:19Z"
  generation: 2
  labels:
    app.kubernetes.io/component: webhook
    app.kubernetes.io/instance: opentelemetry-operator
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: opentelemetry-operator
    app.kubernetes.io/version: 0.67.0
    helm.sh/chart: opentelemetry-operator-0.21.4
    helm.toolkit.fluxcd.io/name: open-telemetry-operator
    helm.toolkit.fluxcd.io/namespace: application
  name: opentelemetry-operator-mutating-webhook-configuration
  resourceVersion: "745462016"
  uid: 3c8746c1-76c1-41c4-a533-bbcfca54c7ee
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURlVENDQW1HZ0F3SUJBZ0lRWWZIR1A0ZGVleS9lRTVnVStKUThBakFOQmdrcWhraUc5dzBCQVFzRkFEQWgKTVI4d0hRWURWUVFMRXhadmNHVnVkR1ZzWlcxbGRISjVMVzl3WlhKaGRHOXlNQjRYRFRJek1ERXlOekU1TXpBeQpObG9YRFRJek1EUXlOekU1TXpBeU5sb3dJVEVmTUIwR0ExVUVDeE1XYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5CllYUnZjakNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNWE14V1dNQ2FLZzBTZWQKU0RTbVBqWkUzVm1nL0NTczA5YkVDUlZha25peWVQd0lKNXVadVhQSUp1MTRQRmxwbmdOQlovV0w5ZHRzL1VibgpiUXA5N0c4d1lkWEIvZjVKSTBJMk9KMlIyZGFNVWkvdXY2b094cjZRSGdoV3dPelhvWjcxVENNNHFhWHpveGxYCjlqTzUyN01Ma29uNkhVaGdBOGY0RjJETSs1VnJFMnp4Z1dxdHNySGpjWEZlblZITEUwQ1hoSDU1UXNoZWEvcVEKanhQNTFDbWx4WDJaU2QxbmkycUg2UXI5SjBJREJjRzJRNTdGTGprTEtJWEdjaE5tZmYzczhyeU11aEtzNXBsbApYcGVJUVdXYzBmbnpDY2FLTGI1TGd6SGVuMzVTdmlDVzRmR1FVUUJNeXlkV0R0MGhRZmwvV2orM3hRa1RwY012Ck1CRVpPL0VDQXdFQUFhT0JyRENCcVRBT0JnTlZIUThCQWY4RUJBTUNCYUF3REFZRFZSMFRBUUgvQkFJd0FEQ0IKaUFZRFZSMFJCSUdBTUg2Q05tOXdaVzUwWld4bGJXVjBjbmt0YjNCbGNtRjBiM0l0ZDJWaWFHOXZheTF6WlhKMgphV05sTG1Gd2NHeHBZMkYwYVc5dUxuTjJZNEpFYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5WVhSdmNpMTNaV0pvCmIyOXJMWE5sY25acFkyVXVZWEJ3YkdsallYUnBiMjR1YzNaakxtTnNkWE4wWlhJdWJHOWpZV3d3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFFTTExVjUrTjJUWE5MMllBTDlLZlVNSlBmZytzZDFVaEdidDgyVDJHNmtSeHY5cwphNXFFTTU0L2h5MjVjT2xMc0doKzd1bnN0T0NwUnUxNlpGWUVtZy9rdlFyRmpYZ0o4K1l5eW9QSVEvUHM0MnhnCklPQTE0c0F5dUdWbTRxeFFSdGl0ZXVJcnlUbTJvZHozN3RjdmhvL3h3emQ4ay9SMDhhUkUzTld0ODlqR3orbkgKUG5HbERrVU9sKzFQRVZJei9tNkh2S2J6RDFWZnFqN2ltSlJPVVdLRWRJWEdWdk9sNmlpd1YweDlNZk1CWFNJSgpUNHB6c3RObE03WUlGcTBpWjMyMjdRYTdGKzdRZGpJNGpSbzQ0VVd4L1RiVTJVR2VqZ2JpY1VnL1ZOMkxCek1KCjR3Y25TVVFYQVNNQ1VVL0Rvb2NtZnZwQU1PRkhPRUZ1c0JhN0tIZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    service:
      name: opentelemetry-operator-webhook-service
      namespace: application
      path: /mutate-opentelemetry-io-v1alpha1-instrumentation
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: minstrumentation.kb.io
  namespaceSelector: {}
  objectSelector: {}
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - opentelemetry.io
    apiVersions:
    - v1alpha1
    operations:
    - CREATE
    - UPDATE
    resources:
    - instrumentations
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10
- admissionReviewVersions:
  - v1
  clientConfig:
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURlVENDQW1HZ0F3SUJBZ0lRWWZIR1A0ZGVleS9lRTVnVStKUThBakFOQmdrcWhraUc5dzBCQVFzRkFEQWgKTVI4d0hRWURWUVFMRXhadmNHVnVkR1ZzWlcxbGRISjVMVzl3WlhKaGRHOXlNQjRYRFRJek1ERXlOekU1TXpBeQpObG9YRFRJek1EUXlOekU1TXpBeU5sb3dJVEVmTUIwR0ExVUVDeE1XYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5CllYUnZjakNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNWE14V1dNQ2FLZzBTZWQKU0RTbVBqWkUzVm1nL0NTczA5YkVDUlZha25peWVQd0lKNXVadVhQSUp1MTRQRmxwbmdOQlovV0w5ZHRzL1VibgpiUXA5N0c4d1lkWEIvZjVKSTBJMk9KMlIyZGFNVWkvdXY2b094cjZRSGdoV3dPelhvWjcxVENNNHFhWHpveGxYCjlqTzUyN01Ma29uNkhVaGdBOGY0RjJETSs1VnJFMnp4Z1dxdHNySGpjWEZlblZITEUwQ1hoSDU1UXNoZWEvcVEKanhQNTFDbWx4WDJaU2QxbmkycUg2UXI5SjBJREJjRzJRNTdGTGprTEtJWEdjaE5tZmYzczhyeU11aEtzNXBsbApYcGVJUVdXYzBmbnpDY2FLTGI1TGd6SGVuMzVTdmlDVzRmR1FVUUJNeXlkV0R0MGhRZmwvV2orM3hRa1RwY012Ck1CRVpPL0VDQXdFQUFhT0JyRENCcVRBT0JnTlZIUThCQWY4RUJBTUNCYUF3REFZRFZSMFRBUUgvQkFJd0FEQ0IKaUFZRFZSMFJCSUdBTUg2Q05tOXdaVzUwWld4bGJXVjBjbmt0YjNCbGNtRjBiM0l0ZDJWaWFHOXZheTF6WlhKMgphV05sTG1Gd2NHeHBZMkYwYVc5dUxuTjJZNEpFYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5WVhSdmNpMTNaV0pvCmIyOXJMWE5sY25acFkyVXVZWEJ3YkdsallYUnBiMjR1YzNaakxtTnNkWE4wWlhJdWJHOWpZV3d3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFFTTExVjUrTjJUWE5MMllBTDlLZlVNSlBmZytzZDFVaEdidDgyVDJHNmtSeHY5cwphNXFFTTU0L2h5MjVjT2xMc0doKzd1bnN0T0NwUnUxNlpGWUVtZy9rdlFyRmpYZ0o4K1l5eW9QSVEvUHM0MnhnCklPQTE0c0F5dUdWbTRxeFFSdGl0ZXVJcnlUbTJvZHozN3RjdmhvL3h3emQ4ay9SMDhhUkUzTld0ODlqR3orbkgKUG5HbERrVU9sKzFQRVZJei9tNkh2S2J6RDFWZnFqN2ltSlJPVVdLRWRJWEdWdk9sNmlpd1YweDlNZk1CWFNJSgpUNHB6c3RObE03WUlGcTBpWjMyMjdRYTdGKzdRZGpJNGpSbzQ0VVd4L1RiVTJVR2VqZ2JpY1VnL1ZOMkxCek1KCjR3Y25TVVFYQVNNQ1VVL0Rvb2NtZnZwQU1PRkhPRUZ1c0JhN0tIZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    service:
      name: opentelemetry-operator-webhook-service
      namespace: application
      path: /mutate-opentelemetry-io-v1alpha1-opentelemetrycollector
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: mopentelemetrycollector.kb.io

Feb 03 '23 13:02 chris-minka

Everything looks ok, not sure what is the issue :/

Anything in operator logs?

Could you check kubectl describe mutatingwebhookconfigurations.admissionregistration.k8s.io opentelemetry-operator-mutating-webhook-configuration or in general kubectl get events to see if anything points to this issue?

Feb 03 '23 13:02 povilasv

I tried to deploy the Collector again and got the same error. Here is the describe:

$ kubectl describe mutatingwebhookconfigurations.admissionregistration.k8s.io opentelemetry-operator-mutating-webhook-configuration
Name:         opentelemetry-operator-mutating-webhook-configuration
Namespace:    
Labels:       app.kubernetes.io/component=webhook
              app.kubernetes.io/instance=opentelemetry-operator
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=opentelemetry-operator
              app.kubernetes.io/version=0.67.0
              helm.sh/chart=opentelemetry-operator-0.21.4
              helm.toolkit.fluxcd.io/name=open-telemetry-operator
              helm.toolkit.fluxcd.io/namespace=application
Annotations:  cert-manager.io/inject-ca-from: application/opentelemetry-operator-serving-cert
              meta.helm.sh/release-name: opentelemetry-operator
              meta.helm.sh/release-namespace: application
API Version:  admissionregistration.k8s.io/v1
Kind:         MutatingWebhookConfiguration
Metadata:
  Creation Timestamp:  2023-02-01T15:30:19Z
  Generation:          2
  Managed Fields:
    API Version:  admissionregistration.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:webhooks:
        k:{"name":"minstrumentation.kb.io"}:
          f:clientConfig:
            f:caBundle:
        k:{"name":"mopentelemetrycollector.kb.io"}:
          f:clientConfig:
            f:caBundle:
        k:{"name":"mpod.kb.io"}:
          f:clientConfig:
            f:caBundle:
    Manager:      cainjector
    Operation:    Update
    Time:         2023-02-01T15:30:19Z
    API Version:  admissionregistration.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:cert-manager.io/inject-ca-from:
          f:meta.helm.sh/release-name:
          f:meta.helm.sh/release-namespace:
        f:labels:
          .:
          f:app.kubernetes.io/component:
          f:app.kubernetes.io/instance:
          f:app.kubernetes.io/managed-by:
          f:app.kubernetes.io/name:
          f:app.kubernetes.io/version:
          f:helm.sh/chart:
          f:helm.toolkit.fluxcd.io/name:
          f:helm.toolkit.fluxcd.io/namespace:
      f:webhooks:
        .:
        k:{"name":"minstrumentation.kb.io"}:
          .:
          f:admissionReviewVersions:
          f:clientConfig:
            .:
            f:service:
              .:
              f:name:
              f:namespace:
              f:path:
              f:port:
          f:failurePolicy:
          f:matchPolicy:
          f:name:
          f:namespaceSelector:
          f:objectSelector:
          f:reinvocationPolicy:
          f:rules:
          f:sideEffects:
          f:timeoutSeconds:
        k:{"name":"mopentelemetrycollector.kb.io"}:
          .:
          f:admissionReviewVersions:
          f:clientConfig:
            .:
            f:service:
              .:
              f:name:
              f:namespace:
              f:path:
              f:port:
          f:failurePolicy:
          f:matchPolicy:
          f:name:
          f:namespaceSelector:
          f:objectSelector:
          f:reinvocationPolicy:
          f:rules:
          f:sideEffects:
          f:timeoutSeconds:
        k:{"name":"mpod.kb.io"}:
          .:
          f:admissionReviewVersions:
          f:clientConfig:
            .:
            f:service:
              .:
              f:name:
              f:namespace:
              f:path:
              f:port:
          f:failurePolicy:
          f:matchPolicy:
          f:name:
          f:namespaceSelector:
          f:objectSelector:
          f:reinvocationPolicy:
          f:rules:
          f:sideEffects:
          f:timeoutSeconds:
    Manager:         helm-controller
    Operation:       Update
    Time:            2023-02-01T15:30:19Z
  Resource Version:  745462016
  UID:               3c8746c1-76c1-41c4-a533-bbcfca54c7ee
Webhooks:
  Admission Review Versions:
    v1
  Client Config:
    Ca Bundle:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURlVENDQW1HZ0F3SUJBZ0lRWWZIR1A0ZGVleS9lRTVnVStKUThBakFOQmdrcWhraUc5dzBCQVFzRkFEQWgKTVI4d0hRWURWUVFMRXhadmNHVnVkR1ZzWlcxbGRISjVMVzl3WlhKaGRHOXlNQjRYRFRJek1ERXlOekU1TXpBeQpObG9YRFRJek1EUXlOekU1TXpBeU5sb3dJVEVmTUIwR0ExVUVDeE1XYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5CllYUnZjakNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNWE14V1dNQ2FLZzBTZWQKU0RTbVBqWkUzVm1nL0NTczA5YkVDUlZha25peWVQd0lKNXVadVhQSUp1MTRQRmxwbmdOQlovV0w5ZHRzL1VibgpiUXA5N0c4d1lkWEIvZjVKSTBJMk9KMlIyZGFNVWkvdXY2b094cjZRSGdoV3dPelhvWjcxVENNNHFhWHpveGxYCjlqTzUyN01Ma29uNkhVaGdBOGY0RjJETSs1VnJFMnp4Z1dxdHNySGpjWEZlblZITEUwQ1hoSDU1UXNoZWEvcVEKanhQNTFDbWx4WDJaU2QxbmkycUg2UXI5SjBJREJjRzJRNTdGTGprTEtJWEdjaE5tZmYzczhyeU11aEtzNXBsbApYcGVJUVdXYzBmbnpDY2FLTGI1TGd6SGVuMzVTdmlDVzRmR1FVUUJNeXlkV0R0MGhRZmwvV2orM3hRa1RwY012Ck1CRVpPL0VDQXdFQUFhT0JyRENCcVRBT0JnTlZIUThCQWY4RUJBTUNCYUF3REFZRFZSMFRBUUgvQkFJd0FEQ0IKaUFZRFZSMFJCSUdBTUg2Q05tOXdaVzUwWld4bGJXVjBjbmt0YjNCbGNtRjBiM0l0ZDJWaWFHOXZheTF6WlhKMgphV05sTG1Gd2NHeHBZMkYwYVc5dUxuTjJZNEpFYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5WVhSdmNpMTNaV0pvCmIyOXJMWE5sY25acFkyVXVZWEJ3YkdsallYUnBiMjR1YzNaakxtTnNkWE4wWlhJdWJHOWpZV3d3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFFTTExVjUrTjJUWE5MMllBTDlLZlVNSlBmZytzZDFVaEdidDgyVDJHNmtSeHY5cwphNXFFTTU0L2h5MjVjT2xMc0doKzd1bnN0T0NwUnUxNlpGWUVtZy9rdlFyRmpYZ0o4K1l5eW9QSVEvUHM0MnhnCklPQTE0c0F5dUdWbTRxeFFSdGl0ZXVJcnlUbTJvZHozN3RjdmhvL3h3emQ4ay9SMDhhUkUzTld0ODlqR3orbkgKUG5HbERrVU9sKzFQRVZJei9tNkh2S2J6RDFWZnFqN2ltSlJPVVdLRWRJWEdWdk9sNmlpd1YweDlNZk1CWFNJSgpUNHB6c3RObE03WUlGcTBpWjMyMjdRYTdGKzdRZGpJNGpSbzQ0VVd4L1RiVTJVR2VqZ2JpY1VnL1ZOMkxCek1KCjR3Y25TVVFYQVNNQ1VVL0Rvb2NtZnZwQU1PRkhPRUZ1c0JhN0tIZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    Service:
      Name:        opentelemetry-operator-webhook-service
      Namespace:   application
      Path:        /mutate-opentelemetry-io-v1alpha1-instrumentation
      Port:        443
  Failure Policy:  Fail
  Match Policy:    Equivalent
  Name:            minstrumentation.kb.io
  Namespace Selector:
  Object Selector:
  Reinvocation Policy:  Never
  Rules:
    API Groups:
      opentelemetry.io
    API Versions:
      v1alpha1
    Operations:
      CREATE
      UPDATE
    Resources:
      instrumentations
    Scope:          *
  Side Effects:     None
  Timeout Seconds:  10
  Admission Review Versions:
    v1
  Client Config:
    Ca Bundle:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURlVENDQW1HZ0F3SUJBZ0lRWWZIR1A0ZGVleS9lRTVnVStKUThBakFOQmdrcWhraUc5dzBCQVFzRkFEQWgKTVI4d0hRWURWUVFMRXhadmNHVnVkR1ZzWlcxbGRISjVMVzl3WlhKaGRHOXlNQjRYRFRJek1ERXlOekU1TXpBeQpObG9YRFRJek1EUXlOekU1TXpBeU5sb3dJVEVmTUIwR0ExVUVDeE1XYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5CllYUnZjakNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNWE14V1dNQ2FLZzBTZWQKU0RTbVBqWkUzVm1nL0NTczA5YkVDUlZha25peWVQd0lKNXVadVhQSUp1MTRQRmxwbmdOQlovV0w5ZHRzL1VibgpiUXA5N0c4d1lkWEIvZjVKSTBJMk9KMlIyZGFNVWkvdXY2b094cjZRSGdoV3dPelhvWjcxVENNNHFhWHpveGxYCjlqTzUyN01Ma29uNkhVaGdBOGY0RjJETSs1VnJFMnp4Z1dxdHNySGpjWEZlblZITEUwQ1hoSDU1UXNoZWEvcVEKanhQNTFDbWx4WDJaU2QxbmkycUg2UXI5SjBJREJjRzJRNTdGTGprTEtJWEdjaE5tZmYzczhyeU11aEtzNXBsbApYcGVJUVdXYzBmbnpDY2FLTGI1TGd6SGVuMzVTdmlDVzRmR1FVUUJNeXlkV0R0MGhRZmwvV2orM3hRa1RwY012Ck1CRVpPL0VDQXdFQUFhT0JyRENCcVRBT0JnTlZIUThCQWY4RUJBTUNCYUF3REFZRFZSMFRBUUgvQkFJd0FEQ0IKaUFZRFZSMFJCSUdBTUg2Q05tOXdaVzUwWld4bGJXVjBjbmt0YjNCbGNtRjBiM0l0ZDJWaWFHOXZheTF6WlhKMgphV05sTG1Gd2NHeHBZMkYwYVc5dUxuTjJZNEpFYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5WVhSdmNpMTNaV0pvCmIyOXJMWE5sY25acFkyVXVZWEJ3YkdsallYUnBiMjR1YzNaakxtTnNkWE4wWlhJdWJHOWpZV3d3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFFTTExVjUrTjJUWE5MMllBTDlLZlVNSlBmZytzZDFVaEdidDgyVDJHNmtSeHY5cwphNXFFTTU0L2h5MjVjT2xMc0doKzd1bnN0T0NwUnUxNlpGWUVtZy9rdlFyRmpYZ0o4K1l5eW9QSVEvUHM0MnhnCklPQTE0c0F5dUdWbTRxeFFSdGl0ZXVJcnlUbTJvZHozN3RjdmhvL3h3emQ4ay9SMDhhUkUzTld0ODlqR3orbkgKUG5HbERrVU9sKzFQRVZJei9tNkh2S2J6RDFWZnFqN2ltSlJPVVdLRWRJWEdWdk9sNmlpd1YweDlNZk1CWFNJSgpUNHB6c3RObE03WUlGcTBpWjMyMjdRYTdGKzdRZGpJNGpSbzQ0VVd4L1RiVTJVR2VqZ2JpY1VnL1ZOMkxCek1KCjR3Y25TVVFYQVNNQ1VVL0Rvb2NtZnZwQU1PRkhPRUZ1c0JhN0tIZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    Service:
      Name:        opentelemetry-operator-webhook-service
      Namespace:   application
      Path:        /mutate-opentelemetry-io-v1alpha1-opentelemetrycollector
      Port:        443
  Failure Policy:  Fail
  Match Policy:    Equivalent
  Name:            mopentelemetrycollector.kb.io
  Namespace Selector:
  Object Selector:
  Reinvocation Policy:  Never
  Rules:
    API Groups:
      opentelemetry.io
    API Versions:
      v1alpha1
    Operations:
      CREATE
      UPDATE
    Resources:
      opentelemetrycollectors
    Scope:          *
  Side Effects:     None
  Timeout Seconds:  10
  Admission Review Versions:
    v1
  Client Config:
    Ca Bundle:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURlVENDQW1HZ0F3SUJBZ0lRWWZIR1A0ZGVleS9lRTVnVStKUThBakFOQmdrcWhraUc5dzBCQVFzRkFEQWgKTVI4d0hRWURWUVFMRXhadmNHVnVkR1ZzWlcxbGRISjVMVzl3WlhKaGRHOXlNQjRYRFRJek1ERXlOekU1TXpBeQpObG9YRFRJek1EUXlOekU1TXpBeU5sb3dJVEVmTUIwR0ExVUVDeE1XYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5CllYUnZjakNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNWE14V1dNQ2FLZzBTZWQKU0RTbVBqWkUzVm1nL0NTczA5YkVDUlZha25peWVQd0lKNXVadVhQSUp1MTRQRmxwbmdOQlovV0w5ZHRzL1VibgpiUXA5N0c4d1lkWEIvZjVKSTBJMk9KMlIyZGFNVWkvdXY2b094cjZRSGdoV3dPelhvWjcxVENNNHFhWHpveGxYCjlqTzUyN01Ma29uNkhVaGdBOGY0RjJETSs1VnJFMnp4Z1dxdHNySGpjWEZlblZITEUwQ1hoSDU1UXNoZWEvcVEKanhQNTFDbWx4WDJaU2QxbmkycUg2UXI5SjBJREJjRzJRNTdGTGprTEtJWEdjaE5tZmYzczhyeU11aEtzNXBsbApYcGVJUVdXYzBmbnpDY2FLTGI1TGd6SGVuMzVTdmlDVzRmR1FVUUJNeXlkV0R0MGhRZmwvV2orM3hRa1RwY012Ck1CRVpPL0VDQXdFQUFhT0JyRENCcVRBT0JnTlZIUThCQWY4RUJBTUNCYUF3REFZRFZSMFRBUUgvQkFJd0FEQ0IKaUFZRFZSMFJCSUdBTUg2Q05tOXdaVzUwWld4bGJXVjBjbmt0YjNCbGNtRjBiM0l0ZDJWaWFHOXZheTF6WlhKMgphV05sTG1Gd2NHeHBZMkYwYVc5dUxuTjJZNEpFYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5WVhSdmNpMTNaV0pvCmIyOXJMWE5sY25acFkyVXVZWEJ3YkdsallYUnBiMjR1YzNaakxtTnNkWE4wWlhJdWJHOWpZV3d3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFFTTExVjUrTjJUWE5MMllBTDlLZlVNSlBmZytzZDFVaEdidDgyVDJHNmtSeHY5cwphNXFFTTU0L2h5MjVjT2xMc0doKzd1bnN0T0NwUnUxNlpGWUVtZy9rdlFyRmpYZ0o4K1l5eW9QSVEvUHM0MnhnCklPQTE0c0F5dUdWbTRxeFFSdGl0ZXVJcnlUbTJvZHozN3RjdmhvL3h3emQ4ay9SMDhhUkUzTld0ODlqR3orbkgKUG5HbERrVU9sKzFQRVZJei9tNkh2S2J6RDFWZnFqN2ltSlJPVVdLRWRJWEdWdk9sNmlpd1YweDlNZk1CWFNJSgpUNHB6c3RObE03WUlGcTBpWjMyMjdRYTdGKzdRZGpJNGpSbzQ0VVd4L1RiVTJVR2VqZ2JpY1VnL1ZOMkxCek1KCjR3Y25TVVFYQVNNQ1VVL0Rvb2NtZnZwQU1PRkhPRUZ1c0JhN0tIZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    Service:
      Name:        opentelemetry-operator-webhook-service
      Namespace:   application
      Path:        /mutate-v1-pod
      Port:        443
  Failure Policy:  Ignore
  Match Policy:    Equivalent
  Name:            mpod.kb.io
  Namespace Selector:
  Object Selector:
  Reinvocation Policy:  Never
  Rules:
    API Groups:
      
    API Versions:
      v1
    Operations:
      CREATE
      UPDATE
    Resources:
      pods
    Scope:          *
  Side Effects:     None
  Timeout Seconds:  10
Events:             <none>

There are a few events related to other things which are deployed, but nothing pertaining to this.

Feb 03 '23 14:02 chris-minka

What about logs of opentelemetry-operator-controller-manager-... ?

Also Kubernetes API Server might have more details, so try to look at it's logs

Feb 03 '23 14:02 povilasv

@chris-minka did you get anywhere with this? I was seeing the same timeout error (I've just disabled webhooks for now).

@povilasv it seems that mopentelemetrycollector.kb.io is coming from the chart, e.g. https://github.com/open-telemetry/opentelemetry-helm-charts/blob/56701aeb4c52a6fcdb459d3774dd91e6c67a723a/charts/opentelemetry-operator/templates/admission-webhooks/operator-webhook.yaml#L68

Feb 27 '23 17:02 milesarmstrong

@milesarmstrong i ended up deploying the collector using its helm chart instead of using the operator. so, unfortunately i do not have a solution. if i were to revisit the issue, i would provide the logs @povilasv requested. also, since we are using a GKE Private Cluster i would review the firewall rules.

Feb 27 '23 19:02 chris-minka

Thanks @chris-minka, we're also using GKE Private. I suspect it is the firewall rules, thanks for the pointer!

Feb 28 '23 09:02 milesarmstrong

I posted this comment in the knative project, I think the issue is very similar: https://github.com/knative/serving/issues/13045#issuecomment-1359356226

We run our clusters on GKE. The webhook call is made from the apiserver. The webhook pod listens on 8443 whilst the service listens on 443. When making the webhook call the GKE apiserver tries to hit the webhook pod on 8443. Only ports 443 and 10250 are open between the apiserver and the GKE nodes.

Mar 03 '23 16:03 paulgrav

I hit this issue also and digging into it, it is definitely a GKE firewall issue, but it is not a port 8443 issue.

When I set the allow to tcp:8443, the collector still would not build. When I opened all ports tcp:1-65535, collector did build.

So now I am trying to determine what actual port it needs and is getting blocked. I tried 8080 also but no luck. I'll report back when I figure it out (unless someone chimes in and knows what the port should be).

Jul 24 '23 19:07 RaiderAdam

It's port 9443 for some reason (I brute forced found it by trial and error), even though the service describe says 8443. adam@AdamPC:~/opentelemetry$ gcloud compute firewall-rules create gke-opentelemetry-webhook \ --action ALLOW \ --direction INGRESS \ --source-ranges 172.16.0.16/28 \ --rules tcp:9443 \ --target-tags gke-cluster-19216119-node Creating firewall... Creating firewall...done. NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED gke-opentelemetry-webhook default INGRESS 1000 tcp:9443 False adam@AdamPC:~/opentelemetry$ kubectl -n opentelemetry apply -f collector.yaml opentelemetrycollector.opentelemetry.io/simplest created

Jul 24 '23 19:07 RaiderAdam

There is an arg for a webhook port on the operator Args: --webhook-port=9443

If that webhook port gets changed to 443, there shouldn't be a need for a special firewall rule, correct? Since the default GKE firewall would be sufficient.

Jul 24 '23 19:07 RaiderAdam

I get the same error with version 0.36.0 of the opentelemetry-operator chart, and downgrading to 0.34.0 resolves it. I suspect the wrong webhook name was added in the newer version of the chart

Sep 06 '23 19:09 PavelPikat

opentelemetry-helm-charts opentelemetry-helm-charts copied to clipboard

Webhook timeout when deploying collector

opentelemetry-helm-charts
opentelemetry-helm-charts copied to clipboard