Helm/gloo-ee: observability deployment not convenient

[alexandrebrg]

Gloo Edge Version

1.11.x

Kubernetes Version

1.22.x

Describe the bug

This issue describes a few problems related to observability module from gloo-ee helm chart which should create dynamically grafana dashboards depending on upstreams. I didn't think it was relevant to split it into multiple issues at it refers to the same component of the helm chart.

1. Documentation refers values that are not used / available in the helm chart.

Most values starting with observability.deployment.customEnv are not used in observability deployment. This statement is done through viewing file templates/9-observability-deployment.yaml.

Not having the ability to add custom secret / environmment variables currently prevent us from working with it as we do use gloo generated secret for grafana credentials.

2. Observability secret generated in templates/9-observability-secret.yaml cannot be disabled, and is created with empty credentials if not defined

Credentials handled can be different between infrastructure (helm-secret, sealedsecret, vault...), it would be convenient to be able to disable generation of this secret in order to generate our own.

3. Observability deployment's pod doesn't exist if there is a problem.

Indeed, if the credentials are wrong or if the grafana instance cannot be reached, the pod will hang without exiting or any healthcheck to alert of the problem.

Steps to reproduce the bug

First two bugs doesn't any step to be reproduced as they are only related to logic / values.

To reproduce the third problem:

1. Run gloo-ee charts with some customGrafana dummy values as follows:

glooee:
  create_license_secret: false
  observability:
    enabled: true
    customGrafana:
      enabled: true
      url: http://some-dummy-grafana.gloo-system.svc.cluster.local

2. Deploy the chart, assert that observability pod appears
3. The pod will try to connect to grafana without succeeding and won't exit

Expected Behavior

For the second problem we would expect to be able to disable the creation of this secret.

For the third problem, we would expect the pod to crash or be unhealthy at least.

Additional Context

We really enjoy the idea of having dynamic dashboards on grafana, we hope to be able to use it soon :)

[chrisgaun]

Thanks for submitting. There may be other issues in the backlog that are related. I will look into that.

[github-actions[bot]]

This issue has been marked as stale because of no activity in the last 180 days. It will be closed in the next 180 days unless it is tagged "no stalebot" or other activity occurs.