"the Alertmanager has no configuration and no fallback specified", even with X-Scope-OrgID

[tina-junold]

Describe the bug

I'm trying to install and configure mimir together with my grafana, but the healthcheck always fail.

To Reproduce

Steps to reproduce the behavior:

install mirmir using this values using external minio

fullnameOverride: mimir
mimir:
  structuredConfig:
    multitenancy_enabled: true # tried also false
    alertmanager_storage:
      backend: s3
      s3:
        access_key_id: minio-access-key
        bucket_name: mimir-ruler
        endpoint: minio.minio.svc:9000
        insecure: true
        secret_access_key: minio-secret-key
    blocks_storage:
      backend: s3
      s3:
        access_key_id: minio-access-key
        bucket_name: mimir-tsdb
        endpoint: minio.minio.svc:9000
        insecure: true
        secret_access_key: minio-secret-key
    ruler_storage:
      backend: s3
      s3:
        access_key_id: minio-access-key
        bucket_name: mimir-ruler
        endpoint: minio.minio.svc:9000
        insecure: true
        secret_access_key: minio-secret-key
chunks-cache:
  enabled: true
index-cache:
  enabled: true
metadata-cache:
  enabled: true
results-cache:
  enabled: true
minio:
  enabled: false

Expected behavior

Grafana should not fail adding mimir as datasource, i also tried to set the X-Scope-OrgID header to "demo".

Environment

k3s/k3d cluster 1.24.4

Additional Context

It seems to be related with this https://github.com/grafana/grafana/issues/49728 but I'm not sure what todo to fix this issue.

[stevesg]

The problem here is exactly the issue you linked to, unfortunately it's not the best user experience if you configure Grafana before setting an Alertmanager configuration (which is normal if you want to use the Grafana UI to configure Alertmanager). It looks like https://github.com/grafana/grafana/pull/53396 has tried to workaround this issue though, and it looks like it will be included in Grafana 9.2.0-beta1.

As a workaround, you could set a fallback configuration by passing -alertmanager.configs.fallback to Mimir (Alertmanager) - I suspect an empty file would be sufficient.

[pracucci]

I also have a proposal to add a built-in default config: https://github.com/grafana/mimir/issues/2772

I suspect an empty file would be sufficient

Unfortunately it doesn't pass the validation, because it requires at least 1 route. The minimum config is:

route:
  receiver: empty-receiver

receivers:
  - name: 'empty-receiver'

[dimitarvdimitrov]

There is also a helm chart change that will resolve this: #2942

[tina-junold]

@dimitarvdimitrov this MR give me an understanding of how the configuration work. I'm not into this topic very well yet. I try to build a (local) cluster that allow distributed tracing and as i read that mimir adds additional metrics for the traces. I don't have the need of multiple tenants since the "big" clusters are separated per customer.