"The specified bucket does not exist" but all buckets are present in minio

[kandeshvari]

Describe the bug

I'm getting the error for all services

ts=2024-03-25T21:35:58.095869334Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: The specified bucket does not exist."

All needed buckets exist in minio. root credentials for minio are correctly passed, and mimir sees it. I can list all created buckets via mc utility with those credentials

part of the values.yaml

mimir:
  structuredConfig:
    common:
      storage:
        backend: s3
        s3:
          insecure: true
          http:
            insecure_skip_verify: true
          endpoint: minio.minio-tenant-mimir.svc.cluster.local
          access_key_id: PA...
          secret_access_key: hXy...

    alertmanager_storage:
      s3:
        insecure: true
        bucket_name: mimir-alertmanager

    ruler_storage:
      s3:
        insecure: true
        bucket_name: mimir-ruler

    blocks_storage:
      s3:
        insecure: true
        bucket_name: mimir-blocks

How can I debug this behavior and figure out the reason why mimir does not see the buckets?

I spent the whole day trying, and I'm close to despair :(

Environment

• Infrastructure: bare-metal Kubernetes 1.27 on Virtual Machines
• Deployment tool: helm, argocd
• versions:
  • grafana/mimir-distributed, 5.2.3
  • minio/operator, 5.0.11
  • minio/tenant, 5.0.11

[andyasp]

This may be because there is a minio section in the values file. If you're setting up your minio seperately then at the top-level you could counter-intuitively set

minio:
  enabled: false

One way you could verify if this is the interaction you're experiencing is by looking at the config that is actually being generated with:

kubectl describe configmap/mimir-config

For instance, if you see mimir-tsdb in the config and not your mimir-blocks, then that's the issue.

[kandeshvari]

Thanks for your reply, @andyasp!

Yes, I tried to use embedded minio, and it works fine. It seems like some policies are not set properly. Is there a guide to set up the junction of mimir and minio?

[andyasp]

It seems like some policies are not set properly.

I'm not sure what this means, but it sounds like maybe you figured it out?

Is there a guide to set up the junction of mimir and minio?

Not that I'm aware of. There's the object storage configuration documentation, but it's not specifically about minio. From Mimir's perspective it is only aware of the fact that it is talking to an s3 backend. There's nothing special about minio in this sense. As long as it knows the buckets and how to talk to it/how to auth to it that's basically it.

The helm chart has that embedded minio included by default so it can work out of the box, but it's suggested to override it for production use cases. You can see an artifact of that in some of the sample value files, like here.

[kandeshvari]

@andyasp . I just found a root cause: that was a confusing error message.

my settings are:

mimir:
  structuredConfig:
    common:
      storage:
        s3:
          insecure: true
          http:
            insecure_skip_verify: true
          endpoint: minio.minio-tenant-mimir.svc.cluster.local
          ...

key point here is insecure: true. This endpoint receives only HTTPS. For some reason, I got the error message

blocks storage: unable to successfully send a request to object storage: The specified bucket does not exist.

instead of

blocks storage: unable to successfully send a request to object storage: HTTP request made to HTTPS server

I removed insecure: true, and everything started to work fine.

Is it a bug?

[andyasp]

Interesting. I don't think that's a bug with the sanity check. It's rather simple and creates the bucket client and GETs a single object (code reference). Here the error it's getting from minio has the code NoSuchBucket and it's reflecting that back.

If there was a simple reproduction procedure (like with curl or minio-go directly) to verify that swapping HTTP/HTTPS is all that's going on then filing an issue with minio may help improve that.

[narqo]

blocks storage: unable to successfully send a request to object storage: The specified bucket does not exist.

I think this is a "non-ideal" behaviour in mimio-go (the s3 API client, we use through the objstore package): they treat all responses with 404 as "bucket not exists" (ref the code in mimio-go).

[andyasp]

they treat all responses with 404 as "bucket not exists"

That isn't true and would be very incorrect behavior. Requesting an object that doesn't exist results in 404 and a NoSuchKey code. The code you linked shows that in the else case. That's what objstore looks for too: objstore ref

[narqo]

they treat all responses with 404 as "bucket not exists"

That isn't true and would be very incorrect behavior. Requesting an object that doesn't exist results in 404 and a NoSuchKey code. The code you linked shows that in the else case

What I wanted to say is that a request to an object, needs to locate the object's bucket, first, by calling to S3's GetBucketLocation API (ref to this chunk in minio-go).

I suspect, the error reported in the issue came from such request, as it returned 404 due to wrong protocol.

[andyasp]

Ah, gotcha.