s3 minio snapshots upload stopped working

[RogerSik]

Describe the bug s3 minio upload was working fine. After some weeks it stopped working and we don't see why.

Environment (please complete the following information):

• OS: ubuntu 22.04
• Kernel: 5.15.0-78-generic
• Containerized?: Kubernetes
• Dragonfly Version: 1.23.2

dragonfly cluster:

---
apiVersion: dragonflydb.io/v1alpha1
kind: Dragonfly
metadata:
  name: dragonfly
  namespace: dragonfly-cluster
spec:
  # https://www.dragonflydb.io/docs/managing-dragonfly/operator/dragonfly-configuration

  # https://github.com/dragonflydb/dragonfly/releases
  image: docker.dragonflydb.io/dragonflydb/dragonfly:v1.23.2

  replicas: 3

  resources:
    limits:
      memory: 8Gi
      cpu: 2000m 
    requests:
      memory: 2Gi
      cpu: 500m

  snapshot:
    cron: '0 * * * *'
    dir: 's3://k8s-dragonfly-cluster/'

  env:
    # CREDENTIALS
    - name: AWS_ACCESS_KEY_ID
      value: 'CHANGE_ME'
    - name: AWS_SECRET_ACCESS_KEY
      value: 'CHANGE_ME'

    # LOCATION
    - name: AWS_DEFAULT_REGION
      value: 'us-east-1'

    - name: DFLY_s3_endpoint
      value: 'minio.minio.svc.cluster.local:9000'

    - name: DFLY_s3_use_https
      value: 'false'

A test upload with the same credentials with the minio pod and winscp was successfully.

The dragonfly pods are not mention that they have some s3 access problems:

I20241015 09:51:35.636667    12 snapshot_storage.cc:190] Creating AWS S3 client; region=us-east-1; https=false; endpoint=minio.minio.svc.cluster.local:9000
I20241015 09:51:35.636746    12 credentials_provider_chain.cc:28] aws: disabled EC2 metadata
I20241015 09:51:35.640914    12 credentials_provider_chain.cc:36] aws: loaded credentials; provider=environment
I20241015 09:51:35.653036    11 snapshot_storage.cc:247] Load snapshot: Searching for snapshot in S3 path: s3://k8s-dragonfly-cluster/
W20241015 09:51:35.659649     1 server_family.cc:923] Load snapshot: No snapshot found
I20241015 09:51:35.668336    12 listener_interface.cc:101] sock[7] AcceptServer - listening on port 9999
I20241015 09:51:35.668480    11 listener_interface.cc:101] sock[8] AcceptServer - listening on port 6379
I20241015 09:51:55.515954    11 server_family.cc:2694] Replicating 10.44.11.5:9999
I20241015 09:51:55.523744    11 replica.cc:566] Started full sync with 10.44.11.5:9999
I20241015 09:52:03.342170    11 replica.cc:586] full sync finished in 7.82 s
I20241015 09:52:03.342244    11 replica.cc:668] Transitioned into stable sync
I20241015 09:52:24.150570    11 replica.cc:692] Exit stable sync