kubernetes icon indicating copy to clipboard operation
kubernetes copied to clipboard

Minio pod crashes after updating from 24.1.0 to 24.1.1

Open tomikonio opened this issue 7 months ago • 6 comments

After updating with existing data from 24.1.0 to 24.1.1 the minio container crashes.
We tested the minio without providing data and it works.

In 24.1.1 minio has been updated - from 2021.4.22-debian-10-r6 to 2024.5.10-debian-12-r2 (link to diff)

Error overview:

  • The minio container was first crashing because it did not meet its readinessProbe constraints
  • After disabling both readiness and liveness probes the container kept crashing - with the following error:
Defaulted container "minio" out of: minio, volume-permissions (init)
 10:58:49.36 INFO  ==> 
 10:58:49.36 INFO  ==> Welcome to the Bitnami minio container
 10:58:49.36 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
 10:58:49.36 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
 10:58:49.36 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
 10:58:49.36 INFO  ==> 
 10:58:49.36 INFO  ==> ** Starting MinIO setup **
minio 10:58:49.38 DEBUG ==> Validating settings in MINIO_* env vars..
minio 10:58:49.40 INFO  ==> Starting MinIO in background...
ERROR Unable to use the drive /data: drive not found: Invalid arguments specified
minio 10:58:59.50 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:04.58 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:09.68 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:14.77 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:19.88 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:24.98 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:30.07 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:35.18 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:40.28 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:45.38 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:50.48 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:55.58 INFO  ==> Adding local Minio host to 'mc' configuration...
Failed to add temporary MinIO server
minio 11:00:00.68 INFO  ==> MinIO is already stopped...

This is our minio configuration in values.yaml (updated from the old-minio accesskey/secretkey to rootUser/rootPassword):

  minio:
    image:
      debug: true
    auth:
      rootUser: "miniouser"
      rootPassword: "miniopassword"
    persistence:
      existingClaim: minio-data
      mountPath: /data
    volumePermissions:
      enabled: true
    livenessProbe:
      enabled: false
    readinessProbe:
      enabled: false

About ERROR Unable to use the drive /data: drive not found: Invalid arguments specified - the /data dir is set to the correct minio container user (1001) due to volumePermissions.enabled=true setting:

I have no name!@my-release-minio-6b9666d756-qw9hj:/$ ls -la / | grep data
drwxrwsr-x  81 1001 1001 4096 Jul  9 08:42 data

And all data is indeed there:

I have no name!@my-release-minio-7b4bdb744d-q79b7:/data$ ls -la
total 356
drwxrwsr-x  81 1001 1001  4096 Jul  9 08:42 .
drwxr-xr-x   3 root root    18 Jun 26 05:01 ..
-rw-------   1 1001 1001     6 Jul  9 11:49 .access_key
drwxrwsr-x.  7 1001 1001  4096 Jul  9 11:57 .minio.sys
-rw-------   1 1001 1001     9 Jul  9 11:57 .root_password
-rw-------   1 1001 1001     6 Jul  9 11:57 .root_user
-rw-------   1 1001 1001    12 Jul  9 11:49 .secret_key
drwxrws---   2 1001 1001 16384 Feb 29 15:39 lost+found
drwxrwsr-x.  2 1001 1001  4096 Feb 29 17:47 prj-1
drwxrwsr-x.  4 1001 1001  4096 Feb 29 17:47 prj-102
.................................

Reverting back to 24.1.0 -> all is working correctly with the data.

Any idea?

tomikonio avatar Jul 09 '24 11:07 tomikonio