Use minio s3 storage output error "Unable to locate credentials"

[lklkxcxc]

/kind bug

What steps did you take and what happened: [A clear and concise description of what the bug is.] init storage-initializer log output:

2024-08-13 01:02:12.899 1 kserve INFO [initializer-entrypoint:<module>():16] Initializing, args: src_uri [s3://model/tensorflow/test] dest_path[ [/mnt/models]
2024-08-13 01:02:12.899 1 kserve INFO [storage.py:download():66] Copying contents of s3://model/tensorflow/test to local
Traceback (most recent call last):
  File "/storage-initializer/scripts/initializer-entrypoint", line 17, in <module>
    Storage.download(src_uri, dest_path)
  File "/kserve/kserve/storage/storage.py", line 86, in download
    Storage._download_s3(uri, out_dir)
  File "/kserve/kserve/storage/storage.py", line 225, in _download_s3
    for obj in bucket.objects.filter(Prefix=bucket_path):
  File "/prod_venv/lib/python3.9/site-packages/boto3/resources/collection.py", line 81, in __iter__
    for page in self.pages():
  File "/prod_venv/lib/python3.9/site-packages/boto3/resources/collection.py", line 171, in pages
    for page in pages:
  File "/prod_venv/lib/python3.9/site-packages/botocore/paginate.py", line 269, in __iter__
    response = self._make_request(current_kwargs)
  File "/prod_venv/lib/python3.9/site-packages/botocore/paginate.py", line 357, in _make_request
    return self._method(**current_kwargs)
  File "/prod_venv/lib/python3.9/site-packages/botocore/client.py", line 565, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/prod_venv/lib/python3.9/site-packages/botocore/client.py", line 1001, in _make_api_call
    http, parsed_response = self._make_request(
  File "/prod_venv/lib/python3.9/site-packages/botocore/client.py", line 1027, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/prod_venv/lib/python3.9/site-packages/botocore/endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
  File "/prod_venv/lib/python3.9/site-packages/botocore/endpoint.py", line 198, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/prod_venv/lib/python3.9/site-packages/botocore/endpoint.py", line 134, in create_request
    self._event_emitter.emit(
  File "/prod_venv/lib/python3.9/site-packages/botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/prod_venv/lib/python3.9/site-packages/botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "/prod_venv/lib/python3.9/site-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "/prod_venv/lib/python3.9/site-packages/botocore/signers.py", line 105, in handler
    return self.sign(operation_name, request)
  File "/prod_venv/lib/python3.9/site-packages/botocore/signers.py", line 199, in sign
    auth.add_auth(request)
  File "/prod_venv/lib/python3.9/site-packages/botocore/auth.py", line 418, in add_auth
    raise NoCredentialsError()
botocore.exceptions.NoCredentialsError: Unable to locate credentials

I use neither serviceaccount secret and kserve.yaml configuration credentials both output last log.

  credentials: |-
    {
       "storageSpecSecretName": "storage-config",
       "storageSecretNameAnnotation": "serving.kserve.io/storageSecretName",
       "gcs": {
           "gcsCredentialFileName": "gcloud-application-credentials.json"
       },
       "s3": {
           "s3AccessKeyIDName": "xxxxx",
           "s3SecretAccessKeyName": "xxxxx",
           "s3Endpoint": "10.10.70.2:9000",
           "s3UseHttps": "0",
           "s3Region": "",
           "s3VerifySSL": "0",
           "s3UseVirtualBucket": "",
           "s3UseAccelerate": "",
           "s3UseAnonymousCredential": "false",
           "s3CABundle": ""
       }
    }

What did you expect to happen:

What's the InferenceService yaml: [To help us debug please run kubectl get isvc $name -n $namespace -oyaml and paste the output]

apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "test-sample"
spec:
  predictor:
    serviceAccountName: sa
    model:
      modelFormat:
        name: tensorflow
      storageUri: "s3://model/tensorflow/test"

kubectl get secret $name -n $namespace -oyaml

kind: "InferenceService"
metadata:
  name: "test-sample"
spec:
  predictor:
    serviceAccountName: sa
    model:
      modelFormat:
        name: tensorflow
      storageUri: "s3://model/tensorflow/test"

[root@k8s-test-1 kubeflow]# cat sa-secret.yaml 
---
apiVersion: v1
kind: Secret
metadata:
  name: s3creds
  annotations:
     serving.kserve.io/s3-endpoint: 10.10.70.2:9000 # replace with your s3 endpoint e.g minio-service.kubeflow:9000
     serving.kserve.io/s3-usehttps: "0" # by default 1, if testing with minio you can set to 0
     serving.kserve.io/s3-region: "minio"
     serving.kserve.io/s3-useanoncredential: "false" # omitting this is the same as false, if true will ignore provided credential and use anonymous credentials
type: Opaque
data: # use `stringData` for raw credential string or `data` for base64 encoded string
  AWS_ACCESS_KEY_ID: xxxxx
  AWS_SECRET_ACCESS_KEY: xxxxxxxx
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa
secrets:
- name: s3creds

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• Istio Version:
• Knative Version:
• KServe Version: v0.13.0
• Kubeflow version:
• Cloud Environment:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
• Minikube/Kind version:
• Kubernetes version: (use kubectl version):v1.26.1
• OS (e.g. from /etc/os-release):