[backend] KFP 2.15.0 breaks IRSA on AWS: AccessKey is now required

[rvalkenaers]

Environment

• How did you deploy Kubeflow Pipelines (KFP)? Standalone deployment using the official KFP Kustomize manifests.

• KFP version: 2.15.0

• KFP SDK version: Not relevant for this issue (problem occurs in api-server startup, independent of SDK).

Steps to reproduce

1. Deploy Kubeflow Pipelines 2.15.0 using the official standalone Kustomize manifests.
2. Configure object storage to use AWS S3 with IRSA authentication.
3. Set the following environment variables to empty values (as was valid in KFP 2.14.x):
   • OBJECTSTORECONFIG_ACCESSKEY
   • OBJECTSTORECONFIG_SECRETACCESSKEY
4. Start the KFP API server. Observed error in the api-server logs:

Failed to initialize ClientManager: failed to initialize object store: failed to build config from environment variables: ObjectStoreConfig.AccessKey is required

Expected result

In KFP 2.14.x it was valid to leave OBJECTSTORECONFIG_ACCESSKEY and OBJECTSTORECONFIG_SECRETACCESSKEY empty when using IRSA authentication on AWS. The API server should accept empty values and fall back to the underlying IRSA credentials, as it did in previous versions.

Materials and Reference

• Use IRSA with the AWS SDK
• Regression confirmed between 2.14.x → 2.15.0
• Likely introduced by the following PR: https://github.com/kubeflow/pipelines/pull/12394

---

Impacted by this bug? Give it a 👍.

[juliusvonkohout]

I strongly recommend to go via seaweedfs so https://github.com/kubeflow/pipelines/tree/master/manifests/kustomize/third-party/seaweedfs#gateway-to-remote-object-storage gateway to aws. our tests are done with seaweedfs

[rvalkenaers]

Thanks for the quick response.

I looked at the SeaweedFS gateway documentation you linked, but the examples only show configurations using explicit access keys and secret keys. I don’t see any guidance on how to use the gateway with IRSA, and our setup relies on IRSA-based authentication instead of static credentials.

According to the official KFP docs (https://www.kubeflow.org/docs/components/pipelines/operator-guides/configure-object-store/#api-server-supported-providers), AWS S3 with IRSA is a supported configuration. Because this worked correctly in KFP 2.14.x, I wouldn’t expect this functionality to break in a minor release, especially without any migration notes or deprecation warnings.

This still looks like a regression caused by the stricter validation introduced in #12394. Happy to provide any additional info if needed.

[juliusvonkohout]

I have also raised https://github.com/kubeflow/website/pull/4252 to update the documentation. Please also check whether the dangerous and insecure artifact proxy must be enabled in pipeline-install-config.yaml for that.