[backend] Cannot get MLMD objects from Metadata store: received initial metadata size exceeds limit

[vkatrychenko]

Environment

• How did you deploy Kubeflow Pipelines (KFP)? deployed kubeflow using k8s manifests (standalone kubeflow components): https://github.com/kubeflow/manifests
• KFP version: kfp 2.4.1
• KFP SDK version: kfp-sdk: 2.12.1, 2.11.0
• MySQL server: 8.0.21 Azure flexible mysql (also, tried to use self-hosted mysql v8.0.26)

Steps to reproduce

Deploy the latest kubeflow ver v1.10.0-rc.2 with dex/oauth2-proxy auth. Set up Microsoft auth method for dex. Then try to run a pipeline.

When running pipelines under a static kf user, everything is working fine.

Expected result

No errors should appear under pipeline-ui.

Materials and Reference

It seems that the issue occurs due to large azure oidc info according to this thread: https://github.com/kubeflow/pipelines/issues/8733#issuecomment-2627771197

Tried to downgrade/upgrade grpc ml medata service, but did not help.

Do you have a workaround for this?

BTW: we saw this error in v1.9.x as well

---

Impacted by this bug? Give it a 👍.

[vkatrychenko]

For context: everything is working fine in kubeflow 1.7.1.

[NJ3rsey]

Fixed by adding the argument to metadata-grpc-deployment:

spec:
  ...
  template:
    ...
    spec:
      containers:
        - args:
            - '--grpc_channel_arguments=grpc.max_metadata_size=16384' <<-- (16kb instead of default 8kb)

[juliusvonkohout]

Fixed by adding the argument to metadata-grpc-deployment:

spec:
  ...
  template:
    ...
    spec:
      containers:
        - args:
            - '--grpc_channel_arguments=grpc.max_metadata_size=16384' <<-- (16kb instead of default 8kb)

@hbelmiro @HumairAK should we make this the default, maybe even 32k ? See also https://github.com/kubeflow/manifests/tree/master/common/oauth2-proxy#known-issues

[droctothorpe]

We appear to have a user running into the same issue. Agree with @juliusvonkohout that 16kb should be the default.

[droctothorpe]

Confirmed the fix worked for us. Tysm @NJ3rsey!