[feature] How to pass secret pipeline parameters?

[cgebbe]

Feature Area

/area frontend /area backend

What feature would you like to see?

• Given a pipeline with a parameter "api_key",
• when a user triggers a run (via the UI or the CLI)
• then his API key is publicly visible in the UI currently,
• but we would like to hide it.

What is the use case or pain point?

We would like to hide such secret parameters as much as possible. Since the API key belongs to a separate system, it would be nice if not even the kubeflow admins could access it.

Is there a workaround currently?

I'm aware of kubeflow secrets, but this does not seem to be targeted at parameters if I understand it correctly. However, it's easily possible that I've missed something. I currently see two workarounds:

• pass secrets, which are only valid for a certain time
• pass encrypted secrets and store the key for decryption as kubeflow secret (or in AWS secret manager or similar)

Finally, this post states that pipeline arguments are not visible in the UI, but I observe differently?!

[Linchin]

Maybe try using a container op and wrap the secret parameter inside? https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod

Here is an example of passing gcp secret: https://github.com/kubeflow/examples/blob/871895c54402f68685c8e227c954d86a81c0575f/pipelines/mnist-pipelines/mnist_pipeline.py#L97 Maybe AWS has some similar function.

[robertschweizer]

Thanks for the suggestion, @Linchin!

Our "api_key" is slightly more complicated: Every user has a different one, so our backend system can check which data each user is actually allowed to access in their pipeline runs, and track who created output data. But we want these API keys to be unknown to other users, so they cannot impersonate each other.

We are using RSA encryption now to pass secret parameters: Our client application uses a public key to encrypt the "api_key" parameter before submitting through the Kubeflow API. Then the container code gets the private key as an AWS SSM Parameter (Similar to the GCP secret you mentioned above) and decrypts the "api_key". This makes it sufficiently hard for users who see other people's pipelines to access their secret API keys.

I have one doubt about this solution though, which applies to any secrets that Kubeflow jobs can access: AFAIK, every user who can see pipeline runs in Kubeflow can also upload and run their own pipeline. So they could upload a malicious pipeline that simply prints the value of an internal secret (like our private key stored as an AWS SSM Parameter).

Do you know any solution to this, other than really implementing secret parameters in Kubeflow?

[rimolive]

Closing this issue. No activity for more than a year.

/close

[google-oss-prow[bot]]

@rimolive: Closing this issue.

In response to this:

Closing this issue. No activity for more than a year.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.