Add secret_env_vars to runtime-env parameter of job submission

[jhasm]

Why are these changes needed?

These changes enable passing secrets to ray jobs without leaking them.

The runtime_env is a variable that is packaged and made available to all workers running the job, and it has env_vars key that allows passing arbitrary environment variables to the workers, scoped to this job. But the env vars are visible on the dashboard and logged in plain text. So this is not suitable for passing secrets. To address this limitation, we introduce another key secret_env_vars in the runtime_env, which behaves similar to the env_vars but will not be exposed anywhere.

Related issue number

Closes #39502

Checks

• [x] I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• [x] I've run scripts/format.sh to lint the changes in this PR.
• [ ] I've included any doc changes needed for https://docs.ray.io/en/master/.
  • [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in doc/source/tune/api/ under the corresponding .rst file.
• [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • [x] Unit tests
  • [ ] Release tests
  • [ ] This PR is not tested :(

[jhasm]

@rynewang I cannot add a reviewer, but wanted to make sure you see this. Thanks!

[rynewang]

cc @pcmoritz @thomasdesr

[stale[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

• If you'd like to keep this open, just leave any comment, and the stale label will be removed.

[anyscalesam]

@rynewang this looks good; do you need final OK from @pcmoritz and @thomasdesr before merging?

[rynewang]

Philipp already talked to this user with another approach. I don’t think we are merging this PR.

On Wed, May 15, 2024 at 03:40 Samuel Chan @.***> wrote:

@rynewang https://github.com/rynewang this looks good; do you need final OK from @pcmoritz https://github.com/pcmoritz and @thomasdesr https://github.com/thomasdesr before merging?

— Reply to this email directly, view it on GitHub https://github.com/ray-project/ray/pull/40549#issuecomment-2111009765, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANLX3X6I764UNUMYDJFNL63ZCJSA7AVCNFSM6AAAAAA6KS7PL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJRGAYDSNZWGU . You are receiving this because you were mentioned.Message ID: @.***>

[prd-tuong-nguyen]

@rynewang Hey bro, can you share a bit about the alternative solution to address this issue?

[rynewang]

@prd-tuong-nguyen the recommended approach is to inject the secret via the mechanisms already established in your environment, e.g. Use K8S Secrets to set env vars, or to use secret managers from your cloud vendor, or similar approaches.

[hongbo-miao]

Hi @rynewang we are using Ray to compute and write result to MLflow, each user need pass their own MLflow credentials as they have different access for different experiments in MLflow. Any suggestions for how should we handle this case? As I feel if we use Kubernetes secret, then hard to dynamic use different user secrets. Thank you!

[rynewang]

@pcmoritz can you share some advices on how to pass secrets to Ray tasks?