[core] Fix max_calls option when used on a worker that is part of a workflow

[SebastianMorawiec]

Why are these changes needed?

Context: https://github.com/ray-project/ray/issues/40252 Because of how workflow frameworks sets max_calls on remote, it is currently impossible to use max_calls option within workflows. Removing the exception workarounds the problem

Related issue number

Checks

• [x] I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• [x] I've run scripts/format.sh to lint the changes in this PR.
• [ ] I've included any doc changes needed for https://docs.ray.io/en/master/.
  • [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in doc/source/tune/api/ under the corresponding .rst file.
• [x] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • [ ] Unit tests
  • [ ] Release tests
  • [x] This PR is not tested :(

[stale[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

• If you'd like to keep this open, just leave any comment, and the stale label will be removed.

[stephanie-wang]

Hi @SebastianMorawiec thanks for the contribution!

However, we cannot simply remove this check on max_calls since that can have correctness issues. max_calls says how many times the task can be executed on the same worker process, so it doesn't make sense to be able to override it.

What you can do instead is modify the workflows library to create a new version of the remote function if max_calls is set, instead of trying to override it via options().

[anyscalesam]

@SebastianMorawiec any follow up questions here? Would love to get this in.

[SebastianMorawiec]

Hi, im sorry - I do not have capacity right now to dig deep into the code to fix it as intended. I might find some later next month tho.