[Feature] Flow run infra overrides

[ohadch]

closes #9893

Hey,

This PR introduces a new infra_overrides column on the FlowRun object. Then, the worker prioritizes the flow run overrides above the deployment overrides.

This is still a draft, but the current status gives a general sense of the code changes. Please let me know if I missed anything. I will mark the PR as ready for review when the code is ready.

Update: This PR is ready for review from my side, I do understand that there is a high chance that it will not get accepted, but due to our high motivation for this feature I'm leaving it here in case it will assist in the development :)

Example

Checklist

• [x] This pull request references any related issue by including "closes <link to issue>"
  • If no issue exists and your change is not a small fix, please create an issue first.
• [x] This pull request includes tests or only affects documentation.
• [ ] This pull request includes a label categorizing the change e.g. maintenance, fix, feature, enhancement, docs.

For documentation changes:

• [ ] This pull request includes redirect settings in netlify.toml for files that are removed or renamed.

For new functions or classes in the Python SDK:

• [ ] This pull request includes helpful docstrings.
• [ ] If a new Python file was added, this pull request contains a stub page in the Python SDK docs and an entry in mkdocs.yml navigation.

[netlify[bot]]

👷 Deploy request for prefect-docs-preview pending review.

Visit the deploys page to approve it

Name	Link
🔨 Latest commit	d38b958fac625c24c880a79c2f14afe394da45b5

[WillRaphaelson]

Thanks @ohadch. This will require some changes to our cloud product as well, which is not a public repo. As such, we're going to take up this work, inspired by this PR. I'll provide an update on timeline around the end of next week.

[ohadch]

Hey @WillRaphaelson , thanks for the update. This feature is super crucial for us :) Is there any way I can assist to help make it happen as soon as possible?

[WillRaphaelson]

thanks @ohadch - unfortunately for the cloud / private repo side theres nothing you can do to help at this time, but we really appreciate this PR on the OSS side. This work is approved internally and the best estimate I can give right now is that we'll get it out in the next month. Sorry I cant be more committal / accurate than that at this time but this issue will be the best place to get additional information.

[ohadch]

Hey @WillRaphaelson , thanks again for updating. I am glad the PR is beneficial. So I understand I should leave it open here for now?

[WillRaphaelson]

Yes for the time being, thanks.

[urimandujano]

hey @ohadch - thank you for taking the time to come up with this implementation (which looks solid). I've just started working on this feature internally and I think it will be easier to manage timing/naming consistency if we close this in favor of new PRs I'll be making. Once again, thank you for the interest and for scoping out what needs to be changed!