PrefectHQ
All environment variables defined in work pools are overridden if any environment variables are defined in a deployment. Improve behavior?
First check
- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar issue and didn't find it.
- [X] I searched the Prefect documentation for this issue.
- [X] I checked that this issue is related to Prefect and not one of its dependencies.
Bug summary
Environment variables are not coming up as expected in google cloud run - push work pools. We have set up env variables in work pool config which is not reflected back in the flow run ~when deploying using prefect.yaml but works with .deploy() method~ if environment variables are provided at deployment time.
- With deploy method from flow code - ~work pool env variables are reflected back in the flow~ workpool env variables are cleared if env variables are passed in along with job variable
- With deployment yaml block style deployment- work pool env variables aren't reflected back into the flow but env variables defined in prefect.yaml works.
Reproduction
set up enviornment variable in google cloud run- push workpool config
deploy flow to read env variable using `prefect deploy' command to the workpool
Error
print(os.environ["TEST"])
File "/opt/homebrew/Cellar/[email protected]/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/os.py", line 680, in __getitem__
raise KeyError(key) from None
KeyError: 'TEST'
Versions
2.13.8
Additional context
Not sure how the precedence of env variables works when we can define it in multiple places - prefect.yaml, workpool config
Hi @EldhoKSuresh. Thank you for raising. Currently, if any environment variables are defined in a work pool they are all overwritten at deployment creation time if any environment variables are provided. At least that's the expected behavior - although, perhaps not the ideal behavior.
I'm going to make the title for this issue a bit broader to encompass that.
@desertaxle
I'd like to be able to set some base variables in job template and then override or append selectively from the deployment
hi @discdiver yes you are right. I didn't test out both deployment properly. Work pool env variables are cleared if env variables are passed in during deployment time. Same behaviour with both deployment yaml & .deploy() method
Selectively overriding a single environment variable for a deployment (especially at runtime) would be really useful.
I hope this gets worked on 🤞
A good description of what we need here is what Ben described: when we merge overrides dictionaries, we should do so recursively (aka, deep merge) rather than a shallow merge.
network_configuration on the ECS worker is a dict where each key in the dict represents an optional value supplied to the RunTask request. If someone has securityGroups set on their work pool, but there's a deployment that omits securityGroups but includes subnets for example, would this end up merging them and including both fields in the request?
@kevingrismore yup that is correct - would that be an issue or an enhancement from your perspective?
