Add execution context in `current` singleton
Currently, the only way to check whether a task is executing locally, on batch or with step-functions is to sniff some environment variables - we can make it a bit easier by including the execution context in the current singleton.
https://outerboundsco.slack.com/archives/C02116BBNTU/p1623947638238600
It would be cool if this context could include some of the things that are in the various env vars now -- like the statemachine name, for example.
Additionally add max_workers, num_splits as well - https://outerbounds-community.slack.com/archives/C02116BBNTU/p1629149990116300 accessible through slack.outerbounds.co
@savingoyal just looking into this and wondering about the best way to implement. As I understand it, current is populated inside MetaflowTask.run_step, which has no knowledge of where the task is being executed (eg. local, batch or sfn). I think the same is true of the other execution context variables mentioned. So I think there are two options:
- Pass the execution context through to the task, and populate
currentthere - not a fan of this as we would be erasing some of the separation between the task and other classes. - Split
currentpopulation into two separate methods - one within the task and one for execution context.
Any and all advice appreciated :)