metaflow icon indicating copy to clipboard operation
metaflow copied to clipboard

Add execution context in `current` singleton

Open savingoyal opened this issue 4 years ago • 3 comments

Currently, the only way to check whether a task is executing locally, on batch or with step-functions is to sniff some environment variables - we can make it a bit easier by including the execution context in the current singleton.

https://outerboundsco.slack.com/archives/C02116BBNTU/p1623947638238600

savingoyal avatar Jun 17 '21 17:06 savingoyal

It would be cool if this context could include some of the things that are in the various env vars now -- like the statemachine name, for example.

corleyma avatar Jun 24 '21 19:06 corleyma

Additionally add max_workers, num_splits as well - https://outerbounds-community.slack.com/archives/C02116BBNTU/p1629149990116300 accessible through slack.outerbounds.co

savingoyal avatar Aug 17 '21 16:08 savingoyal

@savingoyal just looking into this and wondering about the best way to implement. As I understand it, current is populated inside MetaflowTask.run_step, which has no knowledge of where the task is being executed (eg. local, batch or sfn). I think the same is true of the other execution context variables mentioned. So I think there are two options:

  • Pass the execution context through to the task, and populate current there - not a fan of this as we would be erasing some of the separation between the task and other classes.
  • Split current population into two separate methods - one within the task and one for execution context.

Any and all advice appreciated :)

sam-watts avatar Apr 22 '22 09:04 sam-watts